I have heard many conductors say that as important as the notes are to music, the rhythm is more important.
Rhythm includes all of the aspects of the notations of the pitches in the melody and harmony but also keeping the group together while they are playing. Rhythm is interesting in that it is an entity that once established, the performers must join. It continues along whether the performers keep up or not. That is why learning to play to the clicks of a metronome is so important to a performer -- and so annoying!
Some of the problems with keeping up with the rhythm are laziness, lack of confidence, problems with proficiency, taking breaths, and when to use rubato. For a singer or wind player taking breaths can be a real issue for keeping up with rhythm and tempo. It seems that after taking a breath, the player "bends" time to their needs and does not realize that they have gotten off the beat. They need to plan their breaths to match the tempo and be careful not to rejoin the rhythm late or at a different pace. Then there is the singer or instrumentalist who feels the need to "stretch the time" a bit here and there through rubato. The problem with this is two-fold 1) any stretching of the time has to be executed the same by all the players and 2) the original tempo is always compromised by any changes. Thus the players can get out of sync or the tempo can get slower and slower until the performance suffers.
Learning to tap in tempo and establish INNER RHYTHM is a skill that must be practiced and practiced and practiced again. As singers we "count sing" - we sing the notes on pitch but count eighth notes (usually) as the measures go along - 1 and 2 and 3 (or "T') and 4 and. This helps us feel the beat and tempo and gets everyone in the group to stay together. [Conductors often say "T" instead of three because of the instant consonant. He can hear when the the "T" sounds line up.] Then when the same section is sung again with all of the elements - melody, harmony, words, accompaniment - they stay together better. That is the goal anyway. Working at it helps proficiency, but each session can be a struggle unless everyone tries their best and stays focused. One person getting off ruins the performance of the entire group.
So, get with it and tap into that INNER RHYTHM!