Sometimes the collision of the ball to the racket is way off, but he hits it anyway. Like when the ball will be near the ground (about to bounce twice) and the player will attempt a normal swing which misses the ball but he will hit the ball anyway. In those cases the racket-swing animation should be adjusted to perform a lower swing and actually make contact with the ball, or miss the ball. (maybe that type of low last-second swing could be tied to a particularly type of swing tied to a specific button press?) I didn't notice this at first, only when viewing replays that sometimes they are hitting the ball and the racket is nowhere near the ball.
I don't think it should be any harder to hit the ball, but maybe some animation of the racket swing could be added for when there is a low ball, so when you look at the replay he's actually swinging lower and the racket hits the ball. There's plenty of missed swings already in the game where you are slightly late, and those look amazing and realistic, but there are still many times where based on the swing of the racket, he should miss but it is hit anyway.