SPOOKED DEER?

I have a question. I have deer coming in my backyard. Now that there are food and water outside, they tend to visit at nights (mostly). Sometimes during the day.

now i really like them coming and i enjoy watching them feeding at the trees and the birdfeeders.

But i do have some unwanted visitors at the feeder, like squirrels. I was trying to drive them away one day. I use a bounce ball to send away the squirrels to the next yard, then I follow them to the forest cover so that they go far off and not come back in a while. That day, I was doing the same, I approached the squirrels rapidly, then followed him to the forest covering (which is connected to my backyard), then bounced the ball to scare off the squirrel. But I did not notice a young deer was there. he seemed to get a mild shock from the sound of the ball bouncing. It jumped and moved away a few feet when I noticed it. I stopped immediately. The deer also stopped after a few feet and kept staring at me. Then came another deer (who was bigger) from behind enjoying the leaves. He paused as well and stared at me for a while. They did not seem too scared as they were looking at me from the same spot. then they casually slowly shifted. But I came back slowly before them.
Now I want to know, were they spooked? (I mean the deer) Will they not come back to my yard, if they were scared? I mean I did it by accident. I did not know they were there. i was just there to scare away the squirrels, so that the birds could eat.

Spooked deer will return to their bedding area, but when they return depends on how much the intrusion frightened them.