Do the current shrubs produce flowers at any time through the year? Yes= keep them. Are there other interesting features such as showy fruit or nice leaves? Yes= keep them.
Add:
Temporary color for the warm season to small focal points. Like where a walkway meets the sidewalk, or similar places where people can focus on them. Looks like the area has a lot of shade, so I would look into things like Impatiens, Clarkia, Iberis, Mimulus, Myosotis, Nemophila, Nasturtium.
These are the annuals that seem to be somewhat deer resistant in my area. Different species of deer, so research to make sure your deer won't eat what you plant.
Longer term color:
Ditto DoGooder's comments about falling leaves. Select things that will tolerate being buried in leaves. Mostly these are taller plants that won't get buried.
I am not sure which of these are OK in your zone, but here are some evergreen shrubs that are deer resistant in my area. I am just naming genus, do your research to find the right species and variety for you. For example, many of these have variegated leaves, or new leaves come out red, so even without flowers they are not just plain green.
Ilex (not noted for flowers, but showy leaves and berries)
Pieris (different varieties with red new leaves, flowers, variegated leaves)
Rhododendron (but not usually the group called Azalea)
Viburnum
Philadelphus
I think most true ferns are deer resistant. No flowers, but nice leaf shape.
Larger plants, not quite trees, but large, open shrubs:
Magnolia stellata- deciduous, but beautiful show of flowers in the early spring.
Acer palmatum (MANY varieties)- Colored leaves. Superb form. Deciduous.
