Fancy a/c filters, too much of a good thing?

DH spent big bucks (like $30 ea) for a/c filters. they're supposed to be the super-duper, filter everything out of the air. We are empty nesters, not particularly prone to allergies, and our pets are primarily outdoors. We used to use the washable ones until he decided they're not as efficient in collecting the crud.

But I'm wondering: Could you possibly clean your indoor air soooo much that when you go outdoors (as gardeners) all of a sudden you are that much more susceptible to allergens? Or stated otherwise, could this ultra clean air eventually weaken your immunity to outside air impurities?

I would appreciate your thoughts on this matter. It is terribly hot outside for me to be freshening my patio pots, so my mind is wandering....