First Year with Plums on the Tree and there are Worms!!

Hey All! I planted a dwarf methley plum tree in 2012. It has flowered in 2013 and 2014, but this is my first year with actual plums. And I've got a lot (at least well above zero)! I've had several plums fall off the tree, starting when they were tiny, to now, when they are just small plum size. I've always picked them up when I've picked up after the dog. But this past week I've noticed some plums are turning red already, and actually those are the ones that are falling. So, I broke one open and found a nasty little worm in there. Then I started looking at more plums and found that a lot of them have little holes in them. I broke some open this morning and could not find any more worms, but it did seem like the ones turning red already had nasty gunk in them and the green ones (even with holes) didn't.

I guess at this point I'm wondering if they are pretty much all contaminated and I just need to pluck them and chuck them, or if there is a way to know which ones to leave on? I hate to throw away ones that are just a little blemished, but I sure couldn't stomach trying to eat one that's had a worm pooping in it! Also, any ideas on if the ones that look nice may actually have something inside too? (I like surprises, but not that kind!). I read on another website that someone found a worm in an unblemished plum. Also, any natural way of preventing this next year? And do these same type of worms go for apples too? I planted an apple tree at the same time, and it only produced one apple last year, and I think won't produce this year (there are some late blooms now, but only a few). Also, do the worms make the plums turn red faster, or are plums actually turning ripe this time of year? I thought they ripened in July. Lost of questions I know... I'd appreciate any answers you can give!

Thanks!
Pics out of order: first plum is outside view of plum in last pic. Clearly something yucky inside, but didn't see a worm. The second pic is of the inside of the plum in the third pic. Nothing yucky inside, even with the blemish outside. Very confusing!

First pic- Yes, this looks like a pest.
Second pic- NICE dog to pose for a picture!
Third pic- Might just have gotten a minor injury from a branch or something.
Fouth pic- That nasty stuff is called Frass- in common terms, bug poop (or worse).

You will have to do some research about which pests are known in your area. It might be a type of insect that lays eggs on the flower or newly formed fruit and the baby bug grows in the fruit for a while, then exits through a hole like in picture #1.
Here is a site for CA, many pests are the same. You can also see if TX has a similar site.

http://www.ipm.ucdavis.edu/

Thank you Diana_K! I threw away all the green plums that had black holes and this week my plums started becoming ripe so as I picked I threw away anything that looked nasty. Then I went inside and cut open all the ones I thought may just be blemished. So far, it's worked pretty well, although I think a probably threw away some good ones! I am definitely going to do my best to prevent this next year!