Flaky Cast Iron Skillet

For the last 2 times I've made cornbread, I've heated up the oil in the skillet and then poured it into my corbread mix. Normally I just pour the mix into the pan with the oil in it, but this time I decided to read the package directions and it said to empty the oil into the batter. Well, I got huge flakes of iron into the mix. yuck! I've had this pan for years, and never wash it with soap and water, just wipe it out with a damp paper towel. Do you think its time to get all the built up seasoning out and start again? Maybe there's another trick to this?

Thanks!