Anyone grow olive trees in the South US??

Thinking about giving it a try and I just can't find anyone who knows much about olives. I know the basics about them and technically I should be able to grow them but I have no idea if bugs, humidity or something else may give me a real problem.

These sites stress climate as the most important limiting factor, a combination of warm days and cool nights during winter:
http://aggie-horticulture.tamu.edu/extension/fruit/olive/olive.html
http://www.davero.com/trees
http://www.oliveoilsource.com/site.htm

There is an olive orchard in Wimberley, TX. They grow, press and sell their own olive oil.

I have had one in a five gallon black nursery container growing on my deck for the past 3 years. It has sustained the temp drops in Atlanta area here without any added shelter. I did bring it inside one year when temps were to be around 17 degrees. The plant seems to be doing well. It is a vigorous grower once established. I cut mine back every year to about 3 foot stems so it won't get too large. I originally had thought about making a bonsai from it but am beginning to like it as a container plant. It did produce a few blossoms but only one fruit so far. I heard about growing olives on a local gardening radio program where one of the call-in said that his had survived temps dropping to 10 degrees. Give it a try, what can you lose except for the cost of the plant.

Temps aren't a problem for me about 25 degrees is the coldest it will get here. The main problem seems to be with high humidity and having all the fruit fall off way before it's ready. Actually in a warm year I may not get enough cold to warm swing to get the tree to bloom. Found someone up the road from me and he hasn't gotten past having very small olives then having them fall off.