We have 2 Osmanthus fragrans planted in partial sun. Neither has grown significantly beyond their original pot size at planting and both look somewhat bedraggled. Meanwhile, a neighboring Pittosporum is thriving as are cycads and Japanese Maples nearby. We're stumped. Both have been in the ground for 3 years and bloomed intermittently. At the time we planted the soil was amended with compost and manure in a hole considerably larger than the root balls. They have been mulched with shredded bark and dillo dirt in fall.
Stunted Sweet Osmanthus
mine have proven to be slow the first year but the 2nd year they are taking off! Sounds like you are caring for it correctly, so I would just give it some time.