The only clue you've given to what may be wrong with your plant - and something is definitely wrong - is that you purchased a 25" container grown plant.
Larger container grown plants can harbor hidden root conformation problems that most gardeners never investigate or experience. Typically, the plant has been grown for too long in the container (many times, a series of too small containers and continually potted up). This results in spiraling and circling roots, which eventually either constrict the normal trunk and buttress root growth or fail to root out well into surrounding soils.
If you did not loosen and spread out the root system of the container plant when you installed it, I suspect that this may be the "root" of your problem. The only way to determine this for certain is to dig it up. You may not wish to go to this level of forensics.
It is not normal behavior for a Chinese Fringe Tree (Chionanthus retusus) to behave in this manner. I have planted this species a number of times over my career, and I'm familiar with grand old specimens. They leaf out entirely, and then flower on the terminals of branches.
You should take and post additional closeup pictures of all the parts of your tree, including the trunk (especially at the base), the branches, stems, buds, leaves, flowers, and any fruit that might be forming. That way, if there is anything else not immediately evident in your initial images, we might be able to comment on that.
It also might be fun to compare your plant to the better performing neighbor's plant, for reference.
It is interesting that Chinese Fringe Tree can perform well in San Diego. I wonder if the plants you can acquire there are from difference provenance than what is grown in parts of North America where we have cold winters and hot humid summers. Truly an adaptable species!
