Odds overwhelmingly favor your soil as being the primary limiting factor. Once you realize that ample volumes of air in the soil are as important to a plant's health as water and light, you'll have taken a step in the right direction. I've mentioned before that potting soil + worm castings = a very poor environment for root growth/health/metabolism. A plant's roots are its heart, and if we can't make the roots happy, there is no hope of the entire organism being happy; and that we can't see the roots is no reason not to consider the importance of root health to the plant's o/a vitality.
Recently (2 weeks ago), I addressed a large group of specialty growers at Matthaei Botanical Gardens in Ann Arbor, MI. The topic was container media (soil) and the primary message was the importance of aeration and drainage to the health of containerized plants, and how to ensure these properties remain stable for the life of the planting. As in countless other presentations I've made on this particular topic, the message was well received, and I could see the 'lights go on' throughout the entire audience as the evening progressed.
Liebig's Law addresses various factors that affect plant growth and yield; among them are: air, water, light, temperature, soil/media, nutrients ..... Liebig's Law of Limiting Factors states the most deficient factor limits plant growth, and increasing the supply of non-limiting factors will not increase plant growth. Only by increasing the most deficient factor will the plant growth increase. There is also an optimum combination of the factors and increasing them, individually or in various combinations, can lead to toxicity for the plant. What this means altogether, in essence, is that if your soil is the limiting factor, growth will not/can not improve, even if all other cultural factors are made perfect.
Your choice of soil is probably the most important factor to be considered when establishing a planting, and it will probably be the single most significant factor that determines how much effort it will take to reach an effort:reward quotient that pleases you. 'Soil choice' is the foundation of every planting, and it is very difficult to build something that will endure on a weak foundation. I fully understand that it's always up to the individual to decide what methods and materials are to be considered adequate, and if you choose to go forward in your endeavors with the idea that a potting soil/worm castings mix is good enough, my experience in dealing with the issues you face indicates that in all probability you will be unnecessarily and significantly limited in your ability to consistently produce healthy plants.
There is simply nothing a containerized plant needs in worm castings that can't be provided by way of either organic or synthetic fertilizers that have no negative impact on aeration and drainage.
Al
