A sound reason to limit growth can only (and frankly must) come from a higher causal connection as to what kinds of growth contributes to human evolution in conjunction with achieving and maintaining a dynamic and healthy equilibrium with nature.
The endurance of humanity on this planet is a small and evolutionary insignificant smidgen compared to the evolution of the earth. Earth has existed for 4.5 billion years and will for another 3.5 billion or so. If we are lucky and behave ourselves well, we as an evolving species will have lived 2.5 million years with about 1,000 more years to go according to some experts (Steven Hawking and the likes). Our extinction will be anthropogenic, caused solely by ourselves, and thus our ability to manage ourselves must be under immediate and pertinent scrutiny.
So, it behooves us to take your question out of the narrow confines of our dying economic rebellion without a cause, and ponder why and what growth aids in our objective to maximize our sustainability, especially when you realize that the strengthening of our renewability can only secure the prolonging of our sustainability.
Indeed, the quality of our individual time on earth, not the number of our individual years alive, defines how we extend our sustainability as a species.
So, yes, there are many reasons why certain types of economic growth must be controlled, especially when such economic growth harms our collective fiduciary evolutionary responsibilities. But that wherewithal requires an understanding and empathy beyond your selfish existence.
For me, I look at my daughter for one second to remind me how I do not want her future to be defined and constrained by the artificial systems we built-in search of blind growth, a mindless growth considering the above context, so often responsible for our accelerated weakening as a species.