If food becomes an issue, it should be noted that the majority of America's corn crops are fed to cows and other livestock. In doing this, much of the potential nutritional value that can be produced is reduced to almost a tenth of what could have been achieved had those fields been used to grow food for human consumption.
This is an example of how much waste is produced by our lifestyles alone. The problem we will eventually face is that we are depending upon imported fertilizers and contaminating sources of water through our agricultural activities. If we restricted the use of fertilizers and certain irrigation methods in arid locations, we would have lower crop yields, but that might be sustainable. If we skipped feeding most of it to livestock and instead grew crops for human consumption, we could sustain a larger population than we already have in the US.
The issue we currently face is that we in America have been leading our economy by the idea of spend, spend, spend... but that's what lead to the housing crisis we face. If we don't have a sustainable supply of food, water, and energy; then a larger population would only serve to destabilize any future projections we can make about the future.