I saw this chart today (here) and I couldn't believe it so I checked the Census Bureau again and it was true.
Current projections are that the US population will now peak at just under 370 million in 2080, but with it reaching 359 million in 2040, it's a really long plateau.
Given the decline in fertility rates and the hardening of attitudes against immigration and the trend, I would definitely say the risks are to the downside.
There has been a lot of talk in the market about China's population peaking, but China still has a significant amount of rural-urban migration to supplement its productive workforce.
In any case, the United States is much more important to the global economy, and these changes in population have a significant impact on the long-term prospects of almost every consumer-facing business, as well as real estate values.
It is a bigger problem in Europe and many other parts of the world. Changing attitudes about fertility and the number of children people want to have is a real problem for markets.