Something that is emerging in evolutionary biology is a shift towards looking at the distribution of a gene across a population as opposed to focusing on the "fitness" of one individual. From this perspective we have a potentially much larger sample size with a given gene (or set of characteristics) interacting with the environment. It is here that we see a population wide tendency to show favorable or unfavorable adaptations to a given environmental stressor.
In the situation you described you, me and our immediate ancestors have not been impacted to a degree that makes it impossible for us to reproduce. In general we do however have smaller brains, less robust frames and decreased fertility vs our HG ancestors. Certainly a slide away from a better pairing of our genes and the environment.
Now we (as a species) have passed through the: Agricultural, Industrial, and information revolutions. Each of these steps have decreased our activity level and refined our diets, and our health has suffered. We are now facing a time when our diet is SO refined and activity levels SO low that children are failing to enter puberty due to hormonal derangement. Things have gotten bad enough that some WILL be weeded from the gene pool before being able to reproduce. The folks who go on are better adapted to: refined foods, low activity and crushing boredom as a lifeway....unless things shift so dramatically that fertility as a whole drops to un-sustainable levels. That is unlikely to happen but it's not necessary to die early to be removed from the gene-pool.
In addition to natural selection there is epigenics which is the interaction of the organism with the environment. This can effect gene expression to a remarkable degree, although most geneticists do not think this changes heritable traits ala Lemarkism. Good follow up reading includes:
The Blind Watch Maker
"Survival will be neither to the strongest of the species, nor to the most intelligent, but to those most adaptable to change."