You might be right but did you see that very large study done on identical twins covering decades which showed that genetics were the least predictive of life expectancies? Basically very low correlations. You had twins dying 50 years apart. A lot of it came down to circumstances (smoker/non-smoker) and to just staying out of harm's way or illness. I don't have a link but the NY Times ran a long article on it.
I think when people say "it's all down to genetics" it's a modern way of saying "it's all down to Fate." Remember the Norse myth of the goddesses (Norns) who literally spin our lives out on a spinning wheel and decide when to cut it short? That seems to be it.