I guess it depends on what you value most. Many guys who grew up skinny don't mind getting a little soft around the middle if it means they can gain some muscle and push some bigger weights in the gym. Others prefer to look more athletic year round. When you get down to it, both options are rooted in vanity to some extent.
The gallon of milk thing does not have to get you fat though. I had a roomate in college who was very lean but skinny. I took a look at his diet and told him that he was not eating nearly enough to gain mass. He took it a bit far and started eating a pound of bacon every morning! I kid you not. People started calling him "bacon man." Well....it worked. The guy put on almost 30 pounds over the course of the summer and he still had very defined abs. The folks on his rugby team thought he was on steroids but he was not. Extra calories can be very anabolic.
I would say that he was more of an exception and I would not expect most people to respond in the same way.
In any case, it is all about personal preferences. However, I am with you these days. I have no desire to gain any fat, even if it means that my muscle and strength gains will come more slowly.