I agree that most people CAN handle doing more than they currently do - but it has to come from somewhere. If I were to suddenly get a physically active job (please! make my year!), I'd be sleeping like a baby and eating a LOT more than I do now - and loving it! If I were to get a physically active job, have to go around on the same amount of sleep and food as I do now - or worse yet, go to Zoning on 11 blocks, which I KNOW I have to eat more than that - I'd be feeling pretty crappy in short order.
In my personal experience, I don't lose bodyfat unless I'm healthy and happy. If I get sick, guess what? Fat stays on. If I'm stressed out, same thing. If I'm not recovering enough to be ready to at least go hard the next day...you get the picture.
Right now I'm doing more Gant Grimes' hybrid programming and very few WODs. I came back from vacation to set three PRs on Tuesday and went in this morning to completely ditch my workout after having to bail on easy lifts when my brain just couldn't hang onto things. So I just played around after that.
Eh, I have only personal experience, no expertise, but I offer it anyway because sometimes it helps.