Milk isn't paleo, but I have to admit, having removed it from my diet, and then brought it back, I recover much faster drinking milk postworkout. For months I did chicken and sweet potatoes, but there must be some factor that makes milk more bioavailable or something. Then again, sweet potatoes aren't technically paleo either....
That being said, if you're smart about your food choices, you can easily get enough calories. Especially if you go sort of pseudo paleo. I throw heavy cream into my scrambled eggs, heaps of butter into my potatoes, and olive oil on just about everything. Coconut milk is awesome too. I get the lite coconut milk and can literally drink a can at a time. The regular coconut milk is a little thick to drink straight from the can.
If you think about it, there's really no reason why can't build muscle on paleo if you're getting the calories and nutrients.