yeah those are some good points. basically, you should gauge it based on where you are going to run.
i'd have to agree that stubbing your toe is the real danger while wearing VFF. i've stubbed mine once or twice and felt like i asked for it by wearing them.
i typically only run on tracks and on sidewalks in my neighborhood. both of which i feel are fine with VFF, and both i think would be a little rough if fully barefoot. i've tried running on trakcs barefoot a few times, and you really need to build those callouses if you want to go that route. a sidewalk will have glass every once in a while...
as for running in the woods, i tried that once in VFF because i friend wanted me to come. i wish i had brought regular shoes to that outing...
also, i jump rope a lot in them. when i miss, i generally always catch a toe, and the VFF is great for that. you don't feel any pain in them, but you can give yourself a nice whip if you are barefoot and catch your toe. if you use one of those really fast wire ropes, i would make sure that your VFF covers the top of your foot, or else just wear a shoe.