Calories matter...weight loss can happen...will also depend on your calorie load on your IF day (which should be lower than non-IF days). Other factors come into play such as activity level, insulin resistance (which can go up during a fast) and other fun stuff. Keep your IF days clean eating, mostly protein.
I wrote a whole free ebook on Intermittent Fasting here
....so I wouldn't have to cut and paste the same info over and over.
Don't use IF as an excuse to eat junk you wouldn't normally eat (or as I like to call it IFOC, IF on Crap)....won't work out well for you.