Based on the fact that your second option is caloric restriction, I'm assuming your primary goal is fat/weight loss. That being the case, not eating for a significant period of time post-workout should have significant benefits. Consider that folks following a 1-day-on, 1-day-off eating schedule are regularly training on their fasting days, and therefore not eating for possible 24 hrs after training, and the reports we've been getting have been invariably positive.
The degree of improvement varies among individuals. How long have you been experiementing with IF? Have you spent any time on just a caloric-restriction setup? The easiest way to answer that question would be to spend some time on both and compare, although this is slow at best.
Bottom line, I would encourage continuing with IF. If it becomes problematic due to scheduling reasons, try returning to simple calorie restriction, and keep detailed notes for the sake of comparison.
Keep us posted!