First of all, congratulations on leaving CrossFit.
Second of all, rest day means a rest from training no matter if it's heavy or not. Any activity is going to require work output from the muscle tissue. So it's probably more appropriate to ask how significantly your training will be affected.
So if you're just doing something like slow-pitch softball and sitting half of the time, it won't be as tough on you as like a tackle football game.
Either way, to be on the safe side, I would take the extra steps in your recovery habits (ice baths, soft tissue restoration, sleep, nutrition, etc.).