Poison ivy can definitely cause localized swelling. Have you dealt with the surface? There is a product I highly recommend called Tecnu. I use this and then dish soap (something with grease cutting ability). For some gov't approved recommendations:
http://www.fda.gov/forconsumers/cons.../ucm049342.htm
At this point (a few days later), if it is reaction based swelling, then I'd do some things to generally reduce inflammation (ice and non-pharmeceuticals). For myself, I'd be taking some Benadryl and see how the swelling responds.
Remember, poison ivy (urushiol) can get on EVERYTHING. Act a little paranoid, clean everything that you might have come in contact with so you don't get reinfected, and hopefully you'll get back to normal soon.
Best,
Mark