Rubbing with your finger works on the more rubbery stuff, WD40 is good
for this, there's a product called goo-gone, and you can also get
plastic scrapers. You could also cut a small piece of wood into a
'scraper'.

Dave

If WD40 doesn't do it try mineral spirits.

I have painted AL wheels and I use wheel cleaner and a brush with the
wheel off the car. If that doesn't work I resort to mequiar's heavy cut
cleaner or in a couple cases very careful use of a razor blade once it
was softened up by wheel cleaner. Scraping with a plastic tool to cut
gaps in it and then more wheel cleaner helps some.  

I usually leave it wait until I am switching winter and
summer tires to do it.