I have an '86 Camry that has spent most of it's life OUT of OD...

That's always the compromise...fuel economy for performance.  Try it
each way and see if the fuel economy you give up is worth the extra
speed.  

One can argue that if the engine spins less, it will have reduced wear
but also consider that if you have to switch in and out of overdrive
often, it increases wear on the transmission.

I'd suggest that if you often forget to enable OD when you do get out on the
highway, you'd probaly be better off leaving OD enabled all the time.

As for a difference in around-town fuel efficiency, you should try to
measure it and, please, let us know what you find.

Leave the overdrive ON.  It's meant to be that way.  The reduction in
RPMs and corresponding fuel economy increase over leaving it off will
probably surprise you.  The only time I ever switch mine off (03 XR) is
for passing on a two-lane road.

Driving with the overdrive disabled will improve responsiveness at the
expense of fuel efficiency, especially when driving at steady speeds at 30
MPH or higher.  The engine will operate at higher RPM with the OD off so you
will get more wear and tear and lower fuel economy.  The effect on fuel
economy really varies with your driving conditions and driving style.  The
most accurate way to tell is for you to do your normal routine for a couple
of tanks with the OD off and then a couple more with the OD on and compare.

BTW, I would not routinely downshift from 4th to 3rd to provide engine
braking as that will quickly wear down the friction surfaces in the
transmission.  It is a lot less expensive to replace brake pads than tear
down a transmission to replace friction surfaces.