That's a reasonable assumption, although the possibility still exists that
you're more susceptible to detecting the distortion in the fronts due to
their location in front of you.  You're probably right, but it may first be
worth making sure that even if you climb into the backseat and face the 6x9s
they're not distorting as well.

The easy solution is probably to install bass blockers on your 4" drivers.
After all, there's only so much you can get out of 4's.  By virtue of their
size alone (and their smaller cone area, smaller xmax, and perhaps smaller
VC size), it's difficult to get them to faithfully reproduce full freq range
power content.

Aside from that, look for a high xmax driver with decent sensitivity.  Vifa
comes to mind...