The hangers are rubber, yes, but you have a solid mass, the engine, with a long straight tube running several feet along the bottom of the car with no way to flex independently of the engine. The longer and thinner an object is, the more prone to stress it is. Those rubber bushing are there to absord vibration and eliminate exhaust rattles, not compensate for the flexing of the exhaust. Trust me, I've tried to pry my downpipe back just an inch working against the rubber bushings to get a 2.5" O2 housing to fit, and I had a bitch of a time moving it the slightest amount. They're fairly solid.
If you want to try it then by all means go ahead, but I've seen 2 downpipes crack at the exhaust housing flange, and neither one had a flex section. :dunno: