Tax havens

It's a sad commentary on our times that so many of both parties are against removing corporate tax havens. The argument usually goes this way, Without these tax havens we can't be competitive in business.

This is like saying, Without polluting the environment, we can't be competitive in business. Or, Without doctoring our account books, we can't be competitive in business. Or, Without hiring illegal aliens at starvation wages, we can't be competitive in business.

What is wrong with this picture? (But does anyone actually care any more?)