The problem I have with limiting speech because it "harms" someone is that injury done by speech does not leave any tell-tale signs like bruising and broken bones so it is hard to tell if someone was actually harmed. This makes it too easy to abuse. What is to stop a politician from saying that the disclosure of his affair "harmed" his family? How can we prove that a corporation wasn't "harmed" by a bad review of their product? What if the KKK said that they were harmed by being called racists? As you see, it becomes a double-edged sword.
On top of that, such a rule would destroy the stand-up comic industry over night. Shows like South Park, the Simpsons and Family Guy would no longer exist.
The worse part would be we would become a society of mice, no longer willing to speak out for something we believe in for fear that it might offend someone.
"Men fear thought as they fear nothing else on earth -- more than ruin -- more even than death.... Thought is subversive and revolutionary, destructive and terrible, thought is merciless to privilege, established institutions, and comfortable habit. Thought looks into the pit of hell and is not afraid. Thought is great and swift and free, the light of the world, and the chief glory of man." - Bertrand Russel