The problem I have with limiting speech because it "harms" someone is that injury done by speech does not leave any tell-tale signs like bruising and broken bones so it is hard to tell if someone was actually harmed. This makes it too easy to abuse. What is to stop a politician from saying that the disclosure of his affair "harmed" his family? How can we prove that a corporation wasn't "harmed" by a bad review of their product? What if the KKK said that they were harmed by being called racists? As you see, it becomes a double-edged sword.
On top of that, such a rule would destroy the stand-up comic industry over night. Shows like South Park, the Simpsons and Family Guy would no longer exist.
The worse part would be we would become a society of mice, no longer willing to speak out for something we believe in for fear that it might offend someone.
"I am not an advocate for frequent changes in laws and constitutions, but laws and institutions must go hand in hand with the progress of the human mind. As that becomes more developed, more enlightened, as new discoveries are made, new truths discovered and manners and opinions change, with the change of circumstances, institutions must advance also to keep pace with the times." - Thomas Jefferson, 1816