The average American doesn't actually care about Cuba.
One very vocal group does though -- Cuban-Americans. Because Florida still takes in a lot of Cuban refugees, these people understandably don't like Castro. The descendants of refugees don't either.
Because of their influence in Miami and Florida overall, there is a lot of pandering to them in politics and in public image.
Eventually, we'll wake up and realize that ending the embargo will actually improve life in Cuba, however. Fidel is almost dead, and with a market open to American trade, Cuba would most likely become a very fertile area for market liberalization and for expanding political freedom.
Until then, refugees and their descendants will inadvertently encourage the continuance of a needless embargo.
"Chaos... isn't a pit. Chaos is a ladder. Many who try to climb it fail and never get to try again. The fall breaks them.
And some are given a chance to climb, but they refuse. They cling to the realm, or the gods, or love. Illusions.
Only the ladder is real. The climb is all there is."