The fighter, known by his nickname Hurricane, had been battling prostate cancer and passed away in Toronto, Canada.
Carter's boxing career got off to a promising start, but in 1967, he was wrongfully convicted of three murders in his native New Jersey and spent 19 years in prison before he was finally released in 1985 following multiple appeals.
His story inspired 1999 film The Hurricane, starring Denzel Washington as Carter, and the Oscar winner was among the celebrities who shared their condolences after the news broke.
A statement from Washington reads: "God bless Rubin Carter and his tireless fight to ensure justice for all."
Washington brought Carter with him on stage to accept his Golden Globe award for Best Actor in a Motion Picture Drama in 2000, saying, "This man right here is love. He's all love. He lost about 7,300 days of his life, and he's love. He's all love."
Fellow boxing great Tyson took to Twitter.com to express his sorrow, writing, "We lost a great man today, Rubin "Hurricane" Carter, the boxer who was wrongfully accused and became a symbol for racial injustice. Rip."
Evander Holyfield tweeted: "Rip Rubin "Hurricane" Carter (May 6 1937 - April 20, 2014)", along with two black and white photos of Carter in his fighting stance, and another of him knocking out an opponent in the ring.
Olympic gold medallist George Foreman added his own tribute, writing, "A sad day, for Carter was a hero... Sorry to say I never met Hurricane. But he was a real Champ... Prison; so many wrongfully convicted, 20 years he was with the salt of the earth. He is proof, good folks are behind bars."