Matlala, a former boxing champion, died of a lung problem at the Charlotte Maxeke Johannesburg Academic Hospital at 11am on Saturday.
"'Baby Jake' was one of our heroes and ambassadors, who flew the country’s flag high," Mbeki said in a statement.
"On this sad occasion, we should pay tribute to him and say thank you for representing us and making us proud as a people."
Boxing manager and publicist Brian Mitchell was also shocked at the news of his death, which was confirmed by family spokesman Pastor Alan McCauley, of the Rhema Church.
"It's great loss for South African boxing," said Mitchell.
Arguably the most successful boxer produced by South Africa, Matlala, 51, was often described as the little big man, South Africa's smallest boxing giant, or the small fighter with the big heart.
Four times a world champion in the flyweight (50kg) section, he was born in Meadowlands, Soweto on August 1, 1962. His father gave his only child not just boxing lessons, but also life lessons.
"My parents taught me to be focused. I went to school in Soweto. When I came home I did household chores," Matlala said in an interview published on safrica.info.co.za.
His professional career began in Port Elizabeth, in the Eastern Cape, in February 1980 and, by the time he retired in March 2002, his record stood at more than 50 victories.
His titles included World Boxing Organisation flyweight champion in 1993, the light flyweight title in 1995, the International Boxing Association junior flyweight title in 1997 and the World Boxing Union (WBU) flyweight title in 2001.
He was the only South African boxer to have won four world titles and, at 147cm, he was the shortest man to have been a world champion.
Matlala said in an interview in 2003 that he had stopped boxing because there were no big names left for him to fight.
Matlala, who completed a BCom degree at the University of SA, was never put off by being the shorter one in the ring.
"Height is not an issue, it's in the mind," he said.
His strategy was simple -- he constantly threw punches at his opponent's body, until the opponent got tired, and let his head down.
"I work the body, then the head will come," said Matlala.
His death came just days after that of former president Nelson Mandela, who was a boxer in his younger days.
Mandela and US actor Will Smith attended Matlala's farewell fight.
Afterwards Matlala presented his WBU belt to Mandela.
When he retired, Matlala remained actively involved in the community, helped to raise funds for HIV/Aids programmes and supported the SA Police Service in its campaign to get members fit.
He starred in the television reality dance show Strictly Come Dancing, endearing himself to viewers with his amiable personality, even though he was not as comfortable with dancing steps as boxing footwork.
He also had to deal with bad media coverage in his day. In the late 1990s, a family friend claimed that he had raped her. He settled the matter out of court. Some newspapers suggested he paid out as much as R1 million.
Shortly after his retirement, a business venture in which he was involved, fast food outlet Jake's Diner, suffered severe financial losses.
Matlala tried his hand at motivational speaking and enjoyed a stint as a boxing commentator.
However, by the time of his death, he was said to be financially down and out.
In 2010, Golden Gloves boxing promoter Rodney Berman arranged a black-tie charity fight called The Night of the Little Big Man to raise funds for Matlala to cover his medical costs after he was hospitalised for weeks, reportedly with double pneumonia.
The Rhema Church, of which Matlala was a member, also called on the public to help raise funds for him, and SuperSport agreed to sponsor the broadcast of the charity event.
At the time, Berman's publicist, Terry Pettifer, who has himself since died, told The Times newspaper that Matlala had "lost everything and needs all the help he can get".
Matlala and his wife Mapule also tried unsuccessfully to market a DVD/CD combo of his best fights and jazz and gospel songs by his wife.
Matlala leaves his wife, who was his childhood sweetheart, and two sons.