The two governments agreed on "the repatriation from South Africa of Libyan funds", totalling almost R10bn.
It is thought to include diamonds and gold.
The assets were placed in South Africa by the Libya Investment Authority, the Libya Africa Investment Portfolio and the Libya Africa Investment Company - funds closely controlled by the Gaddafi regime.
The Treasury said the assets will be returned in line with UN rules.
The deal was reached at a meeting between Finance Minister Pravin Gordhan and a Libyan government delegation on June 4.
"The decision was informed by the fact that the government of Libya established a single body in 2012 to co-ordinate the repatriation of assets to Libya," the government said.
This body co-operated with the committee formed in terms of the UN Security Council Resolution 1970 (2011) and the panel of experts which co-ordinated repatriation to Libya of assets frozen in various countries.
South African media last week reported that Libyan investigators had approached the treasury with evidence that the assets were held by four local banks and two security companies.
A letter sent by Libyan investigators said large amounts of cash from Libya's state investment funds had been uncovered in South Africa and neighbouring states.
South Africa opposed Nato's military intervention in Libya during the rebel uprising against Gaddafi's regime in 2011.
The veteran leader was captured and killed in October 2011 while he was trying to flee his home town of Sirte.
Libyan investigators learned of state funds stashed in South Africa from Gaddafi's former intelligence chief Abdullah al-Senussi. He is in a Libyan prison after being extradited from Mauritania.
Part of the missing money is allegedly controlled by Gaddafi's former chief of staff Bashir Saleh.
Saleh, who headed Libya's R400bn sovereign wealth fund, is wanted in Libya for fraud and is the target of an Interpol arrest warrant under the alias Bashir al-Shrkawi.
Several high-ranking officials linked to Gaddafi's regime, including members of his family, have been linked to corruption scandals that emerged after his death.
Last month, Libya's new government adopted a law barring Gaddafi-era officials from holding government posts.