According to Algoa FM, police recovered four of the stolen items in a cemetery in the city.
"Our dog unit of Port Elizabeth received information from an informer which led them to the Dutch Reformed Church in Sundridge Park.
"At the back of the property they found four paintings and it seems to be similar to the paintings that were stolen in the Gauteng area," police spokesperson Captain Sandra Janse van Rensburg was quoted as saying.
The whereabouts of the fifth stolen painting is unknown.
The pieces were stolen from the museum's permanent collection on Sunday when thieves posing as art students staged a daring arm robbery.
"Three men, under the pretence of being students and their art lecturer, asked to view specific pieces. After they were shown the paintings, they then tied up the museum official at gunpoint and took off with six paintings," mayoral spokesperson Pieter de Necker said on Monday.
"They left one painting [by] Irma Stern behind. Presumably because it was too big to fit into their [getaway] car."
According to museum staff, the thieves arrived with a "shopping list" of what they wanted, and demanded to be given specific paintings, Sapa reported.
After holding staff member Dawood Khan at gunpoint, they took out their list, and demanded to know where the paintings were.
De Necker hinted that growing international demand for South African artwork might have been the reason for the robbery.
He said he and others believe the thieves were commissioned to go after those specific pieces because of their behaviour at the museum, according to AP.
"We're very, very surprised. It is very uncommon," he said.
"We have realised also that over the last few years ... the overseas market has grown into wanting South African art."
The robbers favoured oil paintings in their theft, grabbing a 1931 painting by artist Irma Stern of brightly coloured fishing boats waiting against a pier (Fishing Boats), a gouache drawing of an eland and bird (Eland And Bird - 1961) by landscape artist JH Pierneef, a pastel-toned street scene (Street Scene) by Gerard Sekoto, a thick-stroked oil painting of a chief (Hottentot Chief) by Hugo Naude and a 1936 picture of a cat near a vase full of petunias (Cat And Petunias) by Maggie Laubser.
The museum closed provisionally, and will re-open on 20 November.
It took precautionary steps by removing other valuable pieces and security has been tightened.