"It is not almost every time I feel like crying during my visits... You could swear no-one lived in that shack," Zuma told premiers, ministers and MECs at the President's Co-ordinating Council in Pretoria.
The family living there had been totally destroyed, he said, explaining that the owner's daughter left home to become a prostitute, and had returned when she fell pregnant.
Zuma made an unannounced visit to the Sweetwaters informal settlement, in Johannesburg, on Monday.
He has also visited, among others, the Madelakufa settlement in Tembisa, east of Johannesburg, where he found the conditions "very bad" and warned councillors to fix the problems.
There is currently a 2.1 million housing backlog and more than 2 700 informal settlements.
Zuma told the President's Co-ordinating Council that an investment in human settlements would be an investment in the future.
Habitable and decent homes would promote human dignity and stability in communities, he said.
He said apartheid laws had made black people foreigners in urban areas, and had forced them to settle in designated locations and temporary residences.
"For those in rural areas, they were made to belong to various Bantustans with no plan to develop roads, transport, electricity, sanitation, running water or any other infrastructure," he said.
The government's focus on human settlements as a key national priority, would attempt to undo this legacy.
Zuma said Sweetwaters resident Enock Vilakazi had appealed to him to improve conditions for the sake of his children. For his part, he had resigned himself to dying there.
"You could hardly think that human beings are sleeping there," Zuma said of Vilakazi's shack.
"We have gone some way towards improving our goals, although clearly there is still a lot to be done in some areas," he said.
Living 'like pigs'
Any attempt to explain why people were still living "like pigs" when the country was nearly celebrating 20 years' of freedom would be meaningless, said Zuma.
The President's Co-ordinating Council was to examine the human settlements delivery agreement and discuss solutions to problems affecting delivery during its meeting.
Zuma said all programmes had to ensure people's dignity, self-esteem and pride was restored. "The living conditions have to improve," he said.
Good progress had been made in changing mindsets from the provision of "housing" to "human settlements".
Human settlements were not just about building houses, but also about access to social amenities like clinics, schools, water, electricity and recreational facilities.
In the last financial year, the government invested R15m in housing subsidies for the poor and 160 000 homes.
However, about three million households - including informal settlements like Sweetwaters - were still waiting for electricity.
Zuma said the government was aware that some people receive houses, but then rented them out and moved back to informal settlements, raising questions about the effectiveness of the system.
Foreigners also appeared to be benefiting when they should not, he pointed out.
"We are working on addressing this... having been government for 16 years, we must know what's working and what is not. We need to change the manner in which we do things and move quicker."