"We noted and we are worried about the downward trend in all the provinces except in KwaZulu-Natal," he told media in Johannesburg.
He was speaking after a meeting of the party's national working committee.
However, he added that the ANC was "proud" to have 5 633 councillors across the country, a presence in every council in the country.
"These results must be celebrated because they constitute a massive victory for our movement," he said.
Mantashe expressed concern about the decline in the ANC's performance among minority communities.
He conceded that there were communities who would be "switched off" by comments which "appeared racist" made by ANC leaders, but added that South Africans remained "too over-sensitive to each other".
He was asked about comments made by ANC Youth League president Julius Malema, for example.
"Comments that are a departure from ANC policy, do not do good for the ANC," Mantashe said.