A religious group had leased the vacant land, demarcated as a public open space, to hold open-air services for its congregation.
However, said property company spokesperson Brian Mahlangu, about three weeks ago the group began erecting a structure which was to become a mosque on the land.
Their lease had expired in November 2009.
The structure was erected without the City's permission or approved building plans, he said.
"The land is zoned public open space which legally does not allow for the erection of any structures on the land besides those... approved by the CoJ (City of Johannesburg) with the co-operation of all its municipal owned entities."
An agreement was subsequently reached with the group that they would vacate the land by August 21. The company will then apply for a demolition order to bring down the structure.
"It was disappointing to find them erecting on our land without the proper procedures," Mahlangu said.
"JPC will do all it can and that which is necessary and allowed by the law to prevent any illegal occupation and building on the City's land."
He was unable to immediately provide the name of the group responsible for erecting the structure.