Now the government is vowing to get tough, with activists hoping mandatory restrictions on shipping emissions will be among a raft of measures announced next week aimed at making the city more environmentally friendly.
A total of 410 560 vessels arrived and left the port in 2011, with cargo ships vying for space in the crowded waters alongside public ferries, tourist junks and luxury yachts.
This level of activity means shipping is a key polluter in a city where, according to the University of Hong Kong, air pollution kills about 3 200 people every year.
Simon Ng of the Civic Exchange think-tank blamed the pollution, which often shrouds the city's dramatic skyline in thick smog, for driving away talent.
"Just imagine a small power plant right next to your doorstep, producing a lot of pollution every day, almost 24 hours a day, what would you do?" he said.
"Ships are now producing a lot more pollutants than we had anticipated, and it is becoming a major problem that we need to address."
Rising emissions from ships, which burn heavily polluting bunker fuel, will have seen shipping overtake the power industry as the biggest source of the colourless toxic gas sulphur dioxide in 2012, Ng predicted.
Nearly 400 Hong Kong people died last year from breathing in pollution from bunker fuel alone, he added, citing a study on marine pollution by his think-tank.
Activists say Hong Kong lags behind the rest of the world on environmental issues ranging from recycling to cycle lanes.
And when it comes to shipping, while vessels calling in northern Europe and North America are mostly restricted to fuels with 1% or less sulphur content, Hong Kong allows 3.5%.
Last year however, it did introduce a voluntary scheme in which ships using 0.5% or less are given a 50% discount in port dues.
Christine Loh, an environmental crusader who has become the government's environment undersecretary, said the scheme was just a "small start".
"We want to regulate. We want it to become mandatory and we want to take the scheme across the border to our neighbours in Guangdong," she told an air quality conference last month.
"We would like, within the next few years, to collaborate and work very closely with the Guangdong province so the whole of the water of the [Pearl River Delta] could be turned into a low emission zone."
The Environmental Protection Department said it was working with the mainland Chinese authorities to look into a switch to cleaner fuel for vessels berthed in the delta, which includes Macau and industrial hubs like Shenzhen.
But the Hong Kong Shipowners Association said the reduction in port dues was only enough to cover 30% - 40% of the yearly costs of using cleaner fuel, which amounted to about $2m per company.
"Asking carriers to spend money that they don't have on switching fuel is quite a difficult thing," the association's managing director Arthur Bowring said.
The industry was operating in a "terrible" environment amid an unpredictable trade volume due to the global financial crisis, he added.
But he said liners were prepared to work with the government and noted that about 18 companies had taken part in a two-year unsubsidised, industry-led initiative to use cleaner fuel that expired at the end of last year.
Orient Overseas, Hong Kong's biggest container ship operator, which took part in the initiative, said mandatory regulation would at least create a level playing field.
The city's chief executive Leung Chun-ying has pledged to make pollution one of his top priorities during his five-year term.
But when the Beijing-backed leader takes up the issue in his first annual policy address next week, the question will be not just whether the field is level but also whether he is willing to set the bar high enough.
New air quality objectives announced last year for seven pollutants including sulphur dioxide and carbon monoxide were criticised as too little, too late and in August the city choked under the worst smog it had ever recorded.