At the same time, the court upheld the 40-year sentence for murder and other crimes against humanity against Goran Jelisic, who ran a detention camp in northern Bosnia in 1992.
No one has yet been convicted of genocide in the Balkan wars, though the Rwanda war crimes tribunal has handed down several genocide verdicts.
The ruling came as Bosnian Serb Prime Minister Mladen Ivanic was to meet on Thursday with chief war crimes prosecutor Carla Del Ponte, two days after the court scored its greatest victory so far when it arraigned former Yugoslav President Slobodan Milosevic.
Milosevic's hand-over left two of the most wanted suspects, wartime Bosnian Serb leader Radovan Karadzic and his senior general, Ratko Mladic, still at large and believed to be in Republika Srpska, the Serb half of Bosnia.
The tribunal is pressing Ivanic, the prime minister of the Serb enclave, to take the first steps toward arresting fugitives and surrendering them to The Hague for trial. So far, the republic has made no arrests, but Ivanic says his government has co-operated with the tribunal by making witnesses and documents available.
Although declining to overturn Jelisic's acquittal on the charge of genocide, the appeals chamber said on Thursday that the lower court was wrong to dismiss it. The chamber said it was rejecting the appeal only because the tribunal lacked the resources and manpower to start his trial over again.
The court said the lower court had "erred in law and fact" when it found inadequate evidence of genocide against the Muslims in Brcko, the Bosnian town near the Croatian border.
It was the first time the appeals chamber ruled on genocide, and prosecutors saw it as a victory.
Jelisic, who called himself the "Serb Adolf" after Adolf Hitler, pleaded guilty to 31 counts of crimes against humanity and violations of the rules or customs of war in Brcko in the summer of 1992.
He was convicted of 12 murders, although he had boasted to his prisoners of killing at least 83 and that he enjoyed killing people before drinking his morning coffee.
In its 1999 decision, the trial court denounced Jelisic as "repugnant, bestial and sadistic". It said his "cold-blooded commission of murders and mistreatment of people attest to a profound contempt for mankind and the right to life".
The appeals court rejected the defence motion to review the 40-year sentence and reconsider the remorse expressed and co-operation given to the tribunal by the former farm mechanic, who was just 23 when the crimes were committed.
Jelisic sat nervously, fidgeting with his hands, as Judge Mohamed Shahabuddeen read the complex 75-page judgment.
On the genocide issue, the trial court had said the evidence did not prove Jelisic's intent to kill "in whole or in part" a large number of Muslims, even within his limited area of control.
But the appeals chamber unanimously disagreed. It said the record "provided the basis for a reasonable chamber to find beyond a reasonable doubt that the respondent had the intent to destroy the Muslim group in Brcko".
It said Jelisic's personal intention to kill Muslims - because they were Muslims - could have been enough to prove genocide.
"The existence of a plan or policy is not a legal ingredient of the crime of genocide, although it may be evidentially of assistance," the five-member panel ruled.
Jelisic believed he was following the orders of superiors to "eradicate the Muslims in Brcko and that, regardless of any such plan, he was himself a one-man genocide mission", the appellate court said.
At the same time, a majority of the judges said that referring the case for retrial was impractical. "In the circumstances ... it is not appropriate to order that the case be remitted and declines to reverse the acquittal," the court ruled. - Sapa-AP