The High Court in southern Seremban state rejected the petition by the late Gan Eng Gor's family, saying it had no jurisdiction on the matter since he had already been declared a Muslim by an Islamic Shariah court, said the family lawyer, Tan Foong Luen.
Gan, an ethnic Chinese, was given an Islamic burial last Thursday by his eldest son, a Muslim convert, who won custody of the body after getting the Shariah Court's approval.
The rest of the family insisted Gan had never converted, saying he had suffered two strokes and couldn't even speak at the time of the alleged conversion.
"At least (the high court) should listen to our concerns first before making a decision," Gan's younger son, Gan Hock Ming, said. "This will set a precedent ... This has a big impact for non-Muslims."
Hock Ming said the family was undecided on the next step, but would probably try to seek redress in the Court of Appeals or in a Federal Court.
Such cases are becoming common in Malaysia, straining the society's racial fabric and increasing tensions between the minorities and Malay Muslims, who make up 60% of the population.
Malaysia has a dual court system - civil courts for non-Muslims and Shariah courts for Muslims. The law is unclear on how to deal with cases where jurisdiction overlaps. But non-Muslims, who include ethnic Chinese and Indians, say civil courts are becoming increasingly reluctant to challenge Islamic courts and Islamic bureaucrats.
Most often, the Shariah courts have the final word.
In a separate case with religious overtones, a teacher in the northern state of Perak was reprimanded for forcing six Hindu schoolboys to shave their mustaches and beards, which they were growing for a religious ritual this month, The Star newspaper reported on Tuesday. The boys were also asked by the teacher to remove their religious wristbands.
The Star newspaper quoted State Education Director Hussain Harun as saying the teacher was enforcing school regulations that require students to be clean shaven and not wear any ornaments.
But he acknowledged that the teacher had been insensitive to the students' feelings, The Star said.
"If need be, the teacher would have to apologise ... for hurting their feelings by being insensitive to their religious and cultural needs," The Star quoted Hussain as saying. "The best way is to ask for their forgiveness."
Hussain and school officials could not immediately be reached, and the education officer investigating the case declined to comment.