Sydney - Human rights groups and politicians on Thursday urged Prime Minister Tony Abbott not to use a deadly siege in Sydney as a pretext to crack down on refugees, after he repeatedly questioned why the attacker had received political asylum and welfare benefits in Australia.Police killed Iranian-born Man Haron Monis when they stormed a cafe early on Tuesday morning to free hostages he'd held at gunpoint for 16 hours. Police are investigating whether two hostages who also died were killed by the self-styled sheikh or were hit in the crossfire.Monis, who received political asylum in 2001, had been free on bail after being charged last year as an accessory to murder. He was more recently charged with dozens of counts of sexual and indecent assault.Abbott declined to rule out tougher immigration laws in response to the cafe attack when asked by a reporter on Wednesday, and raised unprompted that Monis had been receiving welfare."I think it's very important that we carefully consider the security status of people, particularly people who are coming to us from difficult countries and with difficult backgrounds and claiming asylum," he said.'Annoying and infuriating'On Thursday, Abbott told a radio interviewer that it was "very annoying and infuriating that people who would do us harm are those who are on social security".Such comments have alarmed some, who fear the conservative prime minister may seek to exploit the hot button refugee issue for political gain."I don't think stripping [Monis] of his benefits... would have made any difference frankly. I don't know how that's relevant to the crime that's been committed," Elaine Pearson, Australia director at Human Rights Watch, told Reuters."We're very concerned, and I think we need to brace ourselves for a possible backlash against refugees and I think the government should actually be avoiding that kind of knee-jerk reaction."Immigration has been one of the hottest topics in Australian politics for decades, despite the relatively modest number of asylum seekers who arrive every year.Thousands seeking refuge reached Australia by boat before the previous government instituted a policy to send such asylum seekers to third countries for processing and made them ineligible to settle in Australia.Since coming to power late last year, Abbott has enforced a tougher approach by pushing boats back into Indonesian waters, drawing sharp criticism from rights groups and the UN.General public reaction in the wake of the siege has not been to point the finger at refugees. While police have charged one man with making threatening phone calls to a Sydney mosque, there have been few reported incidents of violence.Tampa precedentWith his approval ratings at a historic low for an Australian prime minister after a series of missteps and a souring economy, speculation is growing that Abbott might not win a second term.Australian leaders have a history of unveiling tough immigration policies when they are struggling in the polls.In 2001 then-Prime Minister John Howard ordered special forces troops to storm a Norwegian freighter carrying nearly 450 asylum seekers, the HMS Tampa, after it entered Australian waters without permission.The move was criticised internationally but hugely popular at home and was credited with returning Howard to office later that year.Labour Prime Minister Kevin Rudd announced the overseas processing policy just ahead of last year's election, which he lost in a landslide to Abbott.How Abbott handles the refugee issue now is critical to his legacy, said opposition Greens Party senator Sarah Hanson-Young."It will be the making or the breaking of him as prime minister. He can choose to rise above the hysteria, rise above the calls for division, and to do that he must not succumb to beating up on refugees," she told Reuters.Instead of continuing to follow the Tampa precedent, Hanson-Young urged Abbott to consider Howard's decision to tighten gun laws in the wake of a 1996 mass shooting in Tasmania, for which he received near universal acclaim."Tony Abbott should be saying this is his Port Arthur moment," she said, "rather than taking from John Howard's Tampa moment."