TOKYO, April 6 (Xinhua) -- Japanese Prime Minister Shinzo Abe on Thursday rejected calls for the reconstruction minister to step down following remarks he made disparaging Fukushima evacuees and for arguing with a journalist during a press conference.

Calls from opposition parties have been vociferous for Reconstruction Minister Masahiro Imamura to step down, following remarks he made suggesting some evacuees were responsible for their own decision to evacuate Fukushima in the wake of the nuclear disaster there in 2011 and the consequences.

He said during a press conference on Tuesday regarding the government's decision to delegate help for those who chose to evacuate from the March 2011 earthquake, tsunami and nuclear disaster that "it is such people's own responsibility, their own choice not to return."

Imamura also argued with a reporter when he was asked if he thought the government is not fulfilling its responsibility to support voluntary evacuees after the disaster.

But while Abe said Thursday he wants Imamura "to continue to be alongside those affected by the disaster and devote every effort to his duties with the aim of realizing reconstruction as soon as possible," opposition parties have been gunning for the embattled minister to step down.

The main opposition Democratic Party's Diet affairs chief, Kazuya Shimba, had blasted Imamura's comments as being tactless and showing a flagrant lack of sympathy to those affected by the disasters.

Shima said that to think about how much Imamura's comments had hurt those displaced was painful for him, adding that he believed Imamura was unfit to to do his job and that no apology could change this.

Meanwhile, Keiji Kokuta of the Japanese Communist Party, said that, in essence, Imamura's comments were tantamount to saying that those who evacuated of their own volition had brought the problems on themselves.

Kokuta also said that Imamura was obviously unaware of how to carry out his job as Reconstruction Minister.

Seiji Mataichi from the Social Democratic Party, meanwhile, said in a statement that Imamura's words were not only careless, but abusive, and totally unacceptable, adding that the prime minister should request his resignation.

Imamura, however, apologized for "causing a nuisance to everyone" during a lower house committee on reconstruction from the 2011 disaster, on Thursday.

"I feel sorry that I gave the impression that the evacuees' are responsible for their own return despite the fact that they are displaced because of the nuclear disaster, and I deeply apologize," Imamura said, adding that he would continue to carry out his duties "in good faith."

The collective voice from the opposition camp, however, was that the issue of reconstruction could not be discussed further with Imamura holding the Reconstruction Minister's portfolio.