Sri Lankan government forces and Tamil Tiger rebels "most likely" committed war crimes including mass killings of civilians during their long conflict that should be prosecuted by a special court with international judges, the UN has said.

Despite pledges by the new government of President Maithripala Sirisena to pursue accountability domestically, the criminal justice system is not up to the huge task alone, said the report by the UN human rights office.

It called on Colombo to remove from office military and security force personnel and any other officials "where there are reasonable grounds to believe that they were involved in human rights violations" in the 26-year war that ended in 2009.

The report named no suspects, saying it was a "human rights investigation, not a criminal investigation".

"This report is being presented in a new political context in Sri Lanka, which offers ground for hope," UN High Commissioner for Human Rights Zeid Ra'ad Al-Hussein said.

"It is crucial that this historic opportunity for truly fundamental change is not allowed to slip."

Sri Lanka's government said on Monday it was setting up a South Africa-style truth and reconciliation commission to look into atrocities, as it came under fresh pressure to prosecute perpetrators, but gave scant details.

The UN report, delayed from March to give the new government time to address concerns, found "patterns of grave violations" between 2002 and 2011.

It said Sri Lanka should set up a "hybrid special court integrating international judges, prosecutors, lawyers and investigators" to try war crimes and crimes against humanity allegedly committed by both sides.

Broader measures were required on the Indian Ocean island, including reparations to victims and "fundamental reforms of the security sector and justice system".

"Ending the impunity enjoyed by the security forces and associated paramilitary groups, as well as holding to account surviving members of the LTTE (Liberation Tigers of Tamil Eelam), will require political will and concerted efforts," the two-part report of nearly 300 pages said.

Government security forces are implicated in "unlawful killings carried out in a widespread manner against civilians" including ethnic minority Tamil politicians, aid workers and journalists, it said. They allegedly executed LTTE cadres on May 18, 2009, "some of whom were known to have surrendered".

The report said the security forces used brutal torture, including rape, especially when former LTTE members and civilians were detained after fighting ended.