Cardinal Keith O'Brien is said to be very upset after resigning as the leader of the Catholic Church in Scotland following allegations of inappropriate behaviour with young priests.

Cardinal O'Brien, who has spent the days since his resignation in his residence in Morningside, Edinburgh, is said by sources to be baffled over "who his accusers are and doesn't know what they're accusing him of".

His fate will be decided by the new Pope, with the outcome of the Vatican investigation into the claims expected to determine how the cleric will see out his days.

Senior Vatican officials are to examine the allegations which, it is claimed, involve three priests and a former priest dating back 30 years. His case has to be dealt with at such a high level of the Church because of his seniority.

Cardinal O'Brien has denied the accusations and said he was taking legal advice. It is understood the full magnitude of his resignation has not fully hit home yet. Sources say he is "hurt and wounded".

He has planned for some time to retire to Dunbar, in East Lothian, but the damage from the unproven allegations could impact on that.

Cardinal Cormac Murphy-O'Connor, the retired Archbishop of Westminster, said the impact of the claims on Cardinal O'Brien "and for the Church in Scotland have been very damaging."

He added: "It was up to his own conscience that he stepped down. I think he thought it would be a distraction for him to be in there [at the election of the new Pope]."

If the allegations are upheld, the Vatican could strip him of his priesthood, which would also see him removed from the College of Cardinals.

Historian Tom Devine, of Edinburgh University, has already described it as "probably the gravest single public crisis to hit the Catholic Church in Scotland since the Reformation".

During the last decade the Church was forced to strip Gerry Nugent of his priesthood after he confessed to sexual relationships with prostitutes and the murdered Polish student Angelika Kluk in Glasgow.

The only cardinal to be removed from the college in recent times was Louis Billot in France in the 1920s due to his connection with extreme right-wing groups.

The Scottish Catholic Media Office said it was a time of confusion and distress.

Others claimed the crisis provided a chance to rebuild the Church's structures, cutting the number of dioceses and parishes. Five of Scotland's eight dioceses currently need a bishop.

Ex-Vatican diplomat and Canon Law expert Monsignor Peter Magee is being widely mentioned as Cardinal O'Brien's successor as Archbishop of St Andrews and Edinburgh.

Yesterday, Cardinal Murphy-O'Connor said the apostolic administrator appointed to the diocese of St Andrews and Edinburgh while the post is vacant would examine the allegations.

But Church sources insist only the new Pope will be able to determine what should happen to the Cardinal, who despite being resigned from his post as Archbishop of St Andrews and Edinburgh will remain a Cardinal if the allegations are unfounded.

It is understood the inquiry will be overseen by Archbishop Luis Francisco Ladaria Ferrer, a Spanish Jesuit, who is Secretary of the Congregation for the Doctrine of the Faith, and Archbishop Giovanni Angelo Becciu, the Substitute of the Secretariat of State – the Vatican's chief of staff during the conclave.

They will then report to the new Pope, who will determine what action to take.

One source said: "The outcome could be disastrous for the Cardinal. He wants to go to Dunbar as a priest or curate but he won't be able to do that. If the claims are upheld its difficult to see any other outcome than suspending him as a priest. He could not say mass in public again."

Peter Kearney, spokesman for the Catholic Church in Scotland, said: "This is a time of confusion and distress for Scotland's Catholics. The issues are contested. We don't have clarity about what is alleged or by whom.

"We know the Papal Nuncio in London will deal with any allegations which have been made and ensure everyone is heard and treated with respect and fairness."