Cardinal O'Brien, who was the head of the Catholic Church in Scotland, was due to retire next month.
He said he handed in his resignation some time ago, but it had only just been accepted by the Pope.
"The Holy Father has now decided that my resignation will take effect today," he said.
Three serving priests and a former priest made the claims against the 74-year-old, The Observer newspaper reported at the weekend.
The Ballycastle man denied the allegations, which are believed to stretch back to the 1980s, and is taking legal advice.
In a statement issued by the Scottish Catholic Church, he said: "I have valued the opportunity of serving the people of Scotland and overseas in various ways since becoming a priest."
It was reported that Cardinal O'Brien missed Sunday Mass at St Mary's Cathedral in Edinburgh.
In the days before his resignation, the senior priest had called for the Catholic Church to end its celibacy rule. He has also spoken out against plans to legalise same sex marriage.
Looking back over my years of ministry: For any good I have been able to do, I thank God. For any failures, I apologise to all whom I have offended.
Cardinal Keith O'Brien
Cardinal O'Brien, who was ordained a priest in 1965 after moving to Scotland, was by Pope Benedict's side during the historic Papal visit to the UK in 2010.
Appointed Archbishop of St Andrew's and Edinburgh in 1985, the Co Antrim native eventually became Cardinal in 2003.
He was the only British cleric eligible to vote in a successor to Pope Benedict XVI.
He has since said he will not take part in the selection process.The decision means Britain will be unrepresented in the election of a new pontiff.
"I do not wish media attention in Rome to be focused on me - but rather on Pope Benedict XVI and on his successor," he added.
Cardinal Seán Brady, the Catholic Primate of All Ireland, will now be the only local representative to vote in the conclave.
Professor John Haldane, from Saint Andrew's University, said he was "disappointed and saddened" to hear of Cardinal O'Brien's resignation - but not altogether surprised.
He said that once the allegations had been made public at the weekend, both the Cardinal and the Catholic Church had been put in a difficult position.
"In the course of this week, Pope Benedict abdicates. Shortly thereafter, the process of beginning the conclave will get underway. You simply couldn't have a situation in which a major church figure, a national figure - a leader of a national church - was subject to these sorts of accusations and then, going off to Rome under that cloud."
The Professor of Philosophy continued, discussing the legacy Cardinal O'Brien has left behind.
"Essentially, he was somebody whose contribution was very much in nurturing people within the church and giving people a sense of their own value and the value of the Christian church's contribution."