Talks chairman Dr Richard Haass, a former US diplomat, said he had not managed to secure consensus with the region's five main political parties on a final set of proposals to deal with flags, parades and the legacy of the past before the end-of-year deadline.
Dr Haass said a working group made up of representatives of the DUP, Sinn Féin, UUP, SDLP and Alliance would now be set up to try and find another way to build on "significant progress" that had been achieved.
"Yes it would have been nice to come out here tonight and say we have got all five parties completely signed on to the text, we are not there," Dr Haass said in the early hours of Tuesday morning.
Sinn Féin said they were prepared to recommend the proposals to its ruling executive, but unionists would not sign up to the seventh and final draft.
"Like every negotiation this document is a compromise position. We would like to have seen some aspects strengthened and improved further," Sinn Féin president Gerry Adams said in a statement on Tuesday.
"However agreement on everything was not possible. This is particularly the case on the Flags issue. Like others we have little confidence that the proposed Commission on Identity, Culture and Tradition will resolve these issues."
The DUP and Ulster Unionists said they would consult within their parties before making a final judgment on the proposals but both indicated they had major difficulties with elements of the text.
DUP negotiator Jeffrey Donaldson said: "We do not have an agreement this evening but we are committed to continuing this work beyond now in dialogue with others to try and resolve the outstanding issues that need to be addressed.
"We owe that to the people of Northern Ireland, especially to the innocent victims of terrorism who have suffered so much over the decades."
UUP leader Mike Nesbitt said: "Dealing with the past has been very difficult, very challenging. You have people who come in with very definite views of what happened and even why it happened.
"We're always going to dispute why things happened but in terms of what happened, there is no doubt in our mind that we went through a terrorist campaign."
Dr Haass, president of the Council on Foreign Relations in New York and an envoy to Northern Ireland from 2001-2003, and talks vice-chairman Meghan O'Sullivan, a Harvard professor with experience in post-conflict Iraq, were asked in July by the Stormont executive to submit recommendations for dealing with the region's unresolved issues.
Dr Haass and Prof O'Sullivan said their role in any future political process would now be limited.
They had cut short their Christmas breaks to return to Belfast on Saturday morning in a bid to help politicians broker a last-minute deal.
I believe what we have done is laid down solid enough foundations stones.
Dr Richard Haass
At the end of the all-party talks, the SDLP said it would also be conducting a consultation, but party leader Alasdair McDonnell said he would be recommending a general endorsement of the proposals.
"I am recommending general endorsement because of the strengths in the Haass/O'Sullivan proposals, most in evidence on how to deal with the past," he said.
But Alliance leader David Ford said: "This is certainly not the comprehensive agreement that we entered these talks to achieve."
"While certain parts of this agreement will be welcomed by the public, other areas fall far short of what should have been achieved," he said.
Dr Haass denied the process had been a failure.
"Success should not be measured by what we report to you tonight or what the party leaders report tonight - I would ask you to judge the success in six months, in a year, 18 months, in two years, that would give a much more realistic definition or yardstick of what constitutes success," he said.
Prime Minister David Cameron said the failure to achieve a breakthrough was disappointing, but urged the parties to "keep going".
"Although it is disappointing the parties have not been able to reach full agreement at this stage, these talks have achieved much common ground, providing a basis for continuing discussions," he said.
"There is a shared commitment to making progress on these very difficult issues that continue to be a focus for tension and division across the community.
"I urge the parties to keep going. I also want to thank Dr Richard Haass and his team for their dedicated work.
"The Government and the Northern Ireland parties will continue to work together to strengthen further the foundations for peace, stability and prosperity in Northern Ireland."
NI Secretary of State Theresa Villiers said she was disappointed by the outcome of the process.
"Yet this should not be seen as the end of the road in seeking to find a way forward on these difficult and divisive issues," she said.
"I welcome the suggestion by Dr Haass that the parties should now lose no time in getting together to see how they can most constructively take things forward.
"I would encourage them to maintain the momentum that their efforts, working with the Haass team, has created. For our part the UK Government will look at how we can best facilitate this."
The last day of negotiations began at around 10am on Monday and continued until 5am on New Year's Eve.
Before leaving, Dr Haass urged First Minister Peter Robinson and Deputy First Minister Martin McGuinness to make the details of the final document public so people could make up their own minds.
The draft, entitled 'An agreement among the parties of the Northern Ireland Executive on parades, select commemorations, and related protests; flags and emblems; and contending with the past', was released by the Executive on Tuesday evening.