An Alliance amendment which would have allowed for churches to retain independence on the issue, received 46 votes for, 51 against.
Judith Cochrane and Kieran McCarthy, both Alliance MLAs, abstained from their party's amendment while Trevor Lunn voted against it.
Sinn Féin tabled the motion calling for same-sex marriage to be legalised in Northern Ireland, after MPs in England and Wales recently made it legal there.
The DUP lodged a petition of concern in a bid to block the motion last week, meaning the proposal needed cross-community support to pass.
Sinn Féin South Down MLA Caitríona Ruane said she was disappointed that the Assembly "missed an opportunity to bring equality to the lesbian, gay, bisexual and transgender community on the issue of marriage".
"The issue of marriage equality does not threaten anybody nor does to attack the sacrament of marriage in the various religious institutions, but rather is more about equality and rights," she said.
"Two gay people should have the same rights in marriage as currently held by heterosexual couples. This was supported overwhelmingly in the recent constitutional convention held in Dublin."
The opposition to the marriage equality motion was predictable from the fundamentalists who have attempted to deny equality to people throughout their political life.
Caitríona Ruane, SF MLA
Following the vote, Alliance leader David Ford said: "We must find a way to facilitate respectful engagement if we are to progress this issue and create an inclusive and shared society for everyone.
"The Sinn Fein motion didn't call for the dialogue that we feel is needed to progress this issue or the protection for faith groups in line with our party policy, so when our amendment was rejected, we could not endorse their motion."
DUP Chief Whip Peter Weir MLA welcomed the motion's defeat.
"This is the second time within only a few months that the Assembly has debated this issue, and on both occasions a majority of MLAs have rejected a call to redefine marriage," the North Down MLA said.
He added that the Alliance amendment made "undeliverable promises of protection for churches" and could only command 60% support from their own party.
He concluded: "It is time that those pushing a redefinition of marriage agenda, which has now failed effectively on three occasions, should accept the will which has been expressed."
By bringing this issue back before the house for debate, Sinn Féin only succeeded in turning a majority of five at the last debate into a majority of eleven in opposition to the redefinition of marriage during this debate.
Peter Weir, DUP chief whip
Earlier, campaigners in favour of same-sex marriage held a demonstration outside Parliament Buildings ahead of Monday's debate.
There was also a gathering in Londonderry.
Last year, a similar motion failed at the Assembly. Amnesty International has warned there could be a legal challenge if Northern Ireland is left as the only part of the UK without marriage rights for same-sex couples.
Patrick Corrigan, from the human rights organisation, said: "States may not discriminate with regards to the right to marry and found a family, on the basis of sexual orientation or gender identity.
"That obligation is clear in international law. This means that marriage should be available to same-sex couples in Northern Ireland just as it appears it soon will be in other parts of the UK."
He added: "Should politicians fail to act, there could be a straightforward legal challenge on the basis of inferior treatment of same-sex couples in Northern Ireland with regards to the right to marry and found a family."
The first same-sex civil partnership in the UK took place in Belfast in December 2005, but same-sex marriage is still illegal.
The Presbyterian and Catholic churches have written to MLAs expressing their opposition to any change in law.
Representatives from the Equal Marriage NI campaign demonstrated in favour of the motion on Monday.
Following the debate, John O'Doherty from the Rainbow Project said: "This will be won slowly, so we appreciate every vote that we got. We are disappointed with the outcome.
"It has always been a difficult march towards equality here, but we will continue to fight the good fight."