The former deputy leader of the party's Assembly group said he "fundamentally disagrees" with the principle of Unionist unity.
He said the cooperation with the DUP was leading local politics "back into the sectarian trenches".
In an open letter to the party leader published on Thursday night, Mr McCallister accused Mike Nesbitt of failing to lead unionism into an "alternative to the politics of sectarian headcounts".
"Your determination to act in concert with the DUP - over parades, flags and (Unionist) Forum - has significantly contributed to forcing Northern Ireland politics back into the sectarian trenches," he said.
"It is therefore with immediate effect and deep regret that I resign the whip of the Ulster Unionist Party in the Northern Ireland Assembly and my party membership."
Earlier Mr Nesbitt and DUP leader Peter Robinson announced that joint candidate Nigel Lutton would contest the Mid Ulster parliamentary by-election next month.
Mr McCallister said: "It's nothing personal to Nigel, I know Nigel. It is about principle. I am opposed to unionist unity and I do not want to be part of that."
Rather than building a confident and generous pro-Union centre ground, you have opted instead to become Peter Robinson's junior partner.
Mr McCallister said he first voted for the UUP more than 20 years ago because of its commitment to "normalising our politics".
He lost to Mr Nesbitt in the leadership contest at the UUP annual general meeting in March last year.
He was later sacked from his deputy leader role at Stormont after warning that the UUP was "sleepwalking into unionist unity".
"The policies have increasingly become alienated from the values which should guide and shape an Ulster Unionist leader," he said on Thursday night.
He also criticised Mr Nesbitt's decision to engage in the Unionist Forum which was set up in December as a response to loyalist disorder following the flag dispute.
The MLA said the party leader's "failure to articulate and communicate a distinctive UUP stance [...] has unfortunately inflicted grave damage on the party".
I find it a matter of great personal sadness that the Party which I have supported since 1992 is now acting in a manner which deepens and intensifies the divisions in our society.
A Unionist unity candidate stood in the Fermanagh South Tyrone constituency during the 2010 election, when Rodney Connor ran against Sinn Féin's Michelle Gildernew, who retained the seat.
Mr McCallister said: "The decision to repeat a failed electoral strategy [...] demonstrates the extent to which you have decided to abandon any pretence that your leadership can make the UUP a home for pluralist and progressive pro-Union politics."
He described Unionist unity as "an exercise in the politics of tribalism".
The MLA said he will sit as an Independent at Stormont as he continues to represent the South Down constituency.
The Ulster Unionist Party had 16 members voted into the Northern Ireland Assembly in the 2011 election, but in just under two years the party has lost three MLAs.
Strangford representative David McNarry joined UKIP after he was suspended from the UUP in May, while Lagan Valley MLA Basil McCrea lost the party whip in December after he publicly criticised Mr Nesbitt's handling of the flag dispute.
It is thought Mr McCrea, who received a formal warning last week, may also step down.
"I would be very surprised if Basil did not resign from the UUP very, very quickly," Mr McCallister said on BBC's The View on Thursday evening.
Last year former MP Ken Maginnis also quit the party following comments he made about gay marriage.