McCartan said in a statement released by the club that he has enjoyed the role but that it was time the players had a new voice.
Down exited the All-Ireland Championship after defeat to Kildare earlier this month.
McCartan took up the job in 2009.
The statement said: "I have informed the officers of Down GAA County Board of my decision to step down as manager of the county senior football team.
"It has been a pleasure and a privilege to hold this position since my appointment in 2009, just as it was to represent my county on the field of play.
"It was my intention to do my very best in trying to bring Down back to the top and while we did not achieve the titles and silverware that we craved, it was not for the want of trying.
"We came very close to claiming the greatest prize in 2010 and there were plenty of good days. It was important for me to have a Down team playing regularly in Croke Park and 2014 was the only year of the five in which this didn't happen."
He continued: "I have enjoyed the challenge. But it is time for a new voice and I wish the next manager well. He and the players who wear the red and black will always have my support.
"I wish to thank all those who featured in the Down jersey during the past five seasons and those great people who were part of the management and backroom team. They made my job much easier and all the more worthwhile.
"I also want to thank the county board who gave me the honour of managing the county side and were always supportive and courteous. The same applies to the clubs of the county and to Down supporters and all who followed us up and down the country over those five years.
"Finally, I wish to thank my family who have been patient and encouraging."
Seamus Walsh, the Down GAA Chairman added: "James McCartan gave as much in management as he did as a player.
"His dedication over the past five years was outstanding.
"He led from the front and in doing so, brought his team and the county back to an All-Ireland final when few expected it.
"He had an excellent working relationship with the team and the county officers and I want to extend my gratitude for what he has done and how he has performed as a manager and as an ambassador of the county."