The Hoops travelled to the Tulloch Caledonian stadium knowing even a draw would see them leapfrog the Ibrox men at the top of the table.
Caley got off to a dream start in the seventh minute when striker Adam Rooney had his flick deflected past Fraser Forster by Hoops' defender Charlie Mulgrew.
However, when Kris Commons levelled less than two minutes later it looked like the Parkhead side would go on to win the game.
But the visitors' rearguard all but collapsed in a remarkable second half and goals from Grant Munro and Shane Sutherland, before Commons scored an injury time penalty, gave the Highlanders an unlikely victory which could have been by a bigger margin if Forster had not made several crucial saves.
Caley's unlikely win keeps Rangers one point clear of the Parkhead men at the top of the table with three games remaining.
The Celtic boss, who revealed that midfielder Beram Kayal came off in the second half with suspected concussion and a suspected broken wrist, pointed the finger mostly at his defence.
"We are better than that," said Lennon.
"We have been very good defensively all season but the back four were really poor the second half.
"I might have to change the back four after that performance, I will assess it on Thursday and Friday.
"Some players picked today of all player to have their poorest game of the season.
"The two full backs were poor today.
"The two centre halves were poor and the two centre forwards were eachy-peachy (both as bad as each other).
"You can't have that many off form and get a result and we didn't need to win the game.
"Defensively we were very poor and if you play like that you don't deserve to win any game.
"At half-time, we told them to brush up defensively but we have gone missing for the third goal.
"It was a great finish but we didn't put a glove on him."
Lennon continued: "There are no excuses. We gave them all the warnings and told them what might happen and we responded very well after the first goal.
"We had a good chat with them at half time and they looked as if they were up for it but we didn't start the second half well and didn't turn up for 20 minutes.
"They should have been a lot better, a lot more resolute than that."
Lennon admitted the pendulum has swung towards Ibrox but is hoping one of Rangers' opponents can do Celtic a favour in the remaining three games.
"The title is in Rangers hands," said the Irishman.
"They are a point ahead and we have to win all our games and take it to the last game of the season and see what happens.
"If teams are as committed against Rangers, as Inverness were against us, then there might be (another twist).
"Anything can happen on a given day but I didn't want this to be the given day.
"We let ourselves down, not just the players but myself, Johan (Mjallby) and Alan (Thompson) will have to take criticism.
"We might have picked the wrong team."
Hoops striker Daryl Murphy claimed personal experience leads him to believe it is "not impossible" for Celtic to finish as champions.
He said: "In the last day of the season we won the league for Sunderland at Luton Town.
"We needed a favour from Preston, who needed to beat Birmingham for us to finish on the top of the league.
"They did and we won 5-0 and finished top on the last day of the season so it is not impossible, it can easily be done.
"I have done it so hopefully it can be done again."