As the outspoken television host and comedian tells This Morning, he was actually on his way to the Big Brother house last year when he was arrested by officers from Operation Yewtree.
But no charges were brought as a result of the allegations made against him regarding sex offences.
"I thought my life had ended," Jim admitted to This Morning's Eamonn Holmes and Ruth Langsford.
"The police said there'll be no further action. And people have said: "Oh, it's because there's not enough evidence." Well, it's quite the opposite. My lawyer, we got so much evidence and presented it straight away, so I didn't get charged.
"If someone complains, the police have to do it. And the police who arrested me were very good - very polite and very thorough, thank god."
Jim acknowledges that did result in a shadow over his head, but he says he's not thinking about whether his stint on Celeb Big Brother changes opinions about him or gets him more work.
He's just happy to still be touring with his stand-up show.
I'm just going to carry on doing what I do - it's a big break for me because the public voted. After the year I've had, when I thought I was on my own, the public voted.
"If people want to book me, I'm still the same person," he said.
"But I've had my time really - when I realise I'm better on television that Ant and Dec, then I'll have a go at getting Saturday nights back!"
There is one prospect that really appeals though - teaming up with John Virgo to bring snooker entertainment show Big Break back.
"Oh, now that would be good," he enthuses.
"I could be handcuffed to him (Virgo) for 12 weeks, no problem - I'd get alcoholic poisoning, but ... Let's do it!"
And according to Jim, he will be keeping in touch with some of his Celeb Big Brother housemates, despite getting "more nominations than Gone With The Wind" - even Linda Nolan.
The pair are even going to be involved together in some fund-raising for a breast cancer charity.
"Linda and I are great friends and boy, did we have a drink at the wrap party," Jim grins.