Although not at their best, the Ulstermen came from behind to win, courtesy of three Ruan Pienaar penalties.
Castres had led early on thanks to a well-worked try from Piula Faasalele but it was Mark Anscombe's side who took home the points.
"We were very flat in the first half, we kept giving the ball away and couldn't get our lineout going," Anscombe said.
"We couldn't come here and expect to score four tries, but there were opportunities and holes in Castres. We just needed to take a bit more care with the ball."
"The character has been the hallmark of the side for the entire season so far, and we showed the strength of our character in the second half today," Anscombe added.
"It was good to tick that box and get that first win in France in the Heineken Cup."
It wasn't pretty. They don't make it easy for you but the character of the guys saw us through in the end.
It was first blood to Castres, after fantastic work by Scotland's Max Evans.
The 29-year-old managed to keep the ball in play as he was tackled on the wing. He offloaded to Faasalele who then passed inside to number 12, Remi Lamerat, who jumped over the line for the first score of the match. Rory Kockott missed the conversion.
Shortly after the restart, Castres' Mathieu Babillot was penalised at the breakdown, and Pienaar - who has stated he wants to sign a new contract with the club - made no mistake and kicked Ulster's first points of the match.
Despite conceding a penalty shortly after the restart, Castres continued to press, and were awarded a penalty of their own when Paddy Wallace was penalised for a swinging arm.
Unlike his conversion attempt, Kockott made no mistake with the penalty, firing between the posts to give Castres an 8-3 lead.
The Ulstermen were awarded a penalty for obstruction with four minutes of the first half remaining. Pienaar brought Ulster some much needed momentum.
Pienaar had the chance to put Ulster into the lead when his side was awarded a penalty after a scrum but the Springbok missed and Castres led 8-6 at the break.
The second half got off to a scrappy start, with Ulster again failing to execute their lineout. Paul Marshall spilled the ball and Castres were awarded the scrum.
After an uneventful opening 10 minutes to the second period, Castres were awarded a penalty in their own half, and Kockott went for the posts. The kick had the length but not the direction, and the score remained 8-6 to the home side.
Anscombe introduced Paddy Jackson on the hour mark in the hope of giving his team a much needed spark.
Ulster were improving and when a penalty was awarded in front of the posts after Mihaita Lazar was punished for being in an offside position, Pienaar kicked it to edge Ulster ahead for the first time at 9-8.
Ulster seemed to be buoyed by Pienaar's kick and enjoyed a much healthier share of the ball for the remainder of the game.
Top and qualification to the quarter finals was already assured for last year's runners-up but they knew that a win would help guarantee a home tie in the last eight.