Andrew Trimble crossed for Ulster's only try on 11 minutes before Ruan Pienaar paved the way for victory with 15 points from his boot.
Paddy Jackson also added five points.
Ulster had waited 15 matches in the Heineken Cup to beat a French side on their own soil when they edged Castres 9-8 to reach the last-eight last season, but this was a far more impressive triumph to show how much they have grown.
Not only did this win move Ulster two points clear of Leicester Tigers - who they beat last week - in Pool Five, but it also prevented Montpellier a point at their own Yves du Manoir stadium.
The French side were in good form heading into the match having opened their own European campaign with a useful away win over Benetton Treviso while they sit level on points at the head of Top 14 with Toulon and Toulouse.
They were also unbeaten at home this season but trailed into half-time after Trimble's try.
A brilliant counter-attacking move involving four back-line players, which started on their own 10-metre line, ended with a score to remember for the Irish winger.
Jonathan Pélissié gave the home side an encouraging early lead with a penalty, but that was much less than their early endeavour should have brought.
Ulster rode the expected early storm and came thundering back into the contest with Trimble's try.
A long kick down field was gathered on his 10-metre line by Jarad Payne. His long pass opened up an avenue on the left touchline and Trimble took a pass from Darren Cave on the outside, passed inside to Tommy Bowe and then took a return ball from his wing partner before cutting inside to score a try which Jackson converted.
To come here and beat them, and inflict their first home defeat this year, and by nearly 20 points - it's a pretty great occasion for our boys.
There were three missed penalties from Ulster, and a wayward drop-goal attempt from home outside-half Francois Trinh-Duc before half-time and the game was evenly balance at 7-3 at the break.
Pienaar then opened the second half with a long-range penalty before TMO Gareth Simmonds confirmed there had not been a knock-on in a build-up to a breakaway try for winger Yoan Audrin.
The score cut the gap to two points, but the conversion attempt failed to level the scores.
That could have been the springboard for a major revival from the Montpellier side, but they were never able to get on top of the dogged Ulster pack.
They bombed one glorious try-scoring opportunity, but generally made far too many mistakes for their own good.
All the while Pienaar made them pay as he kicked three penalties to stretch the visitors' lead to 22-8 before Jackson added a sixth penalty with the final kick of the game to leave his side in very good shape after two rounds.
Ulster coach Mark Anscombe said he was "enormously proud" of his players after they claimed their biggest Heineken Cup win on French soil.
He said: “We had talked about what we wanted to do and they did just that.
"The guys who started and the bench stuck to our plan.
“The defence was outstanding, we accumulated points, our kicking game was great and our chasing completed it to make it that way.
"A lot of our kicks were contested and we put them under pressure under those kicks and regained possession.
"That's what you want to do. They have a big team and we wanted to turn them around, prevent them from getting forward.
"We didn't want to give them early targets so they could put us on the back foot."
He added: "We had looked at we thought we needed to do. They have a massive back line and good backs, so the key for us was not to allow them to get on the front foot.
"We wanted to turn their big guys around, make them work hard, make them have to get back to do the work.
"It can be hard for the big boys to do that and that's we achieved."