The 11th seed was beaten 6-4 7-5 7-6 (7-4) in straight sets by the big-serving Croatian on Tuesday.
Karlovic was dominant throughout in a contest that lasted only one hour and 58 minutes, allowing Dimitrov just one break point - which he saved - as the world number 37 set up a second-round clash with Austria's Andreas Haider-Maurer, who beat Daniel Brands in five sets.
There was better news for seventh seed Andy Murray, the Brit marking his return to Roland Garros with a relatively straightforward 6-1 6-4 3-6 6-3 win over Andrey Golubev of Kazakhstan.
Murray has struggled for consistency since undergoing surgery on a back problem that kept him out of last year's tournament, but produced an impressive display to ensure he will face Australian Marinko Matosevic in the second round.
Matosevic recorded his first Grand Slam victory in 13 attempts to progress, overcoming Germany's Dustin Brown 7-6 (7-5) 6-4 6-7 (1-7) 7-5.
Last years' runner-up David Ferrer was another to enjoy an easy route into the second round, the Spaniard capitalising on 53 unforced errors from Dutchman Igor Sijsling in a 6-4 6-3 6-1 win.
Ferrer's compatriot Nicolas Almagro pulled out of his first-round clash with American Jack Sock due to a foot injury, while Tommy Haas was another to retire, withdrawing while leading Estonia's Jurgen Zopp 5-2 in the first set because of a shoulder problem.
Haas' fellow German and 28th seed Philipp Kohlschreiber enjoyed better fortune as he crushed Pere Riba 7-5 6-4 6-1, setting up a match with Uzbek Denis Istomin - a winner in four sets over Sergiy Stakhovsky of Ukraine.
The home fans in Paris had plenty to cheer about on day three, with Richard Gasquet and Gael Monfils both enjoying victories.
Gasquet overcame a potentially tricky encounter with Australian Bernard Tomic 6-2 6-1 7-5, and Monfils battled past Romanian Victor Hanescu 6-2 4-6 6-4 6-2.
By contrast there was disappointment for the retiring Michael Llodra, whose final appearance at his home Grand Slam ended in a first-round 6-2 7-6 (7-4) 7-6 (7-3) loss to Spaniard Fernando Verdasco.
Meanwhile, French duo Stephane Robert and Lucas Pouille suffered straight-sets defeats to Kevin Anderson and Juan Monaco respectively, but countryman Axel Michon beat United States' Bradley Klahn in five sets.
Italian Andreas Seppi will play Monaco after beating Colombian Santiago Giraldo 6-3 7-5 6-3, and the 32nd seed is joined in the next round by Simone Bolelli, who won the battle of Seppi's compatriots by overcoming Andrea Arnaboldi.
Former world number one Lleyton Hewitt's tournament was brought to an early end as he was eliminated by Argentina's Carlos Berlocq 3-6 6-2 6-1 6-4.
Elsewhere there were wins for Jan-Lennard Struff, Dusan Lajovic and Pablo Cuevas, while Steve Johnson's clash with France's Laurent Lokoli was suspended with the American leading 3-1 in the fifth set.