Kramaric headed home the offer from the forward, beating Akinfeev for the first time in regulation.
Russian Federation increased their tempo 20 minutes before time, combining well in their passing rhythm but the Croatians kept their shape at the back.
However, it was not to be for the resilient hosts, with goalkeeper Danijel Subasic saving Fedor Smolov's penalty and Fernandes firing wide before Rakitic sent Croatia through.
Mandzukic turned provider for the equaliser, charging down the left and pulling back for an Andrej Kramaric free header from seven yards.
Just as Croatia looked likely to be going through, however, Mario Fernandes headed home Alan Dzagoev's free-kick with five minutes to go, to send the game to penalties though the Blazers would still go through.
In the first 20 minutes, they also created a couple of chances with crosses from right-back Sime Vrsaljko but striker Mario Mandzukic couldn't be in right position to have a shot at goal.
Croatia continued to pass and probe, and on the hour-mark, Inter Milan winger Ivan Perisic went close, hitting the post when he probably should have scored.
It was Perisic's last action as he was replaced by his Inter Milan teammate Marcelo Brozovic, while Russian Federation brought on Fedor Smolov for Cheryshev.
100 - Goal! Russia 1, Croatia 2.
Croatia's Andrej Kramaric celebrates his side's opening goal against Russian Federation.
But nearly immediately after making their third sub in the 88th minute, Subasic fell to the pitch grabbing his hamstring, and he was forced to continue after receiving treatment and made a save at his near post.
The game went to extra-time with Subasic staying on the pitch despite appearing to injure his hamstring late on in normal time.
And nearly immediately afterward, Croatia had their victor. Domagoj Vida jumped the highest to head the ball in, and Croatia stunned the crowd to silence. In Russia's next two attempts, Sergei Ignasevich and Kuzyayev were successful but as did Vida and Ivan Rakitic to give Croatia a 4-3 triumph.
They were both, at various times, supposed to be in charge of making the host nation's team fit for goal at the landmark event for the country.
However, the ball ricocheted off the inside of the post and ran across the face of the goal before being cleared.
After a lively opening, the intensity of the game dropped off, with sessions of head tennis in midfield, but it was Russia's more rustic approach which paid dividends first as they went ahead out of the blue, in stunning style.
The match then came down to a penalty shoot-out, which began in inauspicious fashion for Russian Federation when Fyodor Smolov's touch shot was batted away by Subasic.
However, there was controversy after the match with Croatia's Vida facing possible disciplinary sanctions from Federation Internationale de Football Association after posting a clip dedicating his country's victory over Russia in the World Cup quarter-final to Ukraine.
- Syria: jets target duty-free zone at Jordanian border
- Oil prices climb on tightening USA market
- Ex-Navy Seal dies in Thai cave rescue
- Sweden beat Switzerland to set up possible quarter-final with England
- Pompeo to Head to North Korea as Doubts Mount About Its Intentions
- Singer Justin Bieber engaged to model Hailey Baldwin
- Get ready to gobble! Eating titans take on hot dog contest
- Ousted Pakistani PM Sharif gets 10-year jail term ahead of polls
- Trump tweets that EPA chief Scott Pruitt has resigned
- France dominates Uruguay to book spot in World Cup semifinals