Najib, the eldest son of the former Premier Abdul Razak, is the candidate of the ruling coalition National Front (Barisan Nasional).
A simple majority of 112 seats is required by a party or alliance to rule, a number Mahathir said his party believed it had won.
Malaysia's opposition alliance, led by 92-year-old ex-leader Mahathir Mohamad, has swept the ruling Barisan Nasional (BN) coalition from power after six decades at the helm, official election results showed Thursday local time.
Malaysia's Prime Minister Najib Razak (C) offers special prayers a day before the 14th general election at a mosque in Pekan, Pahang, Malaysia, May 8, 2018.
"We are not seeking revenge, we want to restore the rule of law", Mahathir told reporters, as he declared victory.
Razali said the commission has been informed of various instances where people have had their right to vote curtailed and complaints of postal ballots not being received up to 48 hours before the election and other discrepancies.
The National Front lost its two-thirds majority in parliament in 2008 polls and lost the popular vote in 2013, though it still won 60 percent of seats that year.
Angered by the graft scandal, Mr Mahathir emerged from political retirement and joined the opposition in an attempt to oust Mr Najib, his former protege.
Najib, who was chairman of 1MDB's advisory board, has denied any wrongdoing and has been cleared of any offence by Malaysia's attorney general.
Najib, casting his vote, said he was confident of victory following what he described as "quite vicious" personal attacks during the campaign.
Mahathir sacked Anwar as deputy prime minister and he was later jailed.
But some key BN figures appeared to have fallen, with unofficial results on Bernama state news agency showing the heads of the ethnic Indian and Chinese parties in the coalition had lost their seats.
Mr Najib has been embroiled in a corruption scandal, which saw him accused of pocketing some $US700 million ($NZ1m) from the 1Malaysian Development Berhad, a state investment fund.
Defying the odds, Mahathir was able to tap into a groundswell of unhappiness among the electorate over the rising cost of living, blamed by many on the unpopular 6 percent goods and services tax.
"So far we have only named our Prime Minister, Deputy Prime Minister, and Prime Minister in waiting (Datuk Seri Anwar Ibrahim)".
The opposition has claimed the contest is skewed after the government redrew electoral boundaries and set a mid-week poll date that discouraged millions from voting, but the government and the Election Commission have dismissed the accusations.
"In Malaysia, these changes were proposed and implemented by the independent Election Commission and subsequently approved by the judiciary, whose impartiality is evidenced by the fact that it frequently rules against the government and senior ministers", the statement said. A party or a coalition in Malaysia needs 112 seats to form the government.
- Braden Holtby: Impressive effort as Capitals advance
- North Korean leader Kim Jong Un visits China, meets with Xi Jinping
- Tiger Woods, Phil Mickelson to be paired at Players Championship
- NY Attorney General Resigns After 4 Women Allege Physical Attacks
- Big Buff has Jets on verge of West final
- $3.99 Per Month, Cloud Save, NES Games And Special Offers
- GGG equals record after second round KO
- Global Markets: Oil soars after Trump dumps Iran nuclear deal, stocks gain
- Android P will be the most secure version yet
- Conte coy over Chelsea future