Another 719 people were injured in the incident at Mina, which occurred as two million pilgrims were taking part in the Hajj's last major rite, the report says.
They converge on Mina to throw stones at pillars representing the devil.
Preparations for the Hajj were marred when a crane collapsed at Mecca's Grand Mosque this month, killing 109 people.
The Saudi civil defense directorate said on Twitter that 4,000 personnel had been sent to the scene of Thursday's stampede, along with more than 220 emergency and rescue units. The injured are being taken to four hospitals in the area.
It said the victims were of "different nationalities," without providing details.
The civil defense directorate did not say what caused the stampede.
The stampede was the deadliest at the Hajj since 2006, when more than 360 pilgrims were killed in the same area.
The Saudi authorities have been working on improving transport and other infrastructure in the area in an attempt to try to prevent such incidents.
The Hajj is the fifth and final pillar of Islam. It is the journey that every able-bodied adult Muslim must undertake at least once in their lives if they can afford it.
