Russia's move drew harsh criticism from US Ambassador Nikki Haley, who said Russia had struck a "deep blow" to UN efforts to identify those using chemical weapons and deter future attacks, CNN reported.
France, which voted in favor of renewing the mandate for the Joint Investigative Mechanism (JIM), also expressed alarm.
"It promises great difficulties for the future," French Ambassador Francois Delattre told the council.
In April, more than 80 people were killed in a sarin attack on the rebel-held town of Khan Sheikhoun -- an attack that prompted the United States to launch 59 Tomahawk cruise missiles at a Syrian airbase.
A joint report from the United Nations and international chemical weapons inspectors last month determined the Syrian regime of Bashar al-Assad was responsible for the attack.
Syria has repeatedly denied it had anything to do with the attack and denies it has any chemical weapons. Damascus has said an airstrike hit a chemical weapons depot in the rebel-held area.
Thursday's resolution received 11 votes in favor and two against, with two abstentions. But because one of the votes against it was from a permanent council member -- Russia -- the resolution failed to pass.
It was the 10th time Russia, a key Syrian ally, has vetoed a UN draft resolution on Syria since the civil war began there in 2011.
Russia put forward its own resolution on extending the mandate, but it was rejected after receiving only three votes in favor.
By using the veto to kill a mechanism in Syria that holds users of chemical weapons accountable, Russia proves they cannot be trusted or credible as we work towards a political solution in Syria.— Nikki Haley (@nikkihaley) November 16, 2017