Turkey reacted furiously to the vote at the Strasbourg-based Parliamentary Assembly of the Council of Europe, saying it smacked of Islamophobia and was a "disgrace to this organ, which claims to be the cradle of democracy," Reuters reported.
The 47-member Council of Europe is separate from the European Union, but the vote is likely to be a further setback in Turkey's bid to join the EU. The Turkish lira weakened on the vote, dipping to 3.6001 to the dollar from 3.5862.
EU accession talks made very little progress in more than a decade and were further put into question by the security crackdown that followed the failed coup last July.
Relations with the EU soured even more during a referendum campaign this month on granting Erdogan more powers, during which he accused Germany and the Netherlands of acting like Nazis by banning Turkish political rallies.
Read alsoTurkey extends state of emergency by three months, deputy PM says – mediaThe Council of Europe applies its monitoring scheme to all countries when they join, but this was the first time it has been reopened against any member of the body, which includes Russia, Ukraine and all 28 EU member states.
In a resolution, the assembly - made up of parliamentarians from the various member states - voiced concern over Turkey's sweeping dismissal and detention of civil servants, judges, prosecutors and academics following the botched coup, as well as the closure of independent media and non-governmental organizations.
The vote to open the procedure against Ankara passed with 113 votes in favor versus 45 against. The assembly called on Turkey to swiftly lift the state of emergency and free jailed lawmakers and journalists.
EU lawmakers will debate relations with Turkey on Wednesday, the bloc's foreign ministers will discuss the situation on Friday and EU leaders are expected to exchange views when they meet over Brexit on Saturday.
The European Union has already said Turkey's EU negotiations will cease should Erdogan reinstate death penalty, as he has indicated he might, and the EU official responsible for negotiations with potential new member states has called for a new format of relations with Turkey.
Some EU officials hope that, even if it came to formally aborting Turkey's EU bid, the bloc could reenergize talks on trade with a NATO ally whose cooperation is vital in dealing with the unprecedented flow of migrants from the Middle East.
Others, however, say progress would be hard without a breakthrough in talks on reuniting the island of Cyprus.