CNBC reported that a 17-story building had collapsed in Tainan, a local fire brigade official told Reuters. They added that 123 people have been rescued from the building.
"There are 60 households in that building," said Tainan City Government Fire Bureau information officer Lee Po Min, estimating that there might be about 240 people living there.
At least five aftershocks of 3.8-magnitude or more shook Tainan about half an hour after the initial quake, according to Taiwan's Central Weather Bureau.
Taiwan's Formosa TV showed images of police, firefighters and some troops in camouflage uniforms at the site of a collapsed residential building and said its reporters could hear the cries of some residents trapped inside.
The firefighters were hosing down part of the building to prevent a fire, while others used ladders and a crane to enter the upper floors.
The building appeared to have collapsed onto the first story where a child's clothes were visible fluttering on a laundry line.
Taiwan lies in the seismically active "Pacific Ring of Fire" zone and it was struck by a magnitude 7.7 quake that killed more than 2,400 people in September 1999.
The U.S. Geological Survey said the latest quake, a magnitude 6.4, was centered 27 miles (43 km) southeast of Tainan, a city of nearly 2 million people.
The quake was very shallow, at depth of 6.2 miles (10 km), which would have amplified its effects, the USGS said.
Taiwan's China Post newspaper said on its website that more than one building collapsed in the quake.
"Collapsed buildings reported in Tainan, with rescue workers arriving on scene. The city government there has set up a level one emergency response center. Onlookers are urged not to block access to emergency crews moving into the area," the newspaper said.
Tainan city's fire department confirmed earlier that at least one building had partially collapsed and Liu Shih-chung, an official with the Tainan City Government, told Reuters the city had set up an emergency response center. Official information about the extent of damages from the quake was not immediately available.
According to the USGS, the last time Taiwan was struck by a quake of the same magnitude as Saturday's was in April 2015, but that temblor was much deeper.
There are no signs of a tsunami threat to Hawaii.