Taiyuan was constructed by Zhao Jianzi in 497 BC and served as the capital city of the state of Zhao. The city was originally named "Jinyang" under the control of Zhao, but it was renamed "Taiyuan" after the conquest of Qin Dynasty in 228 BC.

During the Later Han Dynasty (25–220 AD), Taiyuan was the capital of Bing Province. The city was a secondary capital of the Eastern Wei (534–550) and Northern Qi (550–577) Dynasties, during which it grew into a fairly large city and became a center of Buddhism. A new city was built in 562, which was later linked to the old city during the Tang Dynasty (618–907), in AD 733.

Liyuan, the founder of the Tang Dynasty, began his conquest of China with Taiyuan as a base, and relied heavily on the support of its local aristocracy. Li's successor, Li Shimin, was also from Taiyuan. The cave temples at Tianlong Mountain, just southwest of the city, were constructed during the mid-to-late Tang Dynasty.

The old city was at Taiyuanzhen, a few miles east of the modern city. After the Song dynasty conquered China in 960, Emperor Taizong ordered the old city's destruction, but a new city was set up on the banks of the Fen River in 982. The city became a superior prefecture in 1059 and the administrative capital of Hedong (northern Shanxi) in 1107. It was destroyed by war in 1125, but was rebuilt. It retained its status as provincial capital, with minor changes in its name and status, until the end of the Yuan dynasty (1271-1368).

At the beginning of the Ming dynasty (1368–1644) it was renamed "Taiyuan Fu", and it retained this name until 1912. The city wall was reconstructed in 1568. During the Ming dynasty Taiyuan became the capital of Shanxi, and it remained Shanxi's capital during the subsequent Qing dynasty (1644-1911).

In 1900, during the Boxer Rebellion, seventy-seven unarmed Western missionaries were executed in front of the provincial governor. This incident became known as the Taiyuan Massacre.

During the Republic of China (1912-1949), the city's name was changed to "Yangqu", and it was not renamed back to "Taiyuan" until it was captured in 1949 by Communist forces. Since 1949, Taiyuan has developed a large industrial base with heavy industry (notably iron and steel) of prime importance; local coal production is considerable.