The château was built from 1397 to 1425 by the Bishops of Lausanne to serve as their fortified residence. Begun under Guillaume of Menthonay, it was completed under his successor, Guillaume of Challant, and named after Saint Marius, the first Bishop of Lausanne. It served as the bishop's residence until 1536, when Berne captured Lausanne and secularized the bishopric (the bishop, Sébastien of Montfalcon, escaped through a hidden stairwell). The Bernese installed a bailiff in the château and used it as an armory. Upon the creation of the canton of Vaud in 1803, it became seat of the cantonal government, a role it has retained.
The château was built as a single massive rectangular block, as was common at the time, with brick for the upper portion and sandstone for the lower portion. It originally had Ghibelline merlons, which gave it a somewhat Italian appearance, but due to the wet climate, the roof was extended and the merlons filled, probably in the 16th century. The windows that form a row just below the eaves fill the gaps between the merlons, and the arches above the windows fill the v-shaped openings in the Ghibelline style of merlon.
In 1789, the Bernese built an annex on the west side of the castle, through which it is now entered. A tower that formerly stood next to the château was demolished in 1890, and around the same time, a statue of Abraham Davel was installed against the front wall.