The Barons of Aigle were first mentioned in 1179. At that time they had a small fortification, that became the center of the modern castle, along the road over the Col du Pillon and Col des Mosses passes of the Rhone. However, only traces of this first castle have been archaeologically discovered.
Some time before 1200, the Barons of Aigle ended up as vassals of the powerful Counts of Savoy. In 1232, Count Thomas of Savoy granted Aigle as a fief to the brothers Jacob and Peter of Saillon in exchange for their ancestral castle in Valais. The Saillon family seems to have been closely related with the barons of Aigle.
In the 14th Century, the Lords of Compey inherited the rights of the Saillon family. They were also vassals of the Counts of Savoy and made Aigle into their headquarters. They added turrets and in 1450 built a massive tower in the south corner. This tower was an example of late French Donjon architecture.
Starting in the mid-15th Century, Bern tried to control the city, to gain control of the important mountain passes into the Rhone Valley. Their first unsuccessful attempt was in 1464. The Burgundian Wars (1474-1477), which brought the Swiss Confederation into conflict with Duke Charles the Bold of Burgundy, brought another opportunity. Because Savoy had joined the Burgundian side, Swiss forces attacked the Savoy town of Aigle and devastated the castle. Jean de Compey had to flee and was killed soon afterward in Vevey. His son died fighting for Charles the Bold.
After the war, the Compey family were unable to recover their title to the town or castle. Bern had the castle rebuilt again in 1488 and made it the seat of a provincial governor. The representatives of Bern resided here until the French invasion and creation of the Helvetic Republic in 1798.
In 1804, the castle was acquired by the community of Aigle and until 1976 it was used as a cantonal jail. Since then, it has been a museum open to the public.
Today the castle is home to Musée de la Vigne et du Vin (Vine and wine museum). Next to the castle gates is the Maison de la Dîme, which houses the Musée de l’Étiquette (Wine-Labels Museum).