The construction of the fort was first considered during a build-up of the French armed forces undertaken by Louis XIV between 1661 and 1667, but Vauban, his leading military engineer, famously advised against it, saying "Sire, it would be easier to catch the moon with the teeth than take on such an endeavour in such a location". Fort Boyard was to form a line of fortification with Fort Enet and Fort de la Rade on Île-d'Aix to protect the arsenal of Rochefort from Royal Navy incursions.
Construction of the fort did not begin in earnest until 1801, under Napoleon, in order to protect the coast (and especially the arsenal of Rochefort) from possible incursions by foreign (and especially British) navies. At that time, cannons only had a limited range, and the distance between the two islands of Aix and Oleron was too large to block the passage.
Following difficulties in establishing a firm base (stone blocks had to be installed on the sandy sea bed during low tide, where they could still sink under their own weight) the project was suspended in 1809. Construction resumed in 1837, under Louis-Philippe, following renewed tensions with the United Kingdom. The fortifications were completed in 1857, with sufficient room for a garrison of 250 men.
However, by the time of its completion, the range of cannons had markedly increased, making the fort unnecessary for national defence.
Under the Paris Commune of 1871, Fort Boyard was briefly used as a prison, before being abandoned a few years later. Over time, the fort slowly crumbled and deteriorated into the sea as it was left unmaintained.
In 1961 the Commune sold the fort to the département of Charente-Maritime. Six years later, the final scene of the French film Les Aventuriers was filmed at the remains of the fort.
In 1988, restoration work commenced on Fort Boyard, in order to prepare it for a television game show, also named Fort Boyard. The fort was electrified and a jack-up barge has been installed on the side in order to facilitate access to television crews. Restoration on the fort finished in 1989 and filming commenced in 1990. Fort Boyard has been used as the set for the show to this present day.