The castle has Norman origins and dates from the late 11th century. It is mentioned in the Domesday Book (completed in 1086) which states that at that time it was in the possession of Baldwin de Brionne, the Sheriff of Devon.
Much of the surviving masonry, including the bulk of the Keep and the curtain walls, is of fourteenth century construction.
On the execution in 1539 of its then owner Henry Courtenay, 1st Marquess of Exeter, the castle was partially dismantled. As a 'romantic ruin' it has been much painted and engraved over the centuries, including two paintings by JMW Turner and one by Samuel "Lamorna" Birch.
It is now in the care of English Heritage.