It was built in the corner of an old Roman fort guarding the Stainforth Pass through the Pennines. Around 1136, Alan, Count of Brittany, built a castle in the north-west corner of the site. After the death of Alan's son, Earl Conan the Little, ownership of the castle passed to the crown. King Henry II built a massive stone keep. Most of the work was carried out between 1171 and 1174, including repairs that were made following an attack by King William I of Scotland in 1173. Around 1216 enemies of King John again besieged the castle and it was further besieged in 1322 in a regional feud between Henry Fitzhugh and the then Earl of Richmond. After that the castle fell into ruin and the crumbling remains of Henry's keep are all that are now left.