A succession of buildings has been on the site. The first was a Norman castle built by the Bulmers which was rebuilt by the Nevilles in the late 14th century. For many years the castle was owned by the Neville family until in 1569 it was confiscated by the Crown following the family's involvement in the Rising of the North..
There have been a number of other owners since that time. In the early 1600s the estate was granted by the Crown to Carr and sold by them to Sir Nicholas Cole. In about 1720 the property was acquired by Sir Henry Bellasyse whose daughter was involved with Bobby Shafto. In 1796 the Castle was acquired by the Russells.
The present building is largely a 19th century restoration carried out in the 1820s by John Matthew Russell and improved in mid 19th century by architect Anthony Salvin for William Russell ( High Sheriff of Durham in 1841)
The castle is now privately owned by the Dobson family. There is a craft fair twice a year, and two open air Shakespeare events