The first motte and bailey castle on this site was built in 1080 by Osmund, Bishop of Salisbury. This castle burnt down in 1113 and was rebuilt in stone by Roger, Bishop of Salisbury, by 1120. He occupied it under Henry I and later under Stephen. Roger sided with Stephen and the castle was taken and retaken. It then remained the property of the Crown and it was used as a prison by Henry II and Henry III. It went on to become the property of Catherine of Aragon, the first wife of Henry VIII.
In 1643 during the Civil War it was occupied by Royalist troops and besieged by Parliamentary forces under Sir William Waller. However three days later in the Battle of Roundway Down Waller's army was routed by Royalist forces. The castle and town remained in Royalist hands under the military governorship of Sir Charles Lloyd who defended the town against repeated attacks and bombardments by the Parliamentarians. In September 1645, Cromwell with large forces and heavy artillery invaded the town and laid siege to the castle, which capitulated after a bombardment. In May 1648 the castle was dismantled following a Parliamentary Order, a process known as slighting. All that remains of the castle today is the original mound, the outline of the moat and traces of the foundations of the great hall.
The present castellated Victorian era 'castle', which is Neo Norman/Gothic architecture in style, was built by the Leach family in the 19th century. The building is now divided into flats in private ownership and is not open to the public. The main part of the castle, including seven bedrooms, four bathrooms and six reception rooms, is currently for sale at £2.5million.