The castle was built in 1433 for the Lithuanian noble Jonas Goštautas, voivode of Trakai and Vilnius, replacing the original wooden fortress. In the 1560s, upon the death of the last member of the Goštautas family the castle became the property of king Sigismund II Augustus, who expanded it. The construction was supervised by Hiob Bretfus, military engineer and royal architect. During the reign of Sigismund Augustus the structure served as a royal residence with an impressive treasury and library as well as the main arsenal of the crown. In 1611-1632 the castle was rebuilt again and surrounded with fortifications by Krzysztof Wiesiołowski, starosta of Tykocin.
During the Deluge in 1655 the Radziwiłł army occupied the castle. On december 31, 1655, when the castle was besieged by troops of the Tyszowce Confederation, Janusz Radziwiłł, one of the most powerful people in the Polish–Lithuanian Commonwealth considered by some as the traitor, died here. Ultimately, the castle was captured on January 27, 1657.
In the following years the castle and surrounding lands were donated to Stefan Czarniecki in reward for his contribution in the war. The new owner rebuilt the castle after 1698. In November 1705 the meeting between the king Augustus II the Strong and Peter the Great took place here. During this meeting the Order of White Eagle was established by the King of Poland.
In 1734 the castle was destroyed by fire. Since that time, no inhabited building began to fall into disrepair. In 1771 remains of the castle were destroyed by flood and in 1914, during World War I, the material from the remaining walls was used by the German soldiers to build roads.
Based on the preserved plans of the fortress, found in the archives in Saint Petersburg, the residential part of the castle has been restored (west wing in the style of late Gothic). The original castle was built on a plan of a trapezoid with a courtyard and four cylindrical towers at the corners. The complex was surrounded with fortifications - curtains combined four terrestrial inner bastions.
In about 1565 the king Sigismund Augustus installed at the castle his private treasury and a collection of books, initially stored at the Vilnius Castle. Sigismund Augustus was a passionate collector of jewels. According to nuncio Bernardo Bongiovanni's relation, his collection was allocated in 16 chests. Among the precious items in his possession was Charles V's ruby of 80 000 scudos' worth, as well as the Emperor's diamond medal with Habsburgs Eagle on one side and two columns with a sign Plus Ultra on the other side. He had also a sultan's sword of 16 000 ducats' worth, 30 precious horse trappings and 20 different private-use armours. The papal nuncio wrote in his diary I also saw twenty suits of royal armour of which four were of most wondrous workmanship, namely with a beautiful sculpture and figures set with silver [...] It cost six thousand scudos. The king's possession included a rich collection of tapestries (360 pieces), commissioned by him in Brussels in the years 1550-1560, which decorated the castle's walls. The king's treasures were scattered after his death.