The first masonry castle was constructed by Vytautas, Grand Duke of Lithuania, in 1412-1413 on an island, now a peninsula, in Lake Asveja in order to secure the capital Vilnius from attacks from Livonia. No accounts concerning the architecture of Vytautas' castle have survived. It was acquired by the Radvila family prior to 1508. They constructed a new palace in the Renaissance style in the first half of the 16th century. Barbara Radziwiłł spent 5 months in the castle after her marriage to Sigismund Augustus in 1547. The palace used to be one of the most luxurious residences in the Duchy, lagging not much behind the Royal Palace. Dubingiai Castle was the main seat of the Biržai-Dubingiai line of the Radziwiłł family until the second half of the 17th century, when it was transferred to Biržai Castle. The neglected castle and church gradually fell into ruins. Today only the foundations and several cellars of the castle and church remain and are being researched.
The masonry Calvinist Church of the Holy Spirit was built in the Renaissance style near the castle by Janusz Radziwiłł prior to 1620 and was intended to be the mausoleum of the Radziwiłł family. The most prominent members of the family were interred there, including Mikołaj "the Black" Radziwiłł (1565) and his wife Elżbieta Szydłowiecka (1562), Mikołaj "the Red" Radziwiłł (1584) and Janusz Radziwiłł (1620). Their remains were discovered during archaeological excavations in 2004 and reburied there in 2009.