Local tradition connects the castle to Margaret Fitzgerald, who married Pierce Ruadh, Earl of Ormond around 1485; a stone bench at the tower’s summit is still known as “Mairgread ni Gearoid’s chair”. Whilst she may not have built the tower, this formidable countess reportedly led armed raids against neighbouring enemies from this very stronghold. The castle became the principal seat of the Mountgarret line when Richard Butler, youngest son of the ninth Earl of Ormond, was created Viscount Mountgarret in 1580 and inherited Ballyragget.

The tower house itself, constructed from limestone rubble with a distinctive base batter, measures 13.2 by 9.4 metres externally. Its walls contain a fascinating array of architectural features: spiral stairs rising through the northeast angle, garderobe chambers on each of the four upper floors feeding into twin chutes, and various periods of alteration visible throughout. Notable amongst these is a polished stone fireplace on the fourth floor, inscribed with the date 1591 and the initials of Grissel Mountgarret, wife of the second Viscount. The building once had an adjoining structure on its eastern face, evidenced by two lines of corbels and a first-floor doorway that provided access between them.

The castle’s turbulent history continued through the centuries; it served as a garrison during the 1600 rebellion, became a grim execution site under Cromwellian occupier Axtell (who was himself later hanged at the Restoration), and functioned as a British military post during the 1798 uprising. The Butlers resided here until 1788 when they built Ballyragget Lodge nearby, and the estates eventually passed to their cousins, the McMurrough Kavanaghs of Borris, in 1813. In more recent times, the castle has served various purposes including a hatchery in the 1940s and a lumber store in the 1950s and 60s, though today its partially collapsed wooden floors make exploration hazardous.