The castle here is traditionally credited to Margaret Fitzgerald, who married Pierce Ruadh, Earl of Ormond around 1485. Lady Margaret became something of a local legend; according to historical accounts, she would regularly lead armed raids from the castle against neighbouring families she considered enemies. The estate later passed to Richard Butler, youngest son of the ninth Earl of Ormond, who was created Viscount Mountgarret in 1580 and made Ballyragget the principal seat of this junior branch of the powerful Ormond family.

The site saw military action during the turbulent year of 1600, when the sons of Lord Mountgarret joined a rebellion against the crown. Sir George Carew, Lord President of Munster, garrisoned the town with thirty foot soldiers stationed in what was then called “Mountgarret’s house”. The fortifications depicted on the 1665-6 Down Survey maps show both a castle and bawn still intact under the ownership of Lord Mountgarrett, noted as an “Irish Papist” in the accompanying records.

Today, the roughly square bawn measures approximately 100 metres east to west and 91 metres north to south, with two-storey round turrets at each corner and a tower house positioned north of centre within its walls. The eastern entrance features a round-headed archway protected by gun loops and an overhead double machicolation that served as both murder hole and defensive position. A second entrance pierces the western wall, similarly defended with gun loops. Though various modern structures, including sheds and houses, have been built against and within the bawn walls, and concrete now covers portions of the northern yard, the essential defensive features remain visible. The wall walk that once allowed defenders to patrol the perimeter can still be traced, though some sections have been filled with mortared rubble over the centuries.