This tower house once belonged to George Fay, an Irish Catholic landowner, and appears on the 1657 Down Survey map of Rathgarve parish, marked in the townland of 'Tromore'. The castle's prominent position would have served both defensive and symbolic purposes, allowing its occupants to survey their lands whilst demonstrating their status to all who passed by.

By the time the Ordnance Survey mapped the area in 1837, the castle had already fallen into disrepair. The surveyors recorded two rectangular structures or field boundaries at the site, one specifically annotated as 'Castle'. To the west, they documented a tree-covered walled garden or orchard, roughly 40 metres north to south and 70 metres east to west, enclosed within sub-rectangular walls. This formal garden, typical of Irish tower houses, has since been completely levelled.

Today, visitors to Tromra will find little evidence of the castle that once stood here. Only grass-covered wall footings remain, surrounded by low earthen banks that form an irregular enclosure; possibly the remnants of a bawn, the defensive wall that typically protected Irish tower houses and their outbuildings. Even aerial photography from 2011 reveals no surface traces of either the castle or its associated earthworks. The site stands as a quiet reminder of the many fortified houses that once dotted the Irish landscape, their stories preserved only in old maps and historical records.