The site's history stretches back to 1247, when Martin Cod held the lands of Balkwych for military service, specifically providing a quarter of a knight to the Valence property. This system of feudal tenure continued through the centuries, with Hugh Lampet taking on the same obligation for Balyur (or Balyen) by 1324. The arrangement became more complex over time; by 1390, a jury determined that the estates of Rathaspick and Ballyhire were jointly held through military service, though they'd been divided at some point between the Codd and Lampert families respectively.

The Lampert family's connection to Ballyhire endured for centuries, leaving their mark on both the landscape and historical records. When Jack Lampert died in 1624, he left behind not just the castle but also a stone hall and 100 acres of land. His descendants continued to hold significant acreage in the area, with Philip Lampert owning 100 acres at Ballywith and 10 at Ballyhire in 1640. The family's fortunes took a darker turn during the Cromwellian period; James Lampert and eight dependents were marked for transplantation in 1653, a fate that befell many Irish landowners during that tumultuous era. The Down Survey maps from 1656 to 1658 captured the castle and an attached structure, providing valuable visual evidence of the site's former importance.

Today, visitors to Ballyhire can see the base of what was once a formidable tower, measuring approximately 8.5 metres north to south and 8.45 metres east to west, with walls still standing between 1.2 and 2.3 metres high. Archaeological investigations in 1997 revealed traces of the castle's defensive capabilities, uncovering an east to west bawn wall south of the main structure, along with some cobbled surfaces. Though the castle no longer stands in its full glory, these surviving elements offer a tangible connection to medieval Wexford, when military service bound families to their lands and stone walls provided both status and security in an uncertain world.