The castle formed the centrepiece of extensive Devereux family holdings that stretched across Wexford from the 13th century onwards. The estate came into Devereux hands through Alina de Heding, daughter of Sir Alexander de Heddon, who married Sir Hugh Devereux in the mid-1200s. Together with Adamstown castle and lands in nearby Bantry barony, Ballymagir served as one of two administrative centres for the family’s properties, held by the service of two knights’ fees from 1247.

The surviving structure dates from medieval times, though written records only mention it from the 1600s onwards. Built as a rectangular tower measuring 11.4 metres north to south and 7.1 metres east to west, the castle features thick walls with a distinctive base batter and a pointed doorway on its southern side. Inside, two barrel-vaulted chambers run east to west at ground level, their stone ceilings still intact after centuries. Archaeological examination in 2007 revealed evidence of a western extension, possibly built entirely of timber, though no trace of this structure remains above ground. The castle sits within a larger moated site, its defensive earthworks still visible in the surrounding landscape.

The Devereux family retained Ballymagir through centuries of political upheaval until the Cromwellian confiscations of the 1650s. Nicholas Devereux, who owned the castle and 668 acres in 1654, faced transportation to Connaught with 31 dependents but successfully reclaimed his property through the Court of Claims in the 1660s. By the 19th century, the estate had passed to the Loftus family and the area became known as Richfield. Victorian drainage schemes transformed the landscape dramatically; where tidal waters once reached within 500 metres of the castle walls, reclaimed farmland now stretches over two kilometres to the relocated shoreline at Ballyteige Burrow. The medieval tower was later incorporated into a domestic dwelling, with a newer house built against its northern wall, preserving this fragment of Norman Wexford within more recent architecture.