The castle was likely constructed between 1224 and 1226 by William Marshall the younger, who served as Justiciar at the time, making him the principal government officer in Ireland. Built on what may have been an earlier ringwork castle, this imposing fortress would become a focal point of conflict between English and Irish forces for over a century. The castle changed hands multiple times, falling to Irish forces in 1315-16, 1331, and 1357-8, before the Kavanaghs finally captured it in 1360 and held it for the next two hundred years.

Today, only the southwest and northeast walls of the massive keep survive, along with two drum towers that hint at its former grandeur. Archaeological excavations led by P.D. Sweetman in 1979 revealed fascinating details about the castle's defences, including a rock-cut fosse, or defensive ditch, that measured 5 metres wide and 3.5 metres deep, with an entrance causeway on the southern side. The keep's internal dimensions stretched approximately 27.5 metres from northwest to southeast and 18.5 metres from northeast to southwest, making it one of the larger fortifications of its type. The three-storey structure features cross-loops on the ground floor for defence, whilst the upper floors boast larger trefoil windows that would have let in more light. Particularly noteworthy is the southeast tower, which contains a chapel with a groin-vaulted roof on the first floor, a reminder that these fortresses served both military and spiritual purposes.

After the Kavanaghs surrendered the castle to royal authority in 1543, it passed through various hands, including Sir Thomas Masterson in 1583. The Mastersons managed to maintain control through the turbulent 1640s but surrendered to the Cromwellian general Coote in 1649. Since then, the castle has been largely abandoned, standing as a silent witness to centuries of Irish history until the Office of Public Works undertook conservation efforts. Now designated as National Monument No. 521 under state guardianship, Ferns Castle offers visitors a tangible connection to the complex political and military history of medieval Wexford.