The castle’s history stretches back to the 12th century, when the area may have been held by Milo de Lamporte in 1195. Local tradition tells that when his son Rudolph departed for the Crusades, he left the manor to the Church should he not return, which indeed he didn’t, though this tale lacks historical corroboration. By 1247, the lands of Ballyfistlan were held by Walter Purcell, and over the centuries passed through various hands including John Maunsel in 1324 and William Sandull around 1425. The Browne family of Mulrankin held the manor from at least 1617, when William Browne was leasing Ballyfistelbane with its three townlands, mill and castle, with his descendants maintaining ownership through the 1640s.

The castle itself is a five-storey rectangular tower house measuring approximately 9 metres by 8.6 metres externally, complete with base batter and well-preserved quoins that survive to the wall-walk level. Its defensive architecture is sophisticated; the original pointed-arch entrance on the southwest wall was protected by a murder hole, whilst a secondary entrance was later added. The interior reveals a complex layout with ground floor chambers featuring embrasures and lights in multiple walls, supported by corbels. A mural staircase winds through the southwest wall, connecting all five floors, each with distinct features including fireplaces, garderobe chambers, and window embrasures. The second floor contains a particularly notable large arched embrasure on the northwest wall, whilst the upper floors provided access to both living quarters and defensive positions. An attached gatehouse, offset to the south and west, adds another layer of defence with its own murder hole and draw-bar socket, the timber beam of which remarkably survives in place.

The castle formed part of a larger defensive complex that effectively fortified the entire island. Walls extended from both the gatehouse and tower house, whilst approximately 50 metres northeast stands a severely tilted single tower that once guarded the causeway connecting the island to the mainland. This tower, measuring about 5 metres by 3.6 metres, features a cross-loop commanding the causeway, which was known locally as the ‘togher’, giving Lough Togher its 17th-century name. The semi-saltwater lagoon surrounding the site, formed where seawater seeps through and occasionally breaches the coastal bar, creates a unique environment for this medieval stronghold. Today, whilst a modern raised platform with an altar has been built against the northeast wall of the main tower, obscuring some original features, the castle remains one of the best-preserved examples of late medieval fortification in County Wexford, its stones still bearing witness to centuries of Irish history from the Norman invasion through to the Confederate Wars.