While local tradition credits one Carbery O’Shea with its construction, the castle is more firmly linked to the powerful MacCarthy clan, who likely maintained some form of residence here as early as 1398 when the annals record Taghd Mac Carthaigh’s death at this location. The existing ruins, however, date from later; probably those referenced in a 1569 document as ‘the castle of Valencyen called Ballycarborow’. Throughout the sixteenth century, the O’Connells occupied the castle as MacCarthy wardens, with Morgan O’Connell even rising to become High Sheriff of Kerry during Elizabethan times.

The castle’s fortunes shifted dramatically following the death of Daniel MacCarthy More, Earl of Clancar, in 1596, when possession passed to Sir Valentine Browne. Its military significance came to an abrupt end during the Commonwealth period when parliamentary forces deliberately slighted the structure in 1651-52 whilst fortifying nearby Valencia Harbour. What remains today is a fifteenth-century tower house, once surrounded by a substantial bawn that originally enclosed an area measuring approximately 31.6 metres north to south by at least 35 metres east to west. Much of the southern and eastern walls of both keep and bawn have vanished; some destroyed during the seventeenth-century slighting, others removed in the early twentieth century when large sections of faced masonry were carted away.

The surviving western bawn wall rises from a slightly battered base to about 4.2 metres in height, pierced by narrow vertical loops set in splayed, lintelled embrasures. A gap in the northern section, roughly 1.9 metres wide, likely marks the original entrance, which once featured holed stones on either side for pivoting gates. Atop the wall, remnants of an alure (wall-walk) lead to a small mural chamber that was later converted into a garderobe. This modification may have coincided with the construction of a large eighteenth-century house that once abutted the exterior of the bawn wall; home to the Lauder family until its demolition in the early twentieth century. Early nineteenth-century watercolours by Daniel Grose capture both this lost house and the massive blocks of masonry that once littered the castle’s southern and eastern approaches, offering a glimpse of Ballycarbery’s appearance before its final dismantling.