Writing in 1687, Sir Richard Cox noted the castle's location near what he claimed was "the best blackslatt in Ireland", suggesting the area was known for its high quality slate quarrying. The castle's Irish name, Ard na gCreach, translates to "the hill or height of the plunders", hinting at a rather tumultuous past for this strategic stronghold.

By the 19th century, the castle was already in ruins. John O'Donovan's 1841 account describes it as "totally demolished", with only three clumps of walls remaining to mark where it once stood. Local tradition held that the Fitzgeralds, one of Ireland's most powerful Norman families, had erected the castle, though the exact date of its construction remains unknown. A century later, in 1942, surveyors found even less; just fragmentary walls outlining the castle's footprint, which by then formed boundaries between the farms of James O'Sullivan and Patrick Sullivan.

Today, Ardnagragh Castle has vanished entirely from the landscape. Digital satellite images taken between 2011 and 2013 show no visible surface remains whatsoever. The complete disappearance of this once notable fortress serves as a reminder of how quickly Ireland's built heritage can fade when left unprotected; from a castle worthy of mention alongside the region's finest slate, to farm boundaries, to nothing at all in the span of just over three centuries.