The castle, also known as Kingsfort, stood on level ground to the southeast of the village, though much of its physical presence has been lost to time and development. Historical records from the 1940s describe ruins located behind the local Garda Barracks, consisting of a wall fragment about 8 feet high. The barracks itself, which occupied the site of a house called ‘Old Castle’ on 19th century maps, was demolished in 1992. Local memory recalls a more substantial structure behind the barracks; a wall roughly 5 metres high with crenellations and a wall walk, which was removed within living memory.

The castle’s turbulent history reflects the broader conflicts of medieval and early modern Ireland. Originally an O’Conor stronghold, it was seized by the O’Rorkes in 1567. By the 1630s, contemporary accounts describe it as already lying in ruins. However, the site gained new life in the 1680s when John Leslie rebuilt it as a fortified house, and it even housed a garrison during the Williamite disturbances of 1688 to 1689. The architectural remnants that survived into the 21st century included two lengths of stone wall forming a boundary at the rear of the former barracks site, featuring elements such as a small defensive loop and a gateway with a drawbar socket, details consistent with 16th or 17th century construction.

Unfortunately, much of what remained of these historic walls met a disappointing end in 2003, when construction work at the site resulted in the demolition of significant portions, including the northwest wall and the return wall containing the defensive loop. Today, visitors to Ballintogher will find little physical evidence of the castle that once commanded this location, though its story remains preserved in historical records and archaeological surveys, serving as a reminder of the layers of history that lie beneath Ireland’s modern landscape.