Though no trace of the castle remains today, historical records paint a picture of what once stood here. A 1766 drawing by artist Gabriel Beranger depicts an impressive tower with diagonally positioned turrets and mullioned windows, whilst a later sketch by W.F. Wakeman from the early 19th century shows the building still standing with its gatehouse and battlemented parapet intact.

The manor at Ballyfermot has a long and varied history of ownership. The Fitzwilliam family held it in the early 14th century before it passed to Robert de Clahull in 1327. By the late 1300s, it had come into the hands of Reginald Barnewall, and in 1395 Reginald Butler took up residence there. The Barnewall family returned to the property by 1451 when John Barnewall was recorded as living there. The castle remained occupied until the late 16th century, when it began its slow decline.

After falling into disrepair, the old castle building found new life in the 18th century when it was converted into a school. The structure managed to survive into the early 1800s, as evidenced by Wakeman’s detailed sketch, but sometime after that the last remnants were cleared away. Today, visitors to Le Fanu Park won’t find any physical traces of the castle that once dominated this hilltop site, though its location on the summit of the ridge remains unchanged from medieval times.