The stone castle itself was once enclosed within a square moated site measuring 43 by 43 metres, defined by a fosse six metres wide and nearly a metre deep. Aerial photographs from 1970 reveal the cropmarks of this ancient defensive system, complete with an entrance causeway on the north side and traces of a second, outer fosse that provided an additional layer of protection. These earthworks likely served as the bailey or courtyard for the masonry castle, though some historians suggest they might predate the stone structure, possibly representing the remains of an earlier earth and timber fortification from the late 12th or early 13th century.

The site's layered history becomes even more intriguing when considering its possible origins as a Gaelic moated castle built by the Mac Giollamocholmóg clan in the 13th century, challenging assumptions about Norman dominance in medieval Irish castle building. When the Ordnance Survey visited in 1838, they documented not just the castle and its surrounding rath, but also traces of buildings and enclosures to the north; evidence of a deserted medieval village associated with the manor of Rathdown. Archaeological investigations in 1994 and geophysical surveys in 2016 have confirmed the presence of medieval agricultural features, field systems, and ditched boundaries, though heavy plough damage over the centuries has left many of these remains fragmentary.

Today, much of the archaeological landscape around Rathdown has been altered by modern development, including a sewage treatment works that removed portions of the western and southern ditches. However, beneath the surface, traces of this multi-period site survive, from the medieval settlement that once thrived here to the agricultural features that supported the castle's inhabitants. The nearby church of St. Crispin's Cell, located 195 metres to the southwest, serves as another reminder of the religious and social complexity of this medieval manor, where Norman and Gaelic influences intertwined to create a distinctive Irish castle landscape.