This imposing four-storey limestone structure features a distinctive design with four projecting corner towers, though the northwest and southeast towers are notably larger than their counterparts. According to 18th-century accounts by Wright, the castle was once protected by substantial defensive walls and other fortifications, though these outer defences have long since vanished. The building found new life in the 1940s when the Sisters of St. Louis converted it into a school, but not before historian Tempest documented its medieval features in detail.

The castle’s architecture reveals sophisticated medieval planning, with each corner tower serving specific functions. The southwest tower houses the main stairwell, whilst the northwest tower contains multiple garderobes (medieval toilets) arranged in separate chambers on its eastern side, serving floors one through three. Entry to the castle is through a finely crafted limestone doorway with a two-centred arch, positioned just east of the southwest tower. Unlike many other tower houses in County Louth, where the ground floors of corner towers were typically solid or accessible only from above, Castletown’s ground floor provided direct access to three of its four corner towers; only the southwest stairwell tower was excluded from this arrangement.

The interior layout clearly indicates the social hierarchy of medieval castle life. The second floor, covered by a substantial barrel vault and featuring stone slabs, was evidently the castle’s principal living space, with exclusive access to the third-floor chambers in the southeast and northwest towers via internal staircases. The southeast tower, with its barrel-vaulted floors and traces of a two-light window with ogee heads, may have housed a chapel at second-floor level with the chaplain’s quarters directly above. A murder hole in the southwest angle of the first floor, positioned directly over the entrance passage, served as a grim reminder of the castle’s defensive capabilities. The uppermost levels feature a wall-walk covered with outward-sloping stone slabs, with each corner tower rising above the roofline to create additional small chambers accessible by narrow steps from the allure.