The site’s most prominent feature is a three-storey tower that likely began life as a small tower house, possibly dating to the late medieval period. This rectangular structure, measuring 4 metres east to west and 6 metres north to south, originally had its entrance through a doorway in the west wall, now blocked, which led to a stone-vaulted ground floor. A mural staircase built into the south wall provided access to the upper floors, whilst a small spiral staircase, now ruined, once connected the first floor to the roof.

The 1659 Down Survey records that the castle belonged to Sir Thomas Nugent, described as an Irish Papist in 1641, providing a glimpse into its ownership during a turbulent period of Irish history. By the 17th century, the original tower appears to have been incorporated as the northeast corner tower of a larger complex. The 1838 Ordnance Survey map shows this tower as a small freestanding rectangular structure, whilst depicting Taghmon Castle proper as a long L-shaped building with a possible corner tower at its southeast angle. Today, only fragments of this larger castle survive; the south wall and west gable remain intact to ground floor level, with evidence of a fireplace and wall ovens visible in the destroyed embrasures of the western gable.

The tower underwent a curious transformation in the post-medieval period, when its upper levels were converted into a dovecote, with the walls of the two upper floors lined with pigeon holes. This practical repurposing coincided with the construction of a large walled garden to the south of the castle, featuring three pointed doorways made of roughly dressed limestone. Some doorways show brickwork dating to the 18th or 19th century, suggesting continued use and modification of the site well into the modern period. The remains paint a picture of a site that evolved from medieval defence to agricultural utility, with later additions including 19th-century houses and post-medieval structures built against the old bawn walls. Today, modern farm buildings, including cattle crushes and concrete livestock pens, continue this tradition of practical adaptation, whilst the medieval church and graveyard 220 metres to the south-southeast provide additional context to this historically rich landscape.