This rectangular tower house, measuring approximately 11.5 metres north to south and 9.4 metres east to west, rises two storeys high with distinctive rounded corners and thick limestone walls that reach over two metres in places. A small square turret projects from the north end of the western wall, though its exact purpose remains unclear; some suggest it served as a buttress rather than a functional tower room.

The castle's history traces back to at least 1606, when the lands of Hilton, as Hilltown was then known, formed part of a substantial land grant to Lady Marie Nugent, widow of Christopher, baron of Delvin. The grant included two messuages, several cottages, and 240 acres of arable land. Interestingly, no castle appears on the 1657 Down Survey map of Fore Barony, though records from that period show Nicholas Archbold, described as an Irish papist, owned the lands of 'Hiltowne' in 1641. The absence of the castle from contemporary maps raises questions about when exactly it was constructed.

Today, visitors can still observe the castle's original architectural features, including the main entrance at the western end of the south wall, which opens into a lobby area containing mural stairs built into the eastern wall. Though these stairs are now blocked in the south wall, they once provided access to the first floor near the southeastern angle. The ground floor preserves evidence of a collapsed stone vault, with springers still visible in the eastern wall, whilst the first floor retains its fireplace in the northern wall and a double-splayed window opening in the western wall. A two-bay, two-storey house from around 1725, now serving as an outbuilding, extends from the castle's eastern wall, creating an intriguing architectural timeline that spans several centuries.