This Fitzgerald stronghold served as the seat of the Lords of Decies, descendants of Gerald Mór, who likely constructed the castle in the late 15th century. Gerald Mór himself was the son of James, the seventh Earl of Desmond, placing this castle firmly within one of medieval Ireland's most powerful dynasties.

What makes Dromana particularly remarkable is its survival through Ireland's most turbulent periods; it somehow evaded both the Desmond forfeitures and Cromwell's sweeping confiscations of the 17th century. Through strategic marriage alliances, the castle eventually passed to the Villier-Stuart family in the 18th century, who maintained it as their seat well into the 19th century. Today, visitors can still see fragments of the medieval fortress, including the substantial base of a circular tower, at least 12 metres in diameter and standing about 6 metres high, complete with its defensive base-batter that juts out over the river below. This tower likely formed part of the original bawn's corner defences.

The 17th century saw most of the medieval structures demolished, though builders of that era showed some respect for the past by incorporating architectural elements into their new construction. A particularly fine two-light window with ogee-headed arches, decorated spandrels and hood-moulding from the original tower house was carefully preserved and built into a late 17th-century house that once stretched eight bays wide and two storeys high. This building was captured in illustrations by Daniel Grose around 1820, providing us with valuable documentation of how the site appeared during the Georgian period, though the 18th and 19th-century additions have since been demolished.