Today, little remains of this once grand structure; the eastern half has been incorporated into a farmyard with significantly reduced ground levels, whilst archaeological excavations in 2006 revealed only large stones, red brick and mortar fragments. The site appeared on the 1900 Ordnance Survey map and was still visible as an overgrown mound in 1992, rising about 1.5 metres above the surrounding ground, but even this modest reminder has since been removed.

The castle's story begins in 1172 when Theobald fitz Walter, Chief Butler of Ireland, received Gowran as a reward from Richard de Clare (Strongbow) following the Norman acquisition of Leinster. Initially, a motte and bailey castle was likely erected, but around 1385, James Butler, the 3rd Earl of Ormond, built a new castle that would serve as the Butlers' principal seat until they acquired Kilkenny Castle in 1391. Gowran's position as a frontier town is vividly illustrated by a 1382 document granting local leaders licence to negotiate with Irish enemies, the McMurghs and O'Nolans, allowing them to trade provisions and cloth for defence purposes, though strictly prohibiting the sale of victuals, horses or armour during wartime.

The castle underwent several transformations throughout its existence, including a rebuild around 1500 by Margaret FitzGerald, Countess of Ormond. A 1710 map depicts it as a central two storey block flanked by turrets, connected to the town by a lane running from the eastern end. Its military importance was dramatically demonstrated in 1650 when Cromwell's forces besieged the structure; after artillery bombardment, Colonel Hammond surrendered what Cromwell himself described as "a very strong castle" well stocked with provisions. Hammond and his officers were executed, and the castle was set ablaze. Following a fire in 1713, the deteriorating castle was modified with a fashionable Georgian front featuring nine windows and a central pediment, before being finally replaced by a new house in 1815-16 when the old structure became too dilapidated to maintain.