Yet despite that scale, no town wall has ever been found. What survives instead is a ghost of a defensive system: a handful of gate names preserved in old deeds, two fosses uncovered during excavations on Lower Green Lane, a fragment of walling reported near the east end of Mill Street, and a partially surviving tower on West Street that tradition connects to a thatched gatehouse manned by a Westmeath captain during a seventeenth-century siege. The question of how exactly Callan was defended, and whether it ever had a continuous circuit of masonry at all, remains genuinely open.

The documentary trail is long and fragmentary. The Ormond Deeds, a series of legal records relating to the Butler family's extensive landholdings, mention an East Gate, a South Gate, a West Gate, a Haggard Gate in the northern suburb across the Kings River, and an inner or 'midle' gate along West Street. A deed of 1583 records the town authorities granting a property extension to one Richard Merry on condition that he maintain the town ditch adjoining his holding in 'good defensible and decent manner', which suggests that the fosse, a defensive ditch typically dug around a town perimeter, was partly in private hands to maintain. A map drawn by Thomas Stuish in 1681 and copied in 1765 by Richard Frizell labels a 'Strong Castle' and gate at the eastern end of town, which Carrigan, writing in 1905, identified with a structure known as Coorthfeerish or Pierce's Court, its lower storey having once served as a town entrance. By 1840 the Ordnance Survey recorded it as a ruin; by 1948 it was reduced to a site. The 15th-century walls Carrigan described were already almost entirely gone by his own time. The most dramatic evidence that the defences were real comes from 1650, when Cromwellian forces set up a battery some 300 metres to the south and had to fight hard to take the southern gatehouse, with Captain Marcus Geoghegan of Frevenagh, County Westmeath, then offering further fierce resistance at the middle gate before that too fell.

The likeliest current interpretation is that Callan was never uniformly walled. The western portion of the town may have relied on a ditch alone, while the eastern half, where the main garrison was stationed in 1650 and where the strong castle stood, may have had a more substantial walled enclosure. Skerry's Castle on West Street, a portion of which still stands, sits at the probable boundary between those two zones, which would explain both its designation as a middle or inner gate and the particular ferocity of the fighting there. Archaeological excavation on Lower Green Lane has confirmed at least two fosses along the southern boundary, lending some physical weight to what is otherwise a story told almost entirely through deeds, surveys, and a single seventeenth-century map.