The small manor of Ballyell, which comprised roughly half of Kiscoran parish, was originally held by the Codd family as part of a quarter knight's fee from the Valence estates well into the 15th century. However, by the 1640s, their fortunes had clearly diminished; Nicholas Codd retained just 36 acres at Coddsballyell, and the family no longer controlled the castle at Hill itself.

The Hay family's connection to the area dates back to at least 1345, when Adrian Fitz-Jones Hay of Kilscoran appears in historical records. By 1640, Mathew Hay held 98 acres at Hill, and the castle appears prominently on the Down Survey barony maps created between 1656 and 1658. The property eventually passed to the Nunn family, who incorporated the medieval structure into the northwest corner of a more modern house.

The castle itself was likely a tower house, described in 1940 as standing about 50 feet high with six modern windows inserted into its north side. Sadly, this remnant of medieval Ireland was demolished around 1960, though its memory persists in the landscape. The tower once stood within a circular moated site, a defensive feature typical of Anglo-Norman settlements in medieval Ireland, which still marks the spot where centuries of local history unfolded.