The castle first appears in historical records in 1581, mentioned in the will of Brian Oge, Lord of Upper Ossory, though its stone walls likely date back to the fifteenth century or perhaps earlier. The Down Survey maps from the 1650s reveal that even then, a cluster of large houses surrounded the castle site, suggesting Borris had already developed into a village of sorts by that time.

What remains today are the weathered ruins of what was once a substantial fortification. The ground floor walls, built entirely from rough, unhewn stone without a single chisel mark even on the corner stones, still stand about 25 to 30 feet high; though they probably reached double that height when intact. The interior measured approximately 40 feet from north to south and 25 feet from east to west, originally divided down the middle by a wall that supported two parallel stone arches running the building’s entire length. A clever bit of medieval engineering saw the main entrance door in the north wall near the northeast corner, which led both into the eastern section of the ground floor and, via a turn to the left, into a straight stairway built within the seven foot thick eastern wall that provided access to the upper storey.

The castle’s story continued well beyond its medieval origins. In 1626, Charles I granted it to George Villiers, Duke of Buckingham, and by 1693 it had passed through various hands to Owen Carroll, who leased it before passing it to his son Barnaby. Today, Borris House stands partly built over the castle’s foundations, incorporating the ancient structure into its own design. Beneath the house, the original medieval cellars remain intact and still serve as storerooms, whilst above ground, only scattered fragments of the first and second storeys survive as testament to what was once a formidable Irish stronghold.