The earliest parts of the structure date from before 1700, when a castle occupied this site; clear enough to be marked on the Down Survey maps of 1655-6. Rather than demolishing the old fortress, successive generations chose to incorporate its fabric into what would become a grand country house, creating a unique blend of defensive medieval architecture and Georgian elegance.

The house as it appears today is primarily a late 18th-century creation, originally built as a plain three-storey block in 1731. However, its most significant transformation came after it suffered extensive damage during the 1798 Rebellion. Around 1820, the renowned architects Richard and William Vitruvius Morrison were commissioned to restore the building, and they chose to give it a fashionable Tudor Revival exterior whilst creating rich, largely Classical interiors. This juxtaposition of styles; medieval castle remains, Georgian bones, Tudor Revival façade, and Classical interior decoration; makes Borris House a remarkable chronicle of changing architectural tastes.

Further alterations came around 1870, when a projecting arcaded porch was added along with a crenellated parapet complete with turret finials, enhancing the romantic medieval character the Morrisons had begun. A service wing was also added to the rear during this period. The estate features several noteworthy additions, including a stable complex, walled garden, and a freestanding folly tower dating from 1817. Today, the seven-bay, three-storey house over its basement level remains an intriguing architectural palimpsest, where visitors can trace nearly four centuries of Irish building traditions in a single structure.