The Loftus family built this imposing structure in the late 16th century as a semi-fortified house, essentially an enlarged tower house designed to balance comfort with defence. The original castle follows a rectangular plan with remarkably thick walls, a central spine, and four flanker towers; its walls of calp limestone, now plastered over, rise three storeys with a distinctive battered profile that tapers as it climbs.

The castle changed hands several times over the centuries, each owner leaving their mark on the structure. In the 18th century, it passed to Speaker Connolly, the Hoadleys, and eventually the Earls of Ely, who remodelled it to suit Georgian tastes. The 20th century brought further changes, including a new kitchen wing, a bow window, and a colonnaded front portico that transformed its appearance whilst preserving the medieval core. Archaeological excavations in the early 1990s revealed fascinating details about the original layout, including a vaulted passage that tapers towards the southwest flanker tower.

Many original features survive beneath later additions, offering glimpses into Tudor-era craftsmanship. The ground floor retains its vaulted ceiling and an original fireplace, whilst the second floor boasts another period fireplace with early brickwork comparable to examples from Hampton Court Palace. Stone mullioned windows, exposed during plaster removal, punctuate the thick walls throughout the building. The roof structure incorporates early oak trusses, particularly in the north section, where rough adze marks on the timber speak to centuries-old construction techniques. Originally entered through a great stone arch at the western end, still visible in the facade, the castle may have had its original stairwell near what later became a dumb waiter shaft.