Although little of the original castle is still evident, it retains many features that hint at its ancient past. The dwelling which stood on the site over 900 years ago was thought to be a well fortified peel tower of the type common in Northumberland. Sir Robert de Ogle was given permission to crenelate the manor house in 1341 by Edward III, since when it has been known as Ogle Castle.

In 1346 King David II of Scotland was taken prisoner at the Battle of Neville's Cross and brought to Ogle, where he was held until taken to the Tower of London. The small window of the room in which he was imprisoned can be seen on the second floor of the house in what was formerly a tower.

Around 1640 it is thought that Scottish troops were quartered in the castle at which time a great deal of damage occurred. The present house is most likely the south-west portion of the old castle, the great number of fireplaces indicates that it was probably used as a hunting lodge . The castle was once surrounded by a double moat, the remains of which can be found in the small woodland to the west of the building.

By 1900 the buildings were in a bad state of repair and were used for farm purposes for about 20 years. In recent times a certain degree of modernisation has been carried out, but as far as possible every endeavour has been made to preserve the original atmosphere, the building being now scheduled as an ancient monument.