Castle Fraser is a grand baronial tower house built from local granite and dates back to the 15th century. The building has been altered on a number of occasions throughout its history by its various occupants.

​Since 1976, when The National Trust for Scotland took over the castle, major work has taken place in the walled garden. A new rhododendron walk and woodland garden have also been created.