Hrubá Skála castle was founded sometime before 1353 by Hyněk of Valdštejn, one of the many descendants of Zdeněk of Valdštejn, the founder of the dynasty. All were members of the Markvart family which controlled most of the region in the 13th and 14th centuries.

The castle changed hands several times from the late 14th century onwards, including being confiscated by the King, on at least two occasions. In the early 16th century the castle was acquired by the Smiřický family which converted the gothic building into a Renaissance chateau.

The Smiřický family were one of the leading Protestant houses to rebel against the Catholic emperor in 1618, which led to the start of the 30 Years War. After the battle of White Mountain, which saw the Protestant cause defeated, the Smiřický estates were put under the control of leading Catholic generalissimo Albrecht of Valdštejn, who was related through his mother to the Smiřický family. Albrecht later became the owner of the properties, adding Hrubá Skála to the vast estates he acquired across northern Bohemia, including the old family seat of Valdštejn.

Hrubá Skála stayed in the Valdštejn family, despite the fall from favour and assassination of Albrecht in 1634. The castle suffered damage throughout the 30 Years War, particularly by the occupying Swedish forces, and also was damaged by fire at least twice over the following hundred years. Finally in 1821 the castle was sold by the last Valdštejn owner to the Lexa of Aehrenthal family, which converted the castle to its current neo-gothic appearance.

The castle was confiscated by the state under the Beneš decrees in 1945, and is now a private Castle hotel. The courtyards remain open to the public, and are worth visiting. There is also a castle restaurant with a fine terrace.