The castle was rebuilt many times, but fragments of the castle, relating to the 13th century, it is easy to distinguish by the darker color.
Around 1520, at the time of Emperor Charles V, the castle was completely rebuilt. At that time, Het Steen gained its current name. The building belonged to Emperor Charles V. So, it was known first as "s Heeren Steen" (the King's stone castle), and later simply as "Het Steen" (the stone castle). The Dutch word "steen" means "stone", and is used for "fortress" or "palace", as in the "Gravensteen" in Ghent, Belgium.
From 1303 to 1823 the building was used as a prison, later it was served as a home for disabled soldiers.
In 1862, Pieter Génard (historian and archivist of Antwerp, one of the founders of the Geographical Society of Antwerp) suggested to establish an archaeological museum in each province. According to him, Het Steen was the appropriate place in Antwerp. The city supported the proposal and under the direction of architect Kennes was held restoration of the fortress. August 14, 1864 the museum was open.In 1880s, the city administration has decided to expand the river to prevent it from silting, and build a new, more spacious waterfront. The project was accepted, and soon was demolished much of the castle and about 500 ancient buildings. Yet once in 1889-1890, a new wing and other parts of the building were remodeled.
At the entrance bridge to the castle is a statue to Lange Wapper, folklore character of Antwerp.
The legend of Lange Wapper started in the 16th century. Lange Wapper liked to live near the sea, near rivers or canals. He could make himself so tall that he could move from one town to another with a single giant leap. Lange Wapper used tricks to approach women to get their breast milk. He teased drunks, cheated while playing with children and laughed like the devil.
In 1952, the archaeological museum in Het Steen was replaced by the National Maritime Museum, which however immediately from 1953 until 1958 was inaccessible because of the renovation work. The doors of this museum eventually were closed on December 28, 2008. From 2010, the collection of the museum was transferred to the nearby built Museum Aan de Stroom (Museum on the River).
In 2012, the building was redeveloped to "Het Steen the wise", "active house which invites thinkers, dreamers and doers".