The castle was built in the 14th century by the Piast family, the rulers of the Cieszyn Duchy in Poland, the castle was used as one of their residences for over two centuries. In 1752 the Bielsko state became a duchy owned by the Sulkowski family. The castle was their property until 1945. After World War II the castle was taken over by the Polish State. Since 1983 its only owner has been the Bielsko-Biała Museum.