As it decayed, it became known derogatively by locals as "The White Elephant" in the 1940s and by the 1950s, like many buildings in Melbourne of that time it was earmarked for replacement by office blocks.