Karl Koller began his football career with ASK Blumau. 1949 he went to First Vienna FC where he made his championship debut in 1949. Througout his professional career he played for this club. He played 414 league matches for Vienna and scored 39 goals. Overall he played 836 matches and scored 101 goals. In 1955 he became Austrian Champion and 1961 they played in the Cup final which they lost 3-1 versus SK Rapid.