He lost power when responsible government was instituted in 1848 but continued as leader of what became the Conservative Party and served as Premier from 1857 to 1860 and again from 1863 to 1864 before being appointed to the bench.