Barkla was born in Widnes, England, to John Martin Barkla, a secretary for the Atlas Chemical Company, and Sarah Glover, daughter of a watchmaker.
Barkla studied at the Liverpool Institute and proceeded to Liverpool University with a County Council Scholarship and a Bibby Scholarship. Barkla initially studied Mathematics but later specialised in Physics under Sir Oliver Lodge. During the absence of Oliver Lodge due to ill health, Barkla replaced him in lectures.
In 1899 Barkla was admitted to Trinity College, Cambridge, with an 1851 Research Fellowship from the Royal Commission for the Exhibition of 1851, to work in the Cavendish Laboratory under the physicist J. J. Thomson (discoverer of the electron). During his first two years at Cambridge, under the directions of Thomson, Barkla studied the velocity of electromagnetic waves along wires of different widths and materials.
After a year and a half at Trinity College, Cambridge, his love of music led him to transfer to Kings College, Cambridge, in order to sing in their chapel choir. Barklas voice was of remarkable beauty and his solo performances were always fully attended.  He completed his Bachelor of Arts degree in 1903, and then his Master of Arts degree in 1907. He married Mary Esther Cowell in the same year, with whom he had two sons and one daughter.
In 1913, after having worked at the Universities of Cambridge, Liverpool, and King's College London, Barkla was appointed as a Professor of Natural Philosophy at the University of Edinburgh in 1913, a position that he held until his death.
Barkla made significant progress in developing and refining the laws of X-ray scattering, X-ray spectroscopy, the principles governing the transmission of X-rays through matter, and especially the principles of the excitation of secondary X-rays. For his discovery of the characteristic X-rays of elements, Barkla was awarded the Nobel Prize in Physics in 1917. He was also awarded the Hughes Medal of the British Royal Society that same year.
Barkla proposed the J-phenomenon as a hypothetical form of X-ray behaviour similar to X-ray fluorescence but other scientists were not persuaded that this was a different mechanism from other known effects such as Compton scattering and so the theory was not successful.
From 1922 to 1938 he lived at Hermitage of Braid in south-west Edinburgh.
He died in Edinburgh on 23 October 1944.