Hysaj was born in Albania and started youth career with KF Shkodra, a local team of Shkodër city, but joined his father in Italy at the age of 14, where he joined Empoli's youth setup in 2009, before going on to make his debut for the club in 2011 in the Coppa Italia.
Hysaj blev født i Albanien og startede sin ungdomskarriere med KF Shkodra, et lokalt hold i Shkodër city, men sluttede sig til sin far i Italien i en alder af 14 år, hvor han kom til Empoli's ungdomsopsætning i 2009, inden han fortsatte med at lave sin debut for Klub i 2011 i Coppa Italia.