How is the distance of a star from Earth calculated in the HYG star database The HYG star database can be found here

From the hygdata_v3.csv file, how are the cartesian (x,y,z) coordinates calculated? (Is there a mathematical formula for this?)

And in the dso.csv file, there aren't any x,y,z coordinated. How can I calculate that?
The transformation to $x,y,z$ coordinates, starting from the RA, Dec and a distance (as given in the HYG catalogue) is a simple trigonometric exercise, since the $x,y,z$ (in this case) is referred to a coordinate system aligned with the equatorial coordinate system.

$$x = d \cos({\rm RA}) \cos({\rm DEC}),$$
$$y = d \sin({\rm RA}) \cos({\rm DEC}),$$
$$z = d \sin({\rm DEC}),$$
where $d$ is the distance. How that distance has been obtained is not clear from the README file accompanying the database. It may have been by (incorrectly) inverting Hipparcos parallaxes.

The dso.csv appears to be a catalogue of deep sky objects. There is no information in the catalogue itself which allows you to calculate or estimate a distance.

You could try cross-referencing the catalogue against the  NASA/IPAC Extragalactic Database (NED). https://ned.ipac.caltech.edu/
If the recession velocity there is bigger than about 1000 km/s, then you will get a rough distance by using Hubble's law
$$ d = v/H_0$$
 with $H_0 \sim 70$ km/s per Mpc (or whatever your preferred value is). If the recesson velocity is smaller than this then the object is too close for Hubble's law to be valid.

e.g. The first line inthe catalogue appears to be a galaxy known as IC 5370, that has a recession velocity of 10372.00 km/s (no error bar given and I doubt any more than 4 significant figures is warranted and in any case $H_0$ is not known accurately to more than 2 significant figures), which gives an approximate distance of 150 Mpc.