So it actually turns out, when you think of it that way, that potential energy of any form, but especially gravitational potential energy-- and we'll see electrical potential energy-- it's always in reference to some other point, so it's really a change in potential energy that matters. 