You forget that the work done on the book is simply the applied force multiplied by the distance the object has travelled. In this way of defining work, the gravitational force does indeed do work on the book, it just does negative work, i.e. opposes the motion, which is why you have to do extra work to make up the deficit, i.e. lift harder against gravity.