do rockets follow a straight path but is looks curved because of space time? Does the path of a rocket or anything going straight up look curved because the mass of the earth is curving space-time?

If we assume the earth, the rocket and all the celestial bodies whose gravitational field reaches the earth to be massless then will the rocket follow truly a straight line, because there would be nothing curving the space-time?
This is partly space exploration and partly physics.

Rockets do not normally go straight up. They start off going up, to get into thinner air, then they steer to turn themselves eventually to a horizontal path, until they are going fast enough to stay in orbit.

In general relativity, an object that is free to move in the Earth's gravitational field will appear to accelerate towards the ground as it is following a path (called a geodesic) in curved spacetime. This is not specific to rockets; a cricket ball will follow a curved path when thrown due to the curvature of spacetime around the Earth. And if one is far  from any gravitational field, then a cricket ball will travel in a straight line. 

To directly answer your question: The path of a rocket going straight up will not look curved, it will look like it is going straight up.