Could asteroid mining affect gravitational balance? I'm just a guy who is interested in all topics around space, so forgive me if it is a dumb question.

I have read some things about asteroid mining in the past and I think it's a really interesting topic, as it opens up a lot of possibilities.

But one thing came to my mind. Humans tend to pushing things a little too far sometimes when we want to reach a goal. So, imagine, we are getting really good at off-earth mining at a larger scale and in a larger timespan, we continue to mine thousands and thousands of asteroids. Wouldn't that affect the gravitational balance of our star system at some point, or is the gravitational influence of an asteroid belt negligible? I am talking about extreme examples, like removing nearly the whole asteroid belt beyond Mars. Also, what would happen to all the stuff we don't need from an asteroid?

I'm sure I'm not the first one who thinks of something like this, but I found nothing about this particular topic.
Something to remember is that the total mass of asteroids in the asteroid belt is tiny compared to the masses of the planets. Ceres, the largest body in the asteroid belt by a wide margin, is only 0.00015 times the mass of Earth, Ceres itself accounts for about a third the belt's mass:

Asteroid belt facts from ScienceDaily

I haven't been able to find any papers or articles that suggest the residents of the asteroid belt have any measurable perturbation on the orbits of Jupiter and Mars (the other way around is a different story!). If the asteroids that spend most of their time near Mars and Jupiter were to disappear, I doubt there would be any observable effect on the orbits of the planets.

Consider that comets and asteroids commonly pass close to Earth and the Moon and their influence on Earth's orbit is negligible.

As for what we would do with what we don't need from an asteroid, there are a few options I could think of based on the size of the target.
With smaller asteroids, if there is a risk that any changes to its orbit (such as an impact or accidental explosion) that would send it close to Earth, a complete controlled demolition of it might be warranted, or simply attach large chemical rockets to it and push it away. In the near-future, it is likely that even if we do start mining asteroids, provided we haven't dangerously altered its orbit, they will simply be abandoned afterwards.