Doesn't the concept of marginal utility speak to a cardinal utility function? When we differentiate the utility function with respect to some input $x_i$, we get a number that tells us how "fast" the utility function is changing at some point with respect to $x_i$. Doesn't that mean that when we compare marginal utilities, we are comparing with something that is based on the structure of the numeric output of the utility function, which shouldn't be the case since it's ordinal?

I know I'm missing something here intuitively, but I can't seem to figure out what it is.
To the pure ordinalist who believes that preferences are purely ordinal, the concept of marginal utility (MU) has no meaning. (And a fortiori, the concept of diminishing MU also has no meaning.)
However, the concept of the marginal rate of substitution (MRS) does have meaning.
In the course of our work, we may compute something that we call MU. But to the pure ordinalist, any such number found has no meaning in and of itself.

Example. Say an individual's preferences $\succsim$ over two goods $A$ and $B$ can be represented by the utility function $U:(\mathbb{R}^+_0)^2\rightarrow\mathbb{R}$ defined by $$U(A,B)=AB.$$
The intermediate microeconomics student may then carry out these computations:
$$MU_A=\frac{\partial U}{\partial A} = B.$$
$$MU_B=\frac{\partial U}{\partial B} = A.$$
$$MRS = \frac{MU_A}{MU_B}=\frac{B}{A}.$$
The above says that if, for example, my current bundle is $(A,B)=(200,1000)$, then  $$MU_A=B=1000\text{ and }MU_B=A=200.$$ However, these two numbers have no meaning whatsoever.
The only number that has meaning is the MRS: To get another unit of $A$, I'm willing to give up (approximately) $$MRS=\frac{B}{A}=\frac{1000}{200}=5\text{ units of }B.$$
To the pure ordinalist, the above reasoning is completely legitimate, so long as one assigns meaning only to MRS. What's illegitimate is to assign any meaning to $MU_A=B$ or $MU_B=A$.
The pure ordinalist knows that if $\hat U$ is a strictly increasing transformation of $U$, then $\hat U$ is also a utility representation of $\succsim$. So, for example, if $\hat U:(\mathbb{R}^+_0)^2\rightarrow\mathbb{R}$ is defined by $$\hat U(A,B)=2AB,$$ then $\hat U$ also represents $\succsim$.
However, with $\hat U$, our computations seem to differ slightly from before:
$$M\hat U_A=\frac{\partial \hat U}{\partial A} = 2B.$$
$$M\hat U_B=\frac{\partial \hat U}{\partial B} = 2A.$$
$$\hat{MRS} = \frac{M\hat U_A}{M\hat U_B}=\frac{2B}{2A}=\frac{B}{A}.$$
Any conclusions we arrive at with the new utility representation $\hat U$ are the same as before.
If again my current bundle is $(A,B)=(200,1000)$, then  $$M\hat U_A=2B=2000\text{ and }M\hat U_B=2A=400.$$ However and again, these two numbers have no meaning whatsoever.
The only number that has meaning is the MRS: To get another unit of $A$, I'm willing to give up (approximately) $$\hat{MRS}=\frac{2B}{2A}=\frac{2000}{400}=5\text{ units of }B.$$
The quantity MU by itself has no meaning. Confusion arises only when one attaches meaning to MU and wonders how for example it is that $$M\hat U_A = 2MU_A,$$ and what the above equation means. (Answer: It means nothing.)
Out of convenience, the intermediate microeconomics student will often compute something called $MU_A$ and $MU_B$ and these can often be evaluated as actual numbers. But on their own, these numbers have no meaning (to the pure ordinalist). Only the ratio of the two quantities has any meaning: $$MRS=\frac{MU_A}{MU_B}.$$

Some quotes. Hicks (1939):

We have now to undertake a purge, rejecting all concepts which are tainted by quantitative utility, and replacing them, so far as they need to be replaced, by concepts which have no such implication.
The first victim must evidently be marginal utility itself. If total utility is arbitrary, so is marginal utility. ...
The second victim (a more serious one this time) must be the principle of Diminishing Marginal Utility. If marginal utility has no exact sense, diminishing marginal utility can have no exact sense either.

Dittmer (2005, emphasis added):

Many introductory microeconomics textbook authors derive the law of  demand from the assumption of diminishing marginal utility. Authors of intermediate and graduate textbooks derive demand from diminishing marginal rate of  substitution and ordinal preferences. These approaches are not interchangeable;  diminishing marginal utility for all goods is neither a necessary nor sufficient condition for diminishing marginal rate of substitution, and the assumption of diminishing marginal utility is inconsistent with the assumption of ordinal preferences.