Consider a rigid braided monoidal category, with braiding $\beta\_{x,y} : x \otimes y \cong y \otimes x$, and every object has a dual such that $\epsilon\_x : 1 \to a \otimes a^\*, \bar\epsilon\_x : a^\* \otimes a \to 1$ satisfying the zig-zag identities. Now we have the Reidemeister moves, e.g.
$$ (c \otimes \beta\_{a,b}) \circ \beta\_{a\otimes b, c} = \beta\_{b \otimes a, c} \circ (\beta\_{a,b}\otimes c) $$
saying that braiding the two string with another, and then braiding the two, is the same as first braiding the two and then with the other. This is Reidemeister III. Similarly $\beta\_{a\otimes a^\*, b} \circ (b \otimes \epsilon\_a) = \epsilon\_a \otimes b$ is Reidemeister II, and $(\bar\epsilon\_{a} \otimes a) \circ (a^\* \otimes \beta\_{a,a}) \circ (\epsilon\_a\otimes a) = \mathrm{id}\_a$ is Reidemeister I. This is rather like the [Morse link presentation](http://katlas.math.toronto.edu/wiki/MorseLink_Presentations) of knots and links.


Is there any reference that develops such an idea? How can the theory of quandles be integrated in this picture? I'm guessing that quandles either behave like modules over a knot category, or we have a joint generalization of quandles and knot categories.



Is there a notion of "knot category"?
To expand on my comment, this connection is indeed well-known and the key concept is that of ribbon category. A standard textbook reference is Turaev, Quantum Invariants of Knots and 3-Manifolds.


Braided and rigid is not enough to get links invariants, because RI will not hold in general (and in fact pretty much never). A nice exposition of that issue can be found in Selinger, A survey of graphical languages for monoidal categories (<https://arxiv.org/abs/0908.3347>) (in that reference autonomous means rigid and tortile means ribbon).


Any ribbon category provides an invariant of framed, oriented links for each object $X$. If $X$ is simple then the ribbon element $\theta\_X$ acts as a multiple of the identity so that you can renormalize to get an invariant of oriented links. The choice of an isomorphism $X\cong X^\*$, provided one exists, gets rid of the orientation.


If you want non trivial invariants of oriented links on the nose, you'd need that $\theta\_X=id\_X$ and $\theta\_{X\otimes X} \neq id\_{X\otimes X}$. While this isn't impossible (tautologically, the category of oriented tangles is a ribbon category which satisfies this for example) this is a pretty unnnatural condition.


