The category of topological monoids can be made into a topological category in a naive way. Namely, the space of all continuous homomorphisms between two topological monoids is a closed subspace of the space of all continuous maps between the underlying topological spaces.


My question is that, regarding homotopy theory, is this a "good" construction? More precisely, if we take the homotopy category of this topological category, which is a category enriched over the category of homotopy types, does it give the "correct" homotopy theory?


More generally, consider all the topological operads (with a single color). There is also a naive way of making the category of all topological operads into a topological category, similar to the above one.
Does the homotopy category of this topological category behave as we would expect?



Topological category of topological monoids / operads
I don't think that there's necessarily a right answer to this question. Any category with spaces of maps like you describe has a homotopy category, as well as a lot of other attendant structure. You have to decide what you are interested in.


Let's step back from topological monoids and just talk about topological spaces. There are two different homotopy categories in common usage: the ordinary homotopy category of topological spaces, and the weak homotopy category where you invert maps that are isomorphisms on all homotopy groups. In theory, the first category is a much stronger invariant of the category of topological spaces. In practice, we can say an awful lot more about the second category: it's more amenable to descriptions from algebra; it's equivalent to the homotopy category of CW-complexes (which encode most of the spaces that we are interested in on a day-to-day basis); it's also equivalent to the homotopy category of simplicial sets. Whether you're interested in the ordinary homotopy category or the weak homotopy category depends a lot on whether you're interested in pathological spaces.


There is an exactly analogous question for topological monoids (or topological groups). We could either construct the ordinary homotopy category that you describe, which is a strong invariant, or we could build the weak homotopy category where we invert maps $M \to N$ of topological monoids that are weak equivalences on the level of spaces. These are very different homotopy categories. As an example of this, if you restrict your attention to topological *groups* then the homotopy theory of topological groups is equivalent to the homotopy theory of pointed connected spaces, by the correspondence $G \leftrightarrow BG$. The space $BG$ doesn't remember a lot about the strict group-level structure, such as identities that are satisfied by the multiplication in $G$.


When we looked at this question for topological spaces, it was a little easier to make the decision because most topological spaces of interest are equivalent to CW-complexes, which are built up freely from basic building blocks. The analogues of CW-complexes are topological monoids that don't really arise in nature very often: they include free groups, but not Lie groups. This means that the weak homotopy category of topological monoids ignores a lot of the features that one might be interested in. However, it is a lot easier to build things there: homotopy colimits, homotopy limits, Postnikov towers, and so on. As a result, what you pick probably depends on the applications you have in mind.


(Similar remarks apply to operads.)


