  We present a high-contrast imaging survey of intermediate-mass (1.75--4.5 $M_\odot$) stars to search for the most extreme stellar binaries, i.e., for the lowest mass stellar companions. Using adaptive optics at the Lick and Gemini observatories, we observed 169 stars and detected 24 candidates companions, 16 of which are newly discovered and all but three are likely or confirmed physical companions. Despite obtaining sensitivity down to the substellar limit for 75\% of our sample, we do not detect any companion below 0.3 $M_\odot$, strongly suggesting that the distribution of stellar companions is truncated at a mass ratio of $q_\mathrm{min} \gtrsim0.075$. Combining our results with known brown dwarf companions, we identify a low-mass companion desert to intermediate mass stars in the range $0.02\lesssim q \lesssim0.05$, which quantitatively matches the known brown dwarf desert among solar-type stars. We conclude that the formation mechanism for multiple systems operates in a largely scale-invariant manner and precludes the formation of extremely uneven systems, likely because the components of a proto-binary accrete most of their mass after the initial cloud fragmentation. Similarly, the mechanism to form "planetary" ($q \lesssim 0.02$) companions likely scales linearly with stellar mass, probably as a result of the correlation between the masses of stars and their protoplanetary disks. Finally, we predict the existence of a sizable population of brown dwarf companions to low-mass stars and of a rising population of planetary-mass objects towards $\approx 1\,M_\mathrm{Jup}$ around solar-type stars. Improvements on current instrumentation will test these predictions. 
A low-mass companion desert among intermediate-mass visual binaries: The   scaled-up counterpart to the brown dwarf desert