Q:

NSImageView always displays on top of any other kind of view

I'm working on a database application that has a graphical editor that is very similar to Interface Builder. Unlike IB, however, this editor can be flipped from editor mode to live mode, where the user interface is fully operational (buttons can be clicked, text edited, etc.) To do this, the graphical editor uses standard Appkit interface classes -- NSButton, NSTextView, etc. The editor itself is implemented with a custom subclass of NSView. All of the user interface elements are subviews of this custom NSView, new elements are added using the addSubview: method, making the new element the topmost visible element (note -- the views are not layer-backed, just regular views). The user can also use Bring-to-Front and Send-to-Back commands to change the ordering of the subviews. This movie shows two overlapping NSButton elements (for illustration purposes, of course normally you would never overlap them), and how the program can re-arrange the subview to change the Z-order of the user interface elements.

The problem is, this works with every kind of interface element except NSImageView. In the movie before there are two elements, an NSButton and an NSImageView. The NSButton is actually "on top" the whole time, the NSImageView element should appear behind the button, but no matter the order of the subviews, the NSImageView always appears on top.

If there are two overlapping NSImageView objects, the visible stacking order between them is unpredictable, but they will always appear above all other objects, no matter what the order of the subviews is.
A possibly useful clue is that if I implement my own custom view that draws an image directly in it's drawRect: method, that works fine. So that's one possible solution, but I am reluctant because that means re-implementing a large swath of useful features that NSImageView normally takes care of, some of which are quite complex like supporting animated GIF display. Other than this layering/z-order issue, everything else about NSImageView works fine.
Perhaps NSImageView is using layer-backing without my asking for it, so it isn't mixing in properly with my other objects? I can't find any documentation that indicates this. I am not linking against the QuartzCore framework.
Here is the code that adds the NSImageView element as a subview to the graphic editor view.
- (void)objectDidAppearBelow:(NSView *)nextView
{
    FormView * formView = [FormWindowController currentFormView]; // get view element will be placed into
    NSScrollView * imageContainer = [[NSScrollView alloc] initWithFrame:insideBorderRect];
    ImageView * ixView = [[ImageView alloc] initWithFrame:[self insideFormObjectBorder:objectRectangle]];
    [ixView setOwnerObject:self];
    [imageContainer setDocumentView:ixView];    
    [imageContainer setAutoresizesSubviews:YES];
    [shapeView addSubview:imageContainer Below:nextView];
    imageDocumentView = ixView; // save weak reference to image view so it can be manipulated
}

In other places there is nearly identical code for NSButton (in several variations for push buttons, radio buttons, etc.), NSTextView, NSTableView (for lists and matrixes), NSSlider, NSScroller, NSSegmentedControl, even WebView. All the others work correctly with overlapping objects, including WebView, only NSImageView doesn't work as expected.
For my reference this is #429 in the Panorama X issue tracker.

A:

I discussed this problem with Apple engineers at WWDC 2018. It turns out that as I suspected, in some cases Appkit will use layer backing for NSImageView even if you did not ask for it! So the best solution is to switch all views to layer backing (which will happen automatically with Mojave).
In this particular case the NSImageView was inside a NSScrollView, which I didn't mention because I didn't think it was important (my bad). It turns out, this is the case where Appkit thinks using layer backing would be a good idea (to optimize scrolling). So another way to fix this is to subclass NSImageView (which I had already done for other reasons) and add this method (written on the spot by an Apple engineer who prefers to remain uncredited). 
+ (BOOL)isCompatibleWithResponsiveScrolling {
    if (NSAppKitVersionNumber <= 1561. /* NSAppKitVersionNumber10_13 */) {
        return NO;
    } else {
        return YES;
    }
}

I was assured that this is all part of the public, documented API, though the documentation is minimal (surprise surprise). There is some discussion of responsive scrolling in the What's new in OS X 10.9 release notes. The check with NSAppKitVersionNumber is to make sure to turn off this patch when running on Mojave, since everything is layer backed.

