Under what situations can an aperture mask improve the resolution of a small/medium amateur telescope? Is this demonstrable mathematically? @antlersoft's answer describes some of the challenges to seeing any details in the small disk of Mars in small amateur telescopes. In the case of reflecting telescopes, it mentions the use of either an off-axis parabolic mirror, or an off-axis aperture mask on a standard Newtonian to produce a clear unobstructed view. 

The answer links to a short article on the amateur telescope site BBAstroDesigns which is titled Off-Axis Masks: Light Reduction and Resolution

If I understand the explanation correctly, besides the light reduction function, the off-axis masking excludes the secondary mirror and support, thereby removing diffracted light.

I'm a bit surprised that for amateur viewing on Earth that diffraction from a Newtonian or Cassegrain's secondary mirror and supports would really have any impact on resolution of small extended objects like Mars. The example in the article is a 16 inch telescope. I would have thought that seeing would be the limitation, but maybe for bright objects a smaller diameter (1/3 to 1/6 of the full aperture) is actually better? 

Is it possible to explain the physics/optic behind the use of off-axis apertures to improve resolution as discussed in these links?

If possible, can a bit of math be used to demonstrate this quantitatively and convincingly?
The vast majority of people - almost everyone, really - should not rely on this on a daily basis. The chances are overwhelming that they will reduce the performance of their instruments. There are so many variables involved, and it depends on so many factors, a likely result is worse performance.

The link is to a site maintained by Mel Bartels. Mel is a very experienced observer and telescope builder, with hundreds if not thousands of hours behind the eyepiece. He has enough practical experience so that, just by looking at the overall image, and the shape and size of the Airy disk and diffraction figure, he can tell exactly in what way the telescope's behavior is affected by changes.



The effects of the secondary mirror and spider are a topic which is much misunderstood. You will encounter many strong opinions on this topic with not much basis in reality.

The spider vanes, if straight, will each create two spikes, diametrically opposed, perpendicular to the length of the vane. So a 3-vane spider makes 6 spikes. A 4-vane spider makes 8 spikes, but coincidental two by two (look like 4 brighter spikes). If the vanes are curved, then the diffraction figure is more like a faint, diffuse cloud.

So using an aperture mask can remove the spikes due to the spider vanes. But will it make the image "better"? Spikes don't matter from a practical perspective. If a spike is blocking one of the components of a double star, just rotate the scope. And the theoretical resolving power is diminished. You are, after all, using a smaller aperture.

The secondary mirror will increase the size of the diffraction figure. The Airy disk and the first minimum become larger in diameter - and so do the rest of the rings. This is a matter of much confusion, with some claiming that obstructed telescopes (scopes with a secondary mirror) are essentially inferior to unobstructed designs - this is demonstrably false.

Simple rule of thumb: if D is the diameter of the primary, and d is the diameter of the secondary, then the performance of an obstructed scope is equivalent, from the p.o.v. of resolving power (high frequency components), to an unobstructed instrument of aperture D - d. This is not an exact formula but it's close enough. A 300mm reflector with a 60mm secondary has about the same fine detail performance on planetary observations like an unobstructed 240mm instrument.

There are cases when you're observing close double stars, and it would be great if one of the stars could fall into the diffraction minimum of the other, brighter star, just outside the Airy disk. Sometimes you can do that with an aperture mask.

The article by Mel Bartels makes an important point, and I wish he emphasized it more: the aperture mask should only be used at low or medium magnification. In other words, if you're planning anyway to use low or medium magnification, then in some rare cases the aperture mask might be useful. It NEVER makes sense to use an aperture mask at high magnification - it's like pushing a telescope above its maximum useful magnification, and the result is just a blurry mess.



The linked article also mentions the issue of brightness. Some observers claim that the Moon is too bright in a telescope. Mel Bartels sees this as an opportunity to use an aperture mask (again, NOT at high magnification). Others use Moon filters - neutral density filters that reduce the incoming flux of light.

There is a third school of thought here. Do not become dark-adapted while observing the Moon. Do not observe the Moon from a dark place. Observe it with lights turned on around you, like from the sidewalk of a well-illumined street. This is what I, and others like me, do. Suddenly the Moon is not too bright anymore. The telescope is not performance-limited by the aperture mask, and you can zoom in at high magnification to observe small details of craters and mountains. And your eyes are working in their optimal regime, when the pupil is not fully dilated.

Surface brightness of extended objects never increases in a telescope, it just seems to increase. In fact, it always decreases. The Moon seems too bright because you're dark-adapted - but in reality it's quite pale.



There is another case when the image is demonstrably improved by an aperture mask. This is when the primary mirror has strong defects near the perimeter, like TDE (turned down edge), which is a defect both quite common (for mass-manufactured instruments) and very damaging to performance. A scope with strong TDE has poor resolving power and poor high frequency contrast. Sometimes it's so bad that even an aperture mask smaller than D/2 will improve performance.

The solution here is not the aperture mask, but instead masking off the perimeter. Make a ring with an inner diameter somewhat smaller than the aperture and place it permanently on the mirror, so as to block off the defective edge. Aperture is somewhat diminished, but now the optics are good across the whole surface.



One more case: let's assume seeing is so bad, the image is shaking like a giant, blurry pile of jelly. There's no way you're going to use high magnification anyway. You operate at low-to-medium magnification. Now apply the aperture mask. Is the image better?

Objectively the answer is no. If you're doing lucky imaging with a camera, it's still better to leave aperture alone, and by luck you will capture a few frames where detail resolution is pretty good.

But the human visual system is weird. Some people prefer to let go of the higher performance ceiling, instead opting for an image that's perceived as more smooth and stable. Personally I prefer to let low magnification do its thing, and leave aperture alone, but not everyone is the same.

Even visually, if you don't mask the aperture, sometimes seeing gets better briefly, and you can see small details for a while. With an aperture mask, you will miss those.

But this is a subjective thing, it's not something that can be objectively measured, except as a study of the psychology of visual perception.



If you want to experiment with the aperture mask, by all means go ahead and do it. But be aware of the many, many pitfalls here. It's not a silver bullet. It's not the solution to all of humanity's problems. It's just a trick that sometimes works, in a fairly narrow slice of use cases.