Thursday 31 January 2013

Effects of ISO on MTF50 measurements

How does the ISO setting affect my MTF50 measurements?

This question arose in the comments section of Roger Cicala's blog (specifically, this article). The short answer is: the expected MTF50 score (the mean, in other words) is unaffected by high ISO noise, but the reliability of MTF50 measurements decrease with increasing ISO settings.

The mtf_generate_rectangle tool included in the MTF Mapper package allows us to generate synthetic images of black rectangular targets on a white background, while specifying the exact MTF50 value we would like to obtain. In addition, we can simulate sensor noise either as plain additive Gaussian white noise, or using a more realistic (but still simple) noise model that incorporates three noise terms: read noise, photon shot noise, and pixel-response non-uniformity (PRNU).

To generate noise using the more realistic sensor noise model, we must provide three parameters: PRNU magnitude, read noise magnitude (standard deviation in electrons), and ADC gain (electrons per DN). I used Marianne Oelund's data (posted here) for the D7000 sensor. An example of the command to generate a realistic ISO 100 image using our simulated D7000 sensor would be:

./mtf_generate_rectangle -m 0.3 -l --b16  --pattern-noise 0.0085  --read-noise 3.7 --adc-gain 2.643  --adc-depth 12 -c 0.33

(Please excuse the --adc-depth 12 parameter. I know the D7000 has a 14-bit ADC, but specifying 12 here is a dirty hack to produce an output file that covers the full 16-bit range). I have verified that the statistics of a synthetic image generated using these parameters matches that obtained from an actual raw D7000 image (currently, I have only verified at ISO 100).

To generate a simulated image at ISO 400, you can use this command:
./mtf_generate_rectangle -m 0.3 -l --b16  --pattern-noise 0.0085  --read-noise 2.5 --adc-gain 0.641  --adc-depth 12 -c 0.33

Note that I have compressed the contrast of the simulated edge quite a bit (the -c 0.33 parameter) to avoid clipping at higher ISO values. This will increase MTF50 standard deviation values slightly --- I usually use -c 0.2 when generating ISO 100 images.

Ok, enough of the preamble. I decided on an MTF50 value of 0.3 cycles/pixel, which is close to the best value I have managed to obtain from the D7000. This corresponds to 979 line pairs per picture height (the units that Roger reports his results in). I generated 400 synthetic edges at the various simulated ISO settings; here is the resulting box plot:
Measured MTF50 values (lp/mm), with a target of 979 lp/mm
It is pretty clear from the plot that the median MTF50 value remains largely unchanged over the whole ISO range, only dipping slightly at ISO 12800, but this may simply be a sampling artifact given the large standard deviation.

This graph tells us two important facts: 
  1. The mean MTF50 value (over many experiments) is unaffected by ISO setting, and
  2. The standard deviation of MTF50 values increases with increasing ISO setting. In other words, if you are forced to use ISO 1600, you will have to take many more measurements to obtain a reasonable estimate of your mean MTF50 value.
The same results are presented below in tabular form:


ISO Mean MTF50 (lp/ph) Std.dev
100 980.0 10.1
200 979.7 12.6
400 979.2 16.7
800 978.8 22.4
1600 979.1 32.1
3200 979.1 45.2
6400 979.2 66.0
12800 976.6 91.8

I did perform a quick Shapiro-Wilk normality test, and the data definitely have a Gaussian distribution within each ISO category, so you can go ahead and compute confidence intervals using n=400 and the values in the table above.

If you want to perform MTF50 testing at ISO 800, I would recommend that you compute the trimmed mean using only the middle 50% of the data (around the median). Put differently, at ISO 800 you could capture twice as many MTF50 test charts, and compute your mean using the middle 50%, which should yield comparable results to performing the test at ISO 100.

Even so, you should find that about 80% of your measurements will have an error of less than 29 lp/ph at ISO 800, which amounts to only 3%. I would be willing to bet that other errors (quality of your test chart, vibrations, etc.) will probably be on a similar magnitude, so I would not really worry too much about performing MTF50 measurements at ISO800.

Why is MTF50 not affected by ISO?

These results go counter to any subjective evaluation of sharpness at higher ISO settings. The reason is fairly simple: the slanted edge method used to compute the MTF50 measurements applies fairly heavy noise suppression internally. 

Specifically, the slanted edge method constructs an oversampled edge profile across the edge being measured. This effectively involves computing the mean image intensity along lines running parallel to the edge. Since this mean is unweighted, we obtain maximal suppression of Additive Gaussian White Noise (this is a known property of the unweighted mean). Because we are averaging along lines parallel to the edge, we expect the signal level (intensity) to be constant, so unweighted averaging gives us the maximum likelihood estimate of a constant with additive Gaussian white noise. The result is that the sensor noise (aggravated by higher ISO settings) is filtered out quite effectively.

The moral of the story is that the slanted edge method does exactly what it is supposed to do: measure the modulation transfer curve (MTF) of the optical system. This is a property of the optical system that is independent of sensor noise. It does lead to some confusion, though, since subjective evaluation of noisy high-ISO images definitely create the perception of reduced sharpness. This implies that we must appreciate the difference between an MTF estimated using the slanted edge method, and the (subjective) human perception of sharpness which is sensitive to high-ISO noise.

Why we see increased variability at high ISO

We have seen some evidence (constant mean MTF50 with increasing ISO from the experimental results above) that MTF50 is not affected by typical sensor noise, and we have a plausible mechanism to explain why we should expect slanted edge MTF50 values to be unaffected by increasing sensor noise. Despite this, we observe an increase in MTF50 standard deviation, which could only mean that on some high ISO runs we obtain MTF50 values that differ a lot from the mean MTF50 value.

I can think of two possible explanations:
  1. The slanted edge method cannot suppress the noise completely, thus at higher noise levels we will see some errors creeping in (because the edge profile becomes noisy).
  2. The slanted edge method depends on accurate estimation of the edge orientation. This becomes harder to achieve when the image is noisy. MTF Mapper employs three different techniques to try and improve the accuracy of edge orientation estimates (least-squares fitting to the edge, joint estimation of parallel edges, and systematic fine-tuning to reduce local edge profile variance), but orientation errors do tend to increase with increasing image noise.

Appendix

Of course, people would like to see what the simulated images look like, so here is a selection:
 
ISO 100
ISO 400
ISO 800
ISO 1600
ISO 3200
ISO 6400
ISO 12800

A few comments are in order. These images are simulations of the green channel (which is the channel from which Marianne's data was derived). Also note that the images are displayed here in sRGB space as 8-bit files, but that the actual measurements were performed on their linear 16-bit counterparts. CFA demosaicing may actually affect your MTF50 measurements, but this should not have a huge impact unless the lens has severe spherical or chromatic aberration, so I chose to work only with the green channel for simplicity.

4 comments:

  1. Hi Frans:

    I understand that as image data gets noisier one needs more MTF50 measurements.

    1) Could one simply feed a longer edge to MTFMapper once to obtain the same result? For instance, would feeding a 1000px edge to MTFMapper be equivalent to taking the average of 10 measurements of a 100px edge?

    2) I have noticed that when taking MTF50 measurements off horizontal and vertical edges at high ISOs (say 3200+) the readings tend to diverge: that is the horizontal will get lower while the vertical will get higher - or the other way around. The average of the two sometimes stays the same. Why the different behavior of the two orientations? See an example for instance here http://i.imgur.com/nb0edv4.png

    Great work!
    Jack

    ReplyDelete
    Replies
    1. Hi Jack,

      Thanks for the kind words!

      In response to your first point: Yes, you could reduce the effects of noise by simply measuring along a longer edge. I would expect that one 1000px measurement would be equivalent to the mean of 10 measurement of 100px each, but I do not think this is guaranteed. For example, it could be that the 100px edges have a specific bias (rather than random error) induced by errors in the edge orientation estimates.

      In practice I have found that longer edges are affected by geometric lens distortion. A typical barrel distortion will cause your step edge to bow; the longer the edge, the more obvious the curvature becomes (and the more deleterious to MTF measurements).

      With regards to your second point: I have not explored this phenomenon myself. The mtf_generate_rectangle[.exe] tool should help here; one can generate synthetic images with the desired orientation (and simulated high ISO noise) to check whether this is a problem with mtf_mapper[.exe], or something specific to a particular camera, or merely coincidence :)

      Delete
  2. Ah, very good point about lens distortion, especially for those of us that use uncorrected raw data.

    So with, say, micro4/3 captures which tend to rely heavily on the software to correct for len's distortion it would probably be better to break a long edge down into a number of smaller sections and average MTFMapper's results. If you had 1000px to work with, say 25% of the width of an image, what comfortable minimum size would MTFmapper like to work with (50, 100, 200)?

    I'll do some testing and report back.

    ReplyDelete
    Replies
    1. MTF Mapper is hard-coded to only use the middle-most 400 pixels along the edge (i.e., 200 to each side from the centre of the edge).

      I would say about 100 pixels is a fairly good compromise (good enough accuracy, not likely to be affected much by lens distortion), but that is just a guess based on what I saw on the D7000 at ISO 100.

      Delete