Validating MTF Mapper accuracy, part 2

Validation of synthetic images generated by mtf_generate_rectangle, revisited

The first step in the validation of MTF Mapper accuracy is to generate images with known MTF curves --- this is why I have developed (and included) the mtf_generate_rectangle tool. I have always suspected that the synthetic images generated by mtf_generate_rectangle were reasonably accurate, and this was confirmed in part 1 of the validation exercise.


Of course, having seen some evidence of systematic error (second figure of validation part 1), I could not leave it at that. Well, not for long, anyhow.


The "old" method used to render synthetic images relied upon discrete sampling of the PSF of a pixel, using a quasi-random numbers following the desired Gaussian distribution. This method has a fixed cost per pixel, and is reasonably accurate when enough samples are used.


An alternate method of rendering the synthetic images is conceptually very simple. The target is the desired black rectangular shape we wish to render, and we know that the image value should be zero (black) inside the rectangle, and 1 (white) outside. Thus, to determine the intensity of an arbitrary pixel, we place the Gaussian PSF at the centre of our pixel, and measure the integral of the PSF over the region defined by the rectangle. Since the rectangle is finite, and the surrounding white region is infinite, I choose to integrate over the inside of the rectangle. This is simply a double integral of the PSF, with the limits of integration defined by the target rectangle.


Urgh. Double integrals. Well, I did say it was conceptually simple. I will leave the nasty details for another post (now available), but suffice it to say that this particular double integral can be evaluated in closed form using the erf() function in one axis (I chose x), leaving us to use numerical integration (adaptive Simpson's rule) on the other axis.


Since there is numerical integration involved, we still have to choose some parameters. I went with parameters that produce good results, but still result in run times that are comparable to the old discrete sampling method.


At this point, I assume that you have read part 1. First up, we compare the measured edge spread function (ESF) of the synthetic image with that of the expected analytical ESF, which happens to be an erf() function:

No surprises here. At this scale, the two functions are expected to appear identical. Next is the plot of the difference between the measured and expected ESFs:

This looks much better than the comparable plot of the "old" discrete sampling rendering method. To put this in perspective: the largest absolute difference is now 4/65536. This in no coincidence, since both the measured and the analytical ESFs were sampled as 16-bit unsigned integers. Differences of up to 1/65536 can probably be labelled as "rounding errors" --- you can imagine that if the measured ESF was rounded up to the nearest 16-bit number, and the analytical one happened to be rounded, then you will measure an error of 1/65536.


With the excuses out of the way, we can compare the old discrete PSF sampling and new PSF integral methods side-by-side:

The integral method thus produces significantly more accurate edge profiles: the difference between the measured ESF and the analytical ESF is very close to the quantization error. There is still some structure visible in the difference plot, but at this point I am going to claim that this difference is negligible.


Moving on to the SFR curve, we can plot the difference between the measured SFR and the analytical SFR:

Note that the scale has been magnified 100x compared to the plot in part 1, but we still see very little structure in the SFR difference plot. In the frequency range 0 to 1 cycle per pixel, the maximum absolute difference is only 0.0002967%.


Computing the MTF50 value from the SFR data reveals that the measured MTF50 differs from the analytical MTF50 by 0.000351%.


The whole experiment was repeated on a synthetic image with an expected MTF50 value of 0.06 (just like in part 1). The results were, surprisingly, somewhat worse. For example, the maximum absolute difference in ESF curves increased to 5/65536, which is still negligible, but the difference curve was visibly structured:

The maximum absolute error in the SFR curves amounted to 0.13%, which is much higher than the 0.0002967% seen in the MTF50=0.5 case above, but still substantially better than the 0.356% seen with the old discrete PSF sampled rendering method. This peak error in the SFR curve occurs after the MTF50 frequency, though, so that the difference between the measured MTF50 and the analytical MTF50 came down to only -1.496e-6%, which is negligible, and substantially better than the old PSF sampling result.


These results demonstrate that the synthetic images generated with the mtf_generate_rectangle tool are valid, and that the actual ESF/PSFs of the synthetic images agree with the expected analytical ESF/PSFs to such a high degree that the images can safely be used as ground truth in evaluating the accuracy of an SFR (or MTF50) measurement tool.


ps: No binaries have yet been released with the PSF integral rendering algorithm. You will have to compile from source checked out from the repository to obtain the new rendering algorithm, for now.