In many cases when choosing a camera for an optical application we start with sensor related characteristics – wavelength sensitivity, pixel size, pixel resolution (width and height) and so on. However, the sensor itself is only a part of the story. The camera that wraps the sensor has an impact as well. I do not refer to special features of the camera such as HW trigger or neural network inference capabilities but the statement refers to the raw image formation of the camera from light to digital image. As usual such statements are meaningless without proper examples so in this post, we will compare two different cameras manufactured by two different manufacturers that have identical sensor integrated in them. This camera comparison will be made with application-related merits so as to easily measure and understand the impact of their differences.
Please note that there are standards for objectively comparing cameras: EMVA1288, JIIA, ISOs 12233, 15739 and many more, but the EMVA1288 standard is the most comprehensive for our purposes. This standard gives numeric methods for characterization of cameras and image sensors. It is a very comprehensive standard which details the methods and calculations to reach an absolute number for many important merits that are of interest in any camera such as defect pixels, SNR, Dark current etc.
I do not intend to utilize this standard in this post simply because its methods and calculations are complex and less intuitive to anyone who does not have an optical laboratory in his backyard. For further information about EMVA you may visit their website.
As mentioned above, we will skip the merits of the EMVA standard and compare, head-to-head, the two cameras with their respective merits that we require for an arbitrary application.
The Sensor and Setup
The sensor chosen for our application is the ON Semi AR0521SR with 5Mpix resolution and a 2.2um pixel size. The application would be a simple imaging system with scan illumination in the field-of-view (FOV).
We use a 50mm lens suitable for 5Mpix sensor with f/# of 1.4 for our optics, with a USAF1951 resolution target in the object plane and white LED diffused back-illumination powered by a stabilized power supply with a static load resistor to control the current to give 3 states – high light, medium light and low light. The setup’s ambient temperature was kept within a +/-1 degree during the whole measurement.
Only non-saturated “live” pixels were taken into account in the image processing. “Dead” pixel count was not conducted because it is less of a camera performance parameter and more of a manufacturing parameter.
The whole setup is stationary and only the cameras were changed in order to keep the acquisition conditions exactly the same.
No one-size camera fits all applications
Important note: there is no one-size camera that fits all applications. I wouldn’t want to infer that one camera is better than the other when it probably isn’t. For this reason, the camera models will not be disclosed and I will use Camera 1 and Camera 2 throughout this post. My intention is not to show which camera is better but to demonstrate that there are differences between them.
Both cameras were provided by courtesy of OpteamX.
System Merits
The following details the numeric merits that are of interest in our application. Each merit measurement includes 100 frames taken in 20fps:
- Dark Signal – the dark signal gives us an important indication of the camera’s sensitivity and grey level differentiation. The higher the dark signal the lower the sensitivity. Dark signal measurement is taken when the camera cap is on and when the camera is placed inside a closed black chamber. The result is the average of the whole pixel array over all the images.
- Linearity over exposure time and gain – is simply as it sounds and that is that we’d like our signal to rise by a factor of 2 when we raise the exposure time from 1ms to 2ms. Similarly the same should apply to gain with an asterisk for logarithmic scale gains.
- Temporal noise for different exposures – it is well-known that the temporal noise, i.e. the single pixel’s DTDdev over time, should grow higher with exposure. We will test and compare this statement.
- Resolution – with the use of our lens we expect an effective resolution inside our DOF of 30lp/mm. We will test a lower resolution of 16lp/mm only to keep a safe distance from the optics limit, so when a certain MTF drop is observed it will be attributed to the camera and not the rest of the optical setup. As such, the system was set up to a magnification of X0.28 to apply a sufficient ratio of pixels\resolution.
The resolution will be the most important and relevant merit. - Uniformity over the sensor – obviously we plan to use the whole sensor. We would like to get the same performance for all merits over all the sensor.
Dark signal results
The dark signal was measured in many exposure time and gain settings. There’s no point in presenting the mean signal of it all so let’s discuss only the mean results of max exposure and max gain:
We can see that Camera 1 has a slightly lower mean dark signal, around 0.24GL. Remembering that operating the camera in max exposure and max gain is usually not a standard use case and as such this difference is probably insignificant.
The following chart of the average pixel temporal STDdev is more interesting.
In the chart above we can see how stable the dark signal per pixel is over time for different exposures with no gain at all. We see that Camera 1 has a more stable dark signal per pixel no matter what the exposure is. Furthermore, the difference grows (both in percentage and in absolute numbers) as the exposure grows.
Once again, these numbers are small but the difference is there.
When applying max exposure and max gain this difference is completely eliminated:
Linearity over exposure and gain
See below the exposure linearity test graphs of both cameras.
It is easy to see according to the r-squared values that both cameras show near perfect linearity against exposure time with a few glitches of Camera 2 in high light.
The picture is even better when testing the gain linearity as follows:
Note: we can see that Camera 1 gain was implemented logarithmic – this is obviously neither a downside nor an upside. We will just use the r-squared values to understand the accuracy of the trend.
This test is done with low light because with higher light power we reach pixel saturation far too quickly to verify the trend.
We can see both cameras present perfect trends.
Temporal noise over exposure
This test was done with no gain applied in both cameras in order to keep the noise factors to a minimum. See below results.
It is clear that Camera 2 shows higher temporal noise, regardless of the exposure time.
Remember, we cannot calculate what the camera noise is from these measurements because we did not accurately measure the light source’s temporal noise. However, if all the temporal noise had originated only in the light source or the temporal noise of the two cameras was equivalent we wouldn’t have seen any difference between the two cameras, which clearly is not the case.
Resolution
Eventually, the bottom line for our application is the resolution we require.
Before going into the details of the different target types and positions let’s begin with the overall MTF comparison between the two cameras.
There’s no doubt about it, Camera 1 exhibits better resolution than Camera 2, 11.9% better which, for many vision applications out there. is not insignificant
Going a bit deeper, the following chart shows the same overall comparison with the separation of vertical and horizontal targets. The picture is essentially the same.
One more attribute that is worth looking at is the resolution performance in different fields of the FOV. To eliminate the impact of the illumination non-uniformity the illumination itself was moved over the FOV and gave the following results:
The fact the center (on-axis) exhibits better MTF than the periphery (off-axis) is probably an artifact of the lens because both cameras show approximately the same drop in the exact same locations. Nevertheless, Camera 1 consistently shows better MTF performance under these conditions.
Results Comparison
It is clear that in the conditions used for testing Cameras 1 and 2, Camera 1 showed better performance – dark signal, temporal noise and above all the MTF.
We have provided compelling evidence for two cameras with identical sensors that they exhibit slightly different performance.
Now it is a good time to disclose that Camera 1’s price tag is higher and as mentioned in a previous post, the camera price tag is also one of the parameters we have to look out for.
So now it is down to whether the final application is sensitive to these MTF changes in the system-level or is it also inside the system’s specification margins.
Is the price tag worth the additional performance boost? Maybe… but if the application is price sensitive then maybe not. In addition, especially when the performance differences are in a measurable merit that directly impacts the system performance, maybe it is a good notion to verify what the minimum MTF requirements for that particular resolution are. On the other hand, if our image-processing algorithms are extremely contrast-sensitive these additional 10% of MTF performance of cleaner signal might be the difference between a go-to system and a project failure.
2 Cars with the Same Engine?
What led me to write this post and check-out 2 different cameras in a less demanding optical setup was the recollection that many years ago I had a task to find a camera to an exotic IR wavelength. In that specific instance the differences between different cameras with a certain sensor were very large to the point that some of the cameras didn’t even pass the minimum image formation requirements while the chosen camera for the application did with nearly full compliance – all of the cameras had the same sensor.
It made me think of the Mercedes’ 1.3L turbo four (M282) engine that is powering 5 Mercedes car models as well as a few Renault, Dacia and Nissan models. Same engine, very different cars.
That’s it for this post. The most important message I would like to leave you with is that choosing an image sensor for an application is only the first part of the camera selection and that basic image formation may differ between different cameras.