The target is still moving! In this post as part of our search for the right sensor technology to image a fast-moving target we explore in detail the rolling shutter technology.
This is the second post in this series. If you have not yet read the previous post in this series, I strongly suggest you do. It will make this post much easier to read.
To remind you all, we use a fan target imaged with a X5 Mitutoyo objective. The camera used for this post’s images acquisition was the IDS UI-3250CP Rev. 2 camera provided by courtesy of OpteamX. The illumination used was cold white light LED wide angle light source.
The fan looks as follows:
The Rolling Shutter
Rolling shutter technology is when each row in the sensor’s pixel array is exposed to light and read in a different timing.
See below the image build-up evolution in time for a rolling shutter sensor:
Note that in the sketch above I described the optimal situation where each pixel row experiences the exact same exposure but at a different absolute time. There are sensor implementations where the sensor readout is done sequentially and where the pixels are still gathering photons which adds a rolling bias to the intensity of each pixel. As mentioned in the previous post, we will not address these scenarios here since they are not the mainstream.
So, as opposed to global shutter, each pixel row experiences a different start and stop times.
Static Target & Illumination
This case is not interesting for the purpose of this post but we are required to present it for reference. When the target and illumination are static in time the image we will get with our camera+lens setup will be as follows:
As expected, we see a perfect static image of the fan blade and if we take the build-up evolution as before with the existence of the tip of the blade we will get the following flow:
Moving Target
When dealing with a moving target in a rolling shutter camera things get very strange. See the image below taken in a short exposure time of 0.25ms:
Wait! What has just happened? Where is our fan blade?
Let’s try to break down each time period to understand the resulting image.
Let’s assume for the sake of simplicity that we have only the blade’s tip in a small sensor as we simulated before. In the following set of sketches the edges of the tip are marked in red in each image. The different shades of grey represent the grey-level of the pixels very much like a real sensor grey level (in our specific example, only one shade of grey). We get the following timeline:
Which in a way makes sense. If the tip is moving but the exposure is short enough, each row will have an adequate 1-pixel-row-height image of the blade before the blade moves to the next position.
It shows us that we should expect our blade to be skewed, but how come there is more than one line in the image? When it comes to our real example the existence of multi lines as in this one is better explained in the following set of sketches:
What we see here are two passes of the fan’s blades – the first pass marked in red and the second pass marked in green. Note the stage in the first sketch on the second row: both blades are out of the camera’s FOV hence there are rows in the final image with no trace of any of the blades.
Image Reconstruction
If we calculate the roll time of the IDS camera it is in the order of 15-20ms: 120MHz pixel clock X 1600 pixels width X 1200 pixels height ~ 16ms + overheads. That is the time that takes all the 4 blades of the fan to pass through the camera FOV.
When using the information that we gathered until now we can use the simplest image processing only to shift the correct number of pixels for each relevant part of the image and then the magic happens and suddenly we see something familiar:
In the above figure there are 4 reconstructed images, one for each “line” we had in the original acquired image. What we get in practice is a narrow image for each blade pass-by, for the 4 different blades of the fan but in different positions of each blade. The first batch of rows catch the top of the blade while the second batch somewhere in the top-middle etc. So, in a way, with one shot of the rolling shutter camera we caught all 4 blades of the fan. I admit, though a limited view of each, but still all 4 of them and what’s more the reconstructed image is not blurred. Each row’s exposure is 0.25ms so in practice the limited image of that particular row is not blurred and is similar to what we have seen in the global shutter images with the same exposure in the previous post.
BTW, note that each part of the image needed a different correction factor to get the image. To have this correction to be adaptive and generic requires a complex algorithm formulation.
I did not bother presenting a higher exposure rolling shutter image – the image was too blurred and too messy – just imagine blurred skewed lines all over the frame 
.
One question that pops up though was whether the exposure time has an effect of the resulting image in terms of the movement capture. The answer is yes and no. Yes, obviously the exposure will impact the amount of movement captured which in turn will make each row more blurry with higher exposure – you may even think of each row as one row of global shutter of exposure. However, the skewed lines format seen with the 4 different sections is not going to change. This format is related only to the rolling shutter’s “roll time” and is not correlated to the exposure.
Towards the end of this post there is an important question one may ask and that must be answered: “why should we manufacture rolling shutter at all if the global shutter exists?”. Well, from its beginning the rolling shutter technology is entangled in the CMOS sensors: it is the simpler, cheaper and more efficient way to implement and manufacture a CMOS sensor. Not until the late 1990’s when the industry acknowledged that the rolling shutter has interesting effects in imaging of moving targets (see here an image taken by Eric Fossum in 1999 who was the principal inventor of the CMOS active pixel image sensor).
And last but not least, let’s face it, in quite a lot of applications the objects in the scene do not move fast enough for the system requirements to exclude a rolling shutter camera.
This post ends here. In the next post we will review 3 available technologies side-by-side and compare them for the task of imaging our fan.