Making the Most of Image Sensors in Industrial Automation

Imaging is becoming a key part of the development of many embedded systems. This article looks at the image sensing technology – from sensors to algorithms and IP – that is increasingly becoming available for industrial vision.

Image sensors are becoming increasingly popular in a wide range of industrial automation applications. Maturing CMOS technology is providing larger sensor arrays to give a bigger field of view, as well as faster image capture to keep up with the requirements of visual inspection. At the same time new imaging architectures and new software algorithms are improving the speed of recognition and allowing systems to be more cost effective. As field programmable gate arrays (FPGAs) increase in density and decrease in cost, these algorithms are becoming more usable on the factory floor, again assisting with the popularity of image sensors. While the traditional view is that machine vision requires the highest resolution (see Table 1), new architectures for CMOS devices are providing dramatically more performance, while standard CMOS devices are being used for more imaging and automation applications across the factory floor.

Table 1: Resolution by End-Equipment Types (Source: Altera).

CMOS sensors are the mainstay of industrial imaging. Although they are not as high resolution as bipolar charge coupled devices (CCDs), they are cost effective and rugged for use in industrial imaging. The same ruggedness and reliability that is being applied to designs in the automotive market for applications such as lane detection is now becoming available for industrial vision. While there has been a performance advantage for CCD in industrial vision, the reducing cost of mainstream sensors and cameras is allowing vision to be used cost effectively on the factory floor, and the increasing performance of high-end CMOS sensors is competing directly with CCD technology.

New devices can capture images at 600 frames a second for basic machine vision applications with a dynamic range of more than 64 dB. A pipelined operation for the pixels can enable integrating a frame during readout of the foregoing frame; and it is this architecture, along with low parasitic light sensitivity of 1/50,000, that can reduce a sensor's fixed pattern noise (FPN) to less than 0.1 percent full swing and dark current noise at room temperature.

Figure 1: CMOS technology for industrial imaging.

FPGA support for industrial imaging

The reducing cost and increasing density of FPGAs is also contributing to the increasing penetration of vision systems into industrial automation. Vendors such as Altera and Xilinx are implementing visual processing algorithms on their low-cost FPGA families that now have enough logic and memory density and memory to be highly effective in this market. Lattice is also optimizing its FPGAs to act as interfaces to specific image sensors such as those from Aptina to make building visual inspection systems quicker and easier.

A reprogrammable Cyclone III FPGA from Altera, for example, can now implement a low-cost H.264 encoder or be used as a DSP coprocessor to add visual processing functions such as filters and smoothing functions to a system with minimum engineering effort. This is tremendously powerful for inspection and monitoring systems, allowing a range of third-party algorithms to be implemented for the specific applications. The Cyclone III EP3CLS150F484C8 has over 150,000 logic elements for supporting image IP; and evaluation boards from Altera such as the DK-DEV-3C120N, and the P0037 from Terasic, provide a platform for development with these devices alongside Altera’s Quartus development tools.

Figure 2: Cyclone III FPGA Development Kit.

To support higher bandwidth sensors, Aptina uses a high-speed serial interface called HiSPi. This can operate from one to four lanes of serial data, plus one clock lane. Each signal is differential and can run at speeds up to 700 Mbps. This is an ideal application for an FPGA, and Lattice Semiconductor has created a bridge from HiSPi to a parallel bus (Figure 3) using the Lattice XP2-5 non-volatile FPGA, which is qualified for an industrial temperature range. The RD1120 reference design supports multiple HiSPi formats such as packetized or streaming data and has been tested with Aptina sensors.

Figure 3: The Lattice bridge implementation for Aptina’s HiSPi high-speed image sensor interface.

Lattice also has a reference design for an IP camera, the HDR-60, which is a production-ready HD video camera based on the Lattice FPGA family that can be used for industrial automation applications.

The design is pre-loaded with an image-processing pipeline based on IP cores from Lattice partner Helion in Germany that can provide 1080p resolution at 60 frames per second with 2D noise reduction and High Dynamic Range (HDR). It is designed to fit into commercially available camera housings and capable of supporting two sensors simultaneously with a simple USB interface. Using an Aptina 720p HDR sensor, the kit also offers the industry’s fastest auto-exposure, greater than 120 dB system dynamic range, a highly-efficient auto white balance algorithm and 2D noise reduction – all in streaming mode through the FPGA without the need for an external frame buffer, enabling extremely low latency and reducing system cost. On-board DDR2 memory also enables applications such as 3D noise reduction, image stitching from multiple sensors, image rotation, and de-warping.

While the kit has been designed using a LatticeECP3-70 FPGA, the IP pipeline needs only a lower density LatticeECP3-35 device and the remaining space is available for experimentation and adding custom IP.

Figure 4: The image-processing pipeline for the Lattice HDR-60 IP HD camera reference design links to an Aptina image sensor.

The Xilinx Spartan-6 is also being used for industrial vision systems. This can support a wide range of image processing IP for edge enhancement, defective pixel and color correction, as well as color filter array interpolation.

A reference design for industrial imaging uses a Spartan-6 LX150T FPGA with two daughter cards and a dual image sensor input with DVI/HDMI output and DVI/HDMI input.

Systems

The resolution and frame rate of the sensor itself is just one part of the challenge for the industrial user. There are other issues such as temperature, humidity and EMC that need to be addressed by the camera maker.

The 26100-240 1.3 megapixel camera from Aven Tools uses an Aptina 1/3 in. sensor (3.6 µm x 3.6 µm pixel) to give a horizontal resolution of 1280 x 1024 at 15 fps. This is aimed at industrial applications through a scientific-grade, non-destructive image output via USB 2.0 and a patented design to prevent electromagnetic interference, vital in a harsh industrial environment. The 220 g unit fits easily into the production line and links back to a controller via a standard USB cable.

Figure 5: The 26100-240 1.3 megapixel camera from Aven Tools.

Tiny cameras are making their way into industrial applications that are restricted for space. These cameras, measuring just 29 x 29 x 30 mm, are aimed directly at machine and computer vision applications with 3.2 megapixel sensors operating at 60 fps and a USB 3.0 interface.

USB 3.0 is rapidly gaining industry acceptance, largely due to its higher bandwidth, improved reliability and architecture, and widespread availability on a variety of hardware platforms. Delivering 384 MBytes/s, USB 3.0 is almost 10 times faster than USB 2.0 and five times faster than Firewire. Many embedded systems now provide USB 3.0 ports, which maximizes system compatibility and minimizes the need for add-in PCI or PCI Express interface cards.

Conclusion

CMOS technology has rapidly caught up to the performance of CCD. With new pixel architectures such as Aptina’s A-Pix and the high-speed interfaces such as HiSPi, CMOS sensors are providing the same level of performance as CCD. With FPGA technology increasing in density and reducing in cost, there is now more opportunity to implement new algorithms alongside the image sensors to further increase the accuracy and resolution of the imaging and allow more cost-effective penetration into the industrial market. More accurate edge- and pattern-detection algorithms help the system design make the most of the performance of image sensors in specific applications.

As more CMOS sensors are being adopted in automotive designs with extended temperature range and immunity to interference, the technology is being adopted in industrial imaging and vision systems, reducing size, power consumption, and cost while enabling more automation of the production line. From surveillance cameras to specific visual inspection systems, CMOS image sensors are at the heart of many industrial applications.