Nordson_WhitePaper_WP8901_Deep Learning.pdf
Deep Learning Del ivers Fast, Accurate Solutions for Ob ject Det ection in the Automated Optical Inspection of Electronic Assembl ies By Charl ie Zhu, N ordson TEST & INSPECTI ON When automated opt ical inspection (A…
Deep Learning Delivers Fast, Accurate Solutions for Object Detection in the Automated Optical
Inspection of Electronic Assemblies
By Charlie Zhu, N ordson TEST & INSPECTI ON
When automated optical inspection (AOI) works, it is al most always preferable to human visual
inspection. It can be faster, more acc urate, more consistent, l ess expensive, and it never gets tired. But
there are some challenging applications. Some tasks that are very simpl e for humans are quite difficult
for machines. Object detection is an example. Given an image containing a cat, a dog and a duck, a
human can instantly confirm the objects’ presence, even when they overlap, and tell you exactly what
points in the image are included in each one. This seemingly simple task can be very challenging for A OI .
In electronic assembly, manufacturers may want to confirm the presence or absence of a component
that varies little in position or appearance from assembly to assembly. This i s a relatively simple task. A
more diffi cult problem, and the focus of the work described here, is the detection of corner fill used to
secure integrated circuits (IC) to a substrate. Though the location of the fill is relatively constant, its
shape and size may vary from instance to instance. This variability makes detection much more
complicated.
Background and terminology
Recent advances in artificial intelligence (AI) in the area known as machine learning (ML), and especially
deep learning (DL) with artificial neural networks (ANN), have made signi ficant progress in solving
problems like object detection. Artificial intelligence can be described as the study and development of
intelligent systems that perceive thei r environments and take actions to achieve goals. Machine l earning
focuses on software algorithms that automatically lea rn to improve a machine’s performance in a
certain task based on feedback from past performance. Deep learning is a subset of machine lea rning.
Deep learning runs on artificial neural networks, which learn using processes modeled on biological
networks, i .e., brains. For example, in a brain, the more often a connection is used, the stronger the
connection be comes – “neurons that fire tog ether, wire together”. A neural network would implement
that behavi or in software. The “deep” in deep lea rning refers to the many layered architecture of the
neural networks it uses. Finally, learning may be supervised – the system is trained using labeled data
(explicit examples of the concept being learned), or unsupervised – the system must itself discover
patterns occurring the training data. Supervised learning is faster, but unsupervised learning opens the
possibility that the system may fi nd previously unknown relationships within the data.
The machine vision problems that the DLs are trying to solve can be broken down into three sub-types
of increasing complex ity (Fig 1): classifi cation, detection, and segmentation. Presented with an image
containing an object, classification attempts to match the object to a known class – “Do I recognize an
object in this image?” Given an im age containing an object, no object, or many different objects,
detection tries to detect the objects, classify them, and determine their size and position – “Do I
recognize objects in this image? What are they? Where are they? How big are they?” Finally,
segmentation tries to associate each point (pixel) in the image with an object. Segmentation tries to
draw a closed border around each detected object. Any pixe l inside the border is part of that object. Any
pixel not in an object is background.
Figure 1. Object detection problem s can be di vided into 3 sub-types of increasing complexity:
classifi cation, detection, and segmentation.
Corner fill inspection
A memory manufacturer uses fill under the corners of ICs to bond the package to the substrate. They
needed to inspect for the pres ence or absence of fill and ensure that there is neither too much, nor too
little. They required a solution that could measure the length of the corner fil l and compare it to
specifications.
Traditional methods of corner fill inspection, such as blob analysis, are challenged by the lack of gray
level specifici ty in this application. Blob analysis attempts to find a continuous blob within a certain
intensity or contrast range and sometimes breaks larger blobs into separate small er blobs. Figure 2
shows an exampl e of corner fill in which bl ob analysis incorrectly found multiple smaller blobs, all too
short to meet specification, rather than a single good one, and reported a false negative resu lt. This is an
example of a problem that is challenging for AOI but relatively easy for humans, who can readi ly see that
the fill is continuous in the example. Results from inspection with blob analysis were i nconsistent and
unreliable, with many false negatives. The manufacturer sought a more reliable approach.
Figure 2. Blob analysis incorrectly divided this good corner fill into several smaller pieces and reported a
false negative result.
The manufacturer desired a solution that could confirm the presence of absence of the corner fill and
measure its size – a classic example of object detection, mid-level in complexity among classification,
detection and segmentation as described above. Research in the use of deep learning for object
detection has made dramatic progress in recent years, driven by demand across a variety of
applications, including facial recognition and autonomous driving. Autonomous driving shares som e
requirements with the corner fill application. It needs to be fast, i.e., it nee ds to detect objects, such as
pedestrians and other cars in nearly real-time. It needs to determine how big the object is, and where it
is in the field of view, with enough accuracy to avoid a collision. I t does not need to precisely define the
edges of the object.
A deep learning algorithm for object detection that has gained wide acceptance, is used in autonomous
drivi ng applications. Earlier approaches to object detection repurpose classifiers to perform detection,
while this approach frames object detection as a regression problem to spatially separated bounding
boxes and associated class probabilities. A single neural network predicts bounding boxe s and class
probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single
network, it can be optimized end-to-end directly on detection performance. The approach applies a
single neural network to an image, divides the image into regions, and predicts bounding boxes that are
weighted by class probabilities. It shares the same network architecture across all classes, which
simplifies programming and speeds inferencing. The network can be trai ned on a personal computer
with a single GPU (g raphics processor). Once trained, inferencing can run on a device as simple as a
mobile phone.
Corner fill i s located below the corners of a relatively flat, rectangular package wher e it is not easy to see
from any one point of view. Some corner fill inspection systems use a top-down camera and a mirr or to
view all sides of the package as it rotates – an approach that adds time and complexity to the data
acquisition process. The system used in this work is the Nordson TEST & INSPECTION SQ3000™ Multi-
Function System for AOI, SPI and CMM powered by M ulti-Reflection Suppression™ (MRS™) sensor
technology) incorporates a uni que optical sensor originally designed for three-dimensional inspection
and metrology using phase shift profilometry. The sensor v iews the inspection target simultaneously
through four side-view cameras positioned off the normal axis at azimuths of 0°, 90°, 180°, and 270°. For
corner fi ll i nspection, t he side-view cameras can instantly acquire images of all four sides, without
mirrors or rotating the sample (Figure 3).