OpenCV - Segmentation
Segmentation
What is Segmentation?
Partitioning images into different regions
Contours
Contours are continuous lines or curves that bound or cover the full boundary of an object in an image.
Contours are very important in:
- Object Detection
- Shape Analysis
OpenCV stores Contours in a list of lists.
Code Implementation
cv2.findContours(image, Retrieval Mode, Approximation Method)
Reutrns
contours
,hierachy
NOTE In OpenCV 3.X, findContours returns a 3rd argument which is ret (or a boolean indicating if the function was successfully run).
If you’re using OpenCV 3.X replace line 19 with:
_, contours, hierarchy = cv2.findContours(image, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
The variable ‘contours’ are stored as a numpy array of (x,y) points that form the contour
While, ‘hierarchy’ describes the child-parent relationships between contours (i.e. contours within contours)
1 | import cv2 |
Hierarchy Types:
cv2.RETR_LIST
- Retrieves all contourscv2.RETR_EXTERNAL
- Retrieves external or outer contours onlycv2.RETR_COMP
- Retrieves all in a 2-level hierarchycv2.RETR_TREE
- Retrieves all in full hierarchy
Sorting Contours
Suprisingly useful technique when applying object recognition algorithms or machine learning classification algorithms.
Sorting by Area can assist in Object Recognition (using contour area)
- Eliminate small contours that may be noise
- Extract the largest contour
Sorting by spatial position (using contour centroid)
- Sort characters left to right
- Process images in specific order
Before Sorting
1 | import cv2 |
Sorting by Area
1 | import cv2 |
Sorting by spatial position
1 | import cv2 |
Approximating Contours and Convex Hull
Approximating Contours
cv2.approxPolyDP(contour, Approximation Accuracy, Closed)
- contour – is the individual contour we wish to approximate
- Approximation Accuracy – Important parameter is determining the accuracy of the approximation. Small values give precise- approximations, large values give more generic approximation. A good rule of thumb is less than 5% of the contour perimeter
- Closed – a Boolean value that states whether the approximate contour should be open or closed
1 | import numpy as np |
Convex Hull
1 | import numpy as np |
Shape Matching
cv2.matchShapes(contour template, contour, method, method parameter)
Output – match value (lower values means a closer match)
- Contour Template – This is our reference contour that we’re trying to find in the new image
- Contour – The individual contour we are checking against
- Method – Type of contour matching (1, 2, 3)
- Method Parameter – leave alone as 0.0 (not fully utilized in python OpenCV)
1 | import cv2 |
Identify Shapes by Contours
1 | import numpy as np |
Line Detection
where is the perpendicular distance from origin, is the angle formed by the normal of this line to the origin (measured in radians)
There are 2 algorithms available in OpenCV.
Hough Lines
cv2.HoughLines(binarized/thresholded image, 𝜌 accuracy, 𝜃 accuracy, threshold)
- Threshold here is the minimum vote for it to be considered a line
1 | import cv2 |
Probabilistic Hough Lines
The idea is that it takes only a random subset of points sufficient enough for line detection.
Also returns the start and end points of the line unlike the previous function
cv2.HoughLinesP(binarized image, 𝜌 accuracy, 𝜃 accuracy, threshold, minimum line length, max line gap)
1 | import cv2 |
Circle Detection
cv2.HoughCircles(image, method, dp, MinDist, param1, param2, minRadius, MaxRadius)
- Method - currently only cv2.HOUGH_GRADIENT available
- dp - Inverse ratio of accumulator resolution
- MinDist - the minimum distance between the center of detected circles
- param1 - Gradient value used in the edge detection
- param2 - Accumulator threshold for the HOUGH_GRADIENT method (lower allows more circles to be detected (false positives))
- minRadius - limits the smallest circle to this size (via radius)
- MaxRadius - similarly sets the limit for the largest circles
1 | import cv2 |
Blob Detection
What is a Blob?
Blobs can be described as groups of connected pixels that all share a common property.
How to use OpenCV’s simpleBlob Detector?
- Create Detector
- Input image into Detector
- Obtain Key points
- Draw Key points
Code Implementation
cv2.drawKeypoints(input image, keypoints, blank_output_array, color, flags)
flags:
- cv2.DRAW_MATCHES_FLAGS_DEFAULT
- cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS
- cv2.DRAW_MATCHES_FLAGS_DRAW_OVER_OUTIMG
- cv2.DRAW_MATCHES_FLAGS_NOT_DRAW_SINGLE_POINTS
1 | # Standard imports |
Blob Filtering
Blob filtering - Sape & Size
cv2.SimpleBlobDetector_Params()
Usage :
1 | params = cv2.SimpleBlobDetector_Params() |
Area
params.filterByArea
= True/Falseparams.minArea
= pixelsparams.maxArea
= pixels
Circularity
params.filterByCircularity
= True/Falseparams.minCircularity
= 1 being perfect circle, 0 the opposite
Convexity – Area of blob / Area of Convex Hull
params.filterByConvexity
= True/Falseparams.minConvexity
= 0 to 1
Inertia – Measure of ellipticalness (low being more elliptical, high being more circular)
params.filterByInertia
= True/Falseparams.minInertiaRatio
= 0.01
Object Detection
Object Detection is very important in computer vision.
What for?
- Labeling Scenes
- Robot Navigation
- Self Driving Cars
- Body Recognition (Microsoft Kinect)
- Disease & Cancer Detection
- Facial Recognition
- Handwriting Recognition
- Identifying objects in satellite images etc.
Object Detection vs Recognition
What is the difference?
Object Detection
Only looks at one particular object and does not try to recognize exactly what it seeing.
Recognition
Recognition is basically the second level of object detection.
It allows the computer to actually recognize multiple objects within an image.
Template Matching
This method isn’t very resilient.
- Rotation renders this method ineffective
- Size (Scaling) affects this as well
- Photometric changes (brigntness, contrast, hue etc.)
- Distortions from view point changes (Affine)
Interest Points (Image Features)
Image Features are interesting areas of an image that are somewhat unique to that specific image.
Also called key point features or interest points
Why Important?
Features are important as they can be used to analyze, describe and match images.
Features are used in:
- Image Alignment
- 3D Reconstruction
- Robot Navigation
- Object Recognition
- Motion Tracking etc.
What Interesting?
Interesting areas carry a lot of distinct and unique information at that point
- High change of intensity
- Corners or edges
Be careful that noise can appear informative.
Characteristics
Repeatable
- Can be found in multiple pictures of the same scene
Distinctive
- Each feature is somewhat unique and different and other features of the same scene
Compactness/Efficiency
- Significantly less features than pixels in the image
Locality
- Feature occupies a small area of the image and is robust to clutter and occlusion
Corners as Features
Corners are identified when shifting a window in any direction over that point gives a large change in intensity.
Best summaries images in terms of image matching
- Flat - No Intensity change in any direction
- Edge - Intensity change in one direction
- Corner - Intensity change in all directions
Code Implementation
Harros Corner Detection
Harris Corner Detection is an algorithm developed in 1998 for corner detection and works fairly well
cv2.cornerHarris(input image, block size, ksize, k)
- Input image - should be grayscale and float32 type.
- blockSize - the size of neighborhood considered for corner detection
- ksize - aperture parameter of Sobel derivative used.
- k - harris detector free parameter in the equation
- Output – array of corner locations (x,y)
1 | import cv2 |
Good Features to Track
Good Features to Track is a better algorithm for corner dectection.
cv2.goodFeaturesToTrack(input image, maxCorners, qualityLevel, minDistance)
- Input Image - 8-bit or floating-point 32-bit, single-channel image.
- maxCorners – Maximum number of corners to return. If there are more corners than are found, the strongest of them is returned.
- qualityLevel – Parameter characterizing the minimal accepted quality of image corners. The parameter value is multiplied by the best corner quality measure (smallest eigenvalue). The corners with the quality measure less than the product are rejected. For example, if the best corner has the quality measure = 1500, and the qualityLevel=0.01 , then all the corners with the quality - - measure less than 15 are rejected.
- minDistance – Minimum possible Euclidean distance between the returned corners.
1 | import cv2 |
Problems with corners as features
Tolerant of:
- Rotations
- Translations (e.g. Shifts in image)
- Slight photometric changes (e.g. brightness or affine intensity)
Intolerant of:
- Lage Intensity changes or photometric changes
- Scaling issues
SIFT
SIFT (Scale Invariant Feature Transform)
Note: SIFT is patented and no longer freely available with OpenCV 3.0+.
It is free to use for academic and research purposes :) Try older versions of OpenCV for SIFT.
Basically what SIFT do
- Detect interesting key points in an image using the difference of gaussian method
- Create vector descriptor for these interesting areas.
- Scale invariance is achieved by
- Scanning interest points at several different scales
- The scale which meet a specific stability criteria is selected and endcoded by the vector descriptor
- Regardless of the initial size, the most stable scale is found which allows us to be scale invariant
- Rotation invariance is achieved by obtaining the Orientation Assignment of the key point using image gradient magnitudes
- Once we know the 2D direction, we can normalize this direction.
SURF
SURF (Speeded Up Robbust Features)
SURF was developed to improve the speed of a scale invariant feature detector.
Basically what SURF do
- Use Hessian matrix approximation to detect interesting points
- Use the sum of Haar wavelet responses for orientation assignment
Alternatives to SIFT and SURF
FAST
FAST (Features from Accelerated Segment Test)
- Key point detection only (no descriptor)
- Used in real time applications
BRIEF
BRIEF (Binary Robust Independent Elementary Features)
- Fast
- Computers descriptors quickly (instead of using SIFT or SURF)
ORB
ORB (Oriented FAST and Rotated BRIEF)
- Combines both FAST and BRIEF
- Free to use (Developed out of OpenCV)
Histogram of Oriented Gradients
Histogram of Oriented Gradients (HOGs) are a feature descriptor that has been widely and successfully used for object detection.
- represents objects as a single feature vector as opposed to a set of feature vectors where each represents a segment of image
- computed by sliding window detector over an image, where a HOG descriptor is a computed for each position.
- often used with SVM (support vector machine) classifiers. Each HOS descriptor that is computed is fed to a SVM classifier to determine if the object was found or not.
Basically what HOGs do
E.g. There is a picture pixels.
- Using an 8x8 pixel detection window or cell, we compute the gradient vector or edge orientations at each pixel.
- This generates 64 (8x8) gradient vectors which are then represented as a histogram.
- Each cell is then split into angular bins, where each bin corresponds to a gradient direction (e.g. x,y). 9 bins are used in this case.
- The effectively reduces 64 vectors to just 9 values.
- As it stores gradients magnitudes, it;'s relatively immune to deformations
- We then Normalize the gradients to ensure invariance to illumination changes (i.e. Brightness and Contrast).
- Instead of individual window cell normalization, a method called Block Normalization is used
- Block Normalization takes into account neighboring blocks so we normalize taking into consideration lager segments of the image.