360 CAMERA PROJECT
EE M202A / CS213 EMBEDDED SYSTEMS UCLA
Methodology
Start from Image Stitching
Before video stitching, we need to understand about image stitching. Image stitching is basic step for combining multiple images together to one image.
The procedure of Image stitching includes three steps; keypoint detection, homography calculation, and image stitching
Keypoint
Match
Homography
Calculation
Image
Stitching
1. Keypoint Detection & Matching
Keypoints are particular points selected withing a given criteria
Used for Accurate Matching (Invariance to scale, Rotation, Noise)
​
Different Methods to Detect Keypoints
Harris Corner Detector
SURF (SpeededUp Robust Features)
SIFT (Scale-Invariant Feature Transform)
Quick review of SIFT
Finding Keypoints
Scale-space peak selection
Keypoint Localization
Keypoint Descriptor
Image gradients in local neighborhood of keypoint
4x4 array of histograms, each with 8 orientation bins (128 element vector)
Keypoint Matching
Match each keypoints independently to the database of keypoints:
Find the nearest neighbors (keypoints) based on the Euclidean distance for the vector descriptor
A cluster of 3 matched keypoints agree on an object an its pose.
2. Homography Calculation
Establish the mathematical relationships that map pixel coordinates from one image to another
Quick review of RANSEC Algorithm
Robust Homography Estimation Algorithm
We have two overlapping images I1 and I2
1. For i: from to k
select random 4-pairs of points and compute Hi .
project all the image points of I1 on the plan of I2 using Hi and calculate the distance d(p′, Hi p)
count the number of inliers where the distance d¡threshold (how to set threshold? not easy).
2. Choose the best homography H that has the maximum number of inliners
3. Image Stitching
Seam Estimation
Seam
Seams can be either vertical or horizontal. A vertical seam is a path of pixels connected from top to bottom in an image with one pixel in each row. A horizontal seam is similar with the exception of the connection being from left to right. The importance/energy function values a pixel by measuring its contrast with its neighbor pixels.
Computing Seam
Computing the seam consists of finding the path of minimum energy cost from one end of the image to another. This can be done via Dijkstra's algorithm, dynamic programming, or graph cuts.
Seamless Stitching
Computed seam will be used for splitting area of stitching images. Without this step, we can see the line of each images boundaries. But with seam cut stitching, boundaries of each images will be lessen.
Panoramic Stitching Issue
What if you want a 360° field of view? Need to consider which mapping method should be used.
We should select cylindrical mapping because it reduces tilting of stitching image.
Planar Mapping
Red Image : Pixels are already on the planar surface
Green Image : Map to first image plane
Cylindrical Mapping
Red Image : Compute h, theta on cylindrical surface from (u, v)
Green Image : Map to cylindrical surface
Image Tilted
Image not Tilted
Real Time Video Stitching
Real Time Video Stitching is challenging, because stitch processing takes long time. So we designed a new method for real time video stitching.
Find Homography Matrices Each Step
Try Image Stitch of All 6 Camera
Find Seam and Make Masks for Each Image
Make Real Time Combined Video
1. Find Homography Matrices Each Step
Find homography matrices of consecutive images by each step. At first, we tried to stitch images consecutively. But it didn't work well, because after 3 images stitched next image couldn't find a feature matching points. That was because stitched image includes so many information features. So we tried to find feature matching between only consecutive images and tried to find homography matrices step by step until the end of all cameras.
Find Homography
Feature Matching
Camera 1 & 2
Find Homography
Feature Matching
Camera 2 & 3
2. Try Image Stitch of All 6 Cameras
With previous collected homography matrices, we tried to do image stitch. Because of camera vertical angle difference, stitched imaged sometimes drifted. So we calibrate the vertical angles again and tried to find the best setting which shows almost horizontal stitching results. So we finally got the best setting and homography matrices.
Drifted Stitch Result
Best Straighten Stitch Result
3. Find Seam and Make Masks for Each Image
After determining homography matrices which results good stitch image, we tried to make a masks for each image with finding seam estimation. So we created each 6 masks and this masks were multiplied with images.
4. Make Real Time Combined Video
With prepared homography matrices and masks, we can multiply original image frames from multiple cameras. And those multiplied images will be combined together and then will result one stitched frame. This steps will continuously operate and show the video or save the video of stitched result.
Hardware Preparation
Selection of Embedded platform NVIDIA Jetson TX1
NVIDIA Jetson with GPU-accelerated parallel processing is the world’s leading embedded visual computing platform. It features high-performance, low-energy computing for deep learning and computer vision making the Jetson platform ideal for compute-intensive embedded projects like drones, autonomous robotic systems, mobile medical imaging, and Intelligent Video Analytics (IVA). OEMs, independent developers.
6 Cameras Logitech C920
Camera Mount by 3D Printer
Software Preparation
We made a GUI software of Real Time VIdeo Stitching. It can align cameras to find homography matrices for stitching. And after it find good result of stitching, it can save the homography matrices value and later load it to video stiching. Last part of software is real time video stitching. It will show 360 degree combined video.
