How to stitch multiple cameras together on a moving vehicle

Image stitching is the process of combining multiple images together to create a seamless panorama or a large-scale image. This technique is commonly used in photography, virtual reality, and mapping applications. In this blog, we will take a deep dive into the concepts of image stitching, including sensor and vehicle frame of reference, origin point, extrinsic and intrinsic transformations, field of view, overlapping features, homography, and camera models.

First, let’s talk about the concept of sensor frame of reference. In the context of image stitching, a sensor frame of reference refers to the coordinates of an image in relation to the camera sensor. This includes the position and orientation of the camera at the time the image was taken. It’s important to note that the sensor frame of reference is different from the vehicle frame of reference, which is the coordinates of the camera in relation to the vehicle it is mounted on.

Next, let’s discuss the concept of the origin point on a vehicle frame of reference. The origin point is the point on a vehicle where the coordinates (x, y, z) are all zero. This point is used as a reference point for the vehicle frame of reference. The extrinsic parameters, such as the position and orientation of the camera, are an offset of this origin point. In other words, the extrinsic parameters tell us how the camera is positioned and oriented in relation to the origin point.

Now, let’s talk about the concept of extrinsic and intrinsic transformations. Extrinsic transformations are the changes in position and orientation of the camera, while intrinsic transformations are the changes in the camera’s optics. For example, an extrinsic transformation would be moving the camera to a different location, while an intrinsic transformation would be adjusting the zoom or focus of the camera. Both extrinsic and intrinsic transformations can affect the final image and must be taken into account when stitching images together.

When stitching images together, it’s important to consider the field of view and finding overlapping features. The field of view is the area of the scene that can be captured by the camera. When stitching images together, it’s important to find overlapping features between the images so that they can be aligned properly. This is known as feature matching and is typically done using algorithms such as SIFT or ORB.

Once the overlapping features have been found, the next step is to use homography to stitch the images together. Homography is a mathematical transformation that maps one image to another. It’s used to align the overlapping features and create a seamless panorama. Homography can also be used to correct for perspective distortion and align images taken from different viewpoints.

Another important concept to consider when stitching images together is the difference between equirectangular and pinhole camera models. Equirectangular images are typically used in virtual reality and panoramic photography, while pinhole camera models are commonly used in traditional photography. One key difference between the two is that equirectangular images have a wider field of view and are more susceptible to distortion. Pinhole camera models, on the other hand, have a narrower field of view but produce less distorted images.

Distortion is another important concept to consider when stitching images together. Distortion refers to the deviation of an image from its true shape. In equirectangular images, distortion can occur due to the wide field of view, while in pinhole camera models, distortion can occur due to the camera optics. Undistorted images are important for accurate stitching and can be achieved through the use of camera calibration and correction algorithms.

It’s important to note that AI trained on pinhole camera models will not perform well on equirectangular images, and vice versa. This is because the two types of images have different characteristics and require different approaches for image processing and analysis. For example, an AI model trained on pinhole camera images will be optimized for detecting features and identifying objects in a narrow field of view with minimal distortion. On the other hand, an AI model trained on equirectangular images will be optimized for detecting features and identifying objects in a wide field of view with more distortion.

To achieve the best results when using AI for image stitching, it’s beneficial to train models independently for equirectangular and pinhole images. This allows the AI to be tailored to the specific characteristics of each type of image, resulting in more accurate and efficient image stitching.

In conclusion, image stitching is a complex process that requires a deep understanding of various concepts, including sensor and vehicle frame of reference, origin point, extrinsic and intrinsic transformations, field of view, overlapping features, homography, and camera models. With the advancements in AI and computer vision, image stitching has become more efficient and accurate, but it’s important to consider the differences between equirectangular and pinhole camera models and the impact of distortion. By training models independently for each type of image, we can achieve the best results for image stitching.

Camera <> LiDAR Fusion

In addition to image stitching, another important application of image processing in the automotive industry is the fusion of Lidar data with stitched equirectangular images. Lidar, or Light Detection and Ranging, is a technology that uses lasers to measure the distance to objects and create a 3D point cloud of the environment. When combined with stitched equirectangular images, Lidar data can provide a more accurate and detailed representation of the environment, which is essential for autonomous driving and other advanced applications.

The process of fusing Lidar data with stitched equirectangular images starts with undistorting the equirectangular images. As mentioned earlier, equirectangular images are more susceptible to distortion due to their wide field of view. To correct for this distortion, algorithms such as camera calibration and correction can be used. Once the images have been undistorted, they can be converted to cube maps.

A cube map is a representation of an environment that uses six images, each representing a face of a cube. By converting equirectangular images to cube maps, we can create virtual camera perspectives that represent different viewpoints of the environment. These virtual camera perspectives can then be used to project the point cloud into each cube map perspective.

The process of projecting the point cloud into each cube map perspective involves aligning the point cloud with the virtual camera perspectives. This can be done using algorithms such as ICP (Iterative Closest Point) or RANSAC (Random Sample Consensus). Once the point cloud has been aligned, it can be projected onto each virtual camera perspective, creating a 3D representation of the environment.

The resulting 3D representation of the environment can be used for various applications, such as autonomous driving and object detection. For example, by projecting the point cloud onto the virtual camera perspectives, we can create detailed 3D models of the environment that can be used for obstacle detection and path planning. Additionally, by fusing Lidar data with equirectangular images, we can also improve the accuracy and detail of the 3D models, which is essential for advanced applications such as autonomous driving.

In conclusion, the fusion of Lidar data with stitched equirectangular images is an important application of image processing in the automotive industry. By undistorting equirectangular images and converting them to cube maps, we can create virtual camera perspectives that represent different viewpoints of the environment. By projecting the point cloud onto these virtual camera perspectives, we can create a detailed and accurate 3D representation of the environment that can be used for various applications such as autonomous driving and object detection.

Camera <> Radar Fusion

In addition to the fusion of Lidar data with stitched equirectangular images, another important application of image processing in the automotive industry is the fusion of radar data with equirectangular images. Radar, or Radio Detection and Ranging, is a technology that uses radio waves to measure the distance and speed of objects. When combined with stitched equirectangular images, radar data can provide a more accurate and detailed representation of the environment, which is essential for autonomous driving and other advanced applications.

One of the key benefits of radar tracking is its ability to be used to train the depth model for predicting object distance and object state estimations. Radar has the ability to track objects through time and space and can segment foreground and background data. This allows for the prediction of the object’s distance and speed, which can be used to improve the accuracy of object detection and tracking.

The camera, on the other hand, has the ability to detect and track objects in the pixel space. This allows for the detection of objects in the image and the tracking of their movement over time. By fusing radar and camera data, we can combine the advantages of both technologies to create a more accurate and detailed representation of the environment.

The process of fusing radar and camera data starts with the radar to camera registration process. This process involves aligning the radar data with the camera data so that the object distance and velocity can be attributed to the correct object in the image. There are several techniques that are commonly used to achieve this, such as using fiducial markers, feature matching, and extrinsic calibration.

Another technique that is commonly used for radar to camera registration is the use of a Kalman filter. A Kalman filter is a mathematical algorithm that can be used to estimate the state of a system over time. By using a Kalman filter, we can estimate the position and velocity of objects in the radar data and match them with the objects in the camera data.

In conclusion, the fusion of radar data with stitched equirectangular images is an important application of image processing in the automotive industry. By combining the advantages of radar and camera data, we can create a more accurate and detailed representation of the environment. The radar to camera registration process is an essential step in this process, and there are several techniques that are commonly used to achieve this, such as fiducial markers, feature matching, extrinsic calibration, and Kalman filter. This can be used to improve the accuracy and detail of object detection and tracking, which is essential for advanced applications such as autonomous driving.