LiDAR Point Cloud Colorization with Paired Camera Images

1. Intro

LiDAR sensors are essential in modern autonomous robots and vehicles, providing precise distance measurements of obstacles. As active sensors, LiDAR systems are largely unaffected by ambient light, ensuring robustness in extreme lighting conditions. However, LiDAR measurements, known as point clouds, are not as intuitive or easily interpretable as images.

To enhance interpretability, a common approach is to fuse LiDAR data with other modalities, such as cameras. Cameras produce dense, informative images but cannot directly measure distances. This combination is widely adopted in the autonomous driving industry. Cameras can detect and identify objects and segment different semantic areas in 2D, aiding the vehicle's understanding of its surroundings, while LiDAR sensors reconstruct the environment in 3D space.

In this report, we will demonstrate a basic method for fusing camera and LiDAR data. We will map each LiDAR point to its corresponding color from the camera image using known extrinsic calibration parameters between the sensors. Finally, we will implement a visualization in a Python notebook to intuitively showcase the real-world data. Demo code can be found here.

2. Methodology

To align the data from camera images and LiDAR points, we require their measurement models and the relative poses between the sensors. The relative poses, often known as extrinsic calibration parameters, describe the rigid transformations between the sensor coordinates using 3D rotation and translation. For LiDAR, each point is described directly in the form of 3D Euclidean coordinates.

2.1. The Pinhole Camera Model

For the camera model, the process is a bit more complex since the measurements are on a discrete 2D plane. However, this plane is generated according to the physical projection model, where light travels in straight lines. In this demonstration, we will not delve into the details of camera models. Instead, we will introduce its form, which provides the mapping between 3D points in the world and the 2D pixels on the image. For people who are interested in camera models, the document from OpenCV might be a good starter.

We start with introducing three different coordinate systems, namely, image coordinate system , camera coordiate system and LiDAR coordinate system .

An illustration of pinhole camera model. Source from OpenCV Doc. In the image, a point is projected on the 2D image plane.

As illustrated in the image, the system represents the 2D image plane, which is perpendicular to the optical axis of the camera. The axes of are parallel to the remaining two axes in the camera coordinate system. The origin of is located at the top left corner of the image, and its coordinates are discrete in pixels. The center of the camera coordinate system is the focal point of the lens, with the axis being collinear with the optical axis. Ideally, this relationship forms a pair of similar triangles for any 3D point in the camera system and its projection point on the 2D image. Therefore, for any 3D point , we have

where is the projection matrix, composed with the so called camera intrinsic calibration parameters. is the location of the pixel on the image plane. In case the value of the coordinates are not integers, we can assign them to the nearest pixel, or interpolate the value at the exact location given the neighboring pixels.

2.2. Transformation from LiDAR to Camera

As both system and system are Euclidean systems, their relationship is described by a rigid transformation, involving 3D rotation and 3D translation. The rotation from the LiDAR coordinate system to the camera coordinate system is represented by the 3D rotation matrix . Similarly, the corresponding translation is denoted by . These parameters are obtained from the extrinsic calibration between the two sensors.

As a result, for any LiDAR point , the full mapping to the pixel coordinate is determined via conversion to the camera coordinate system, followed by projection to the image plane. We denote such transformation as . Therefore, the optical color of any LiDAR point is where denotes the pixel value of the given position.

3. Visualization

We provide the source code of visualization at this git repository.

The test data is provided by the KITTI Dataset. Below the image shows the scene captured by optical camera mounted on the roof of the test vehicle.

To illustrate the LiDAR point cloud, we present the frontward points with their depth values, shown in viridis colormap. It reflects the basic geometry of the surrounding environment. We can vaguely infer the car on the left, the cyclist on the right, the tree trunks, the road curbs, the traffic poles and the rail tracks, given the prior information we have already seen from the camera image.

After properly associating all the LiDAR points with the pixels from the camera image, we give each 3D point with its original color. See the image below. The wall on the right is clearly colored in brown, along with the green grassland near it. However, due to the sparse nature of LiDAR point clouds, things are not as visually clear as shown in the form of a dense camera image. Nevertheless, we still fuse the optical and visual information from the camera with the geometry information from the LiDAR, enriching the information each point carries, and therefore enable further processes for high-level tasks, e.g., object detection and instance segmentation.