The total body scanner developed in the iToBoS project mounts four collaborative robots (cobots) that move independently in order to gather the skin lesions on a patient’s body.
In order to fulfil this goal, the scanner must first construct a 3D map of the patient’s skin surface, in order to properly plan the positions of the moles to inspect. In this article, we will give a glimpse of the methodology followed to obtain such 3D map.
The imaging payload mounted at the end-effector of each cobot in the scanner comprises a triplet of cameras:
- A high-resolution RGB Lucid Triton camera, collecting images of the skin at naked-eye resolution.
- A 3D Lucid Helios2 camera, collecting depth readings and, consequently, 3D points clouds that reveal the 3D shape of the body of the patient.
- An Flir Oryx camera coupled with a liquid lens from Optotune, able to collect a stack of images at different focus distances that, when combined, render an image of a quality that resembles that of standard dermoscopy.
Mainly the Triton and Helios2 cameras are used when creating the 3D map of the patient, while the Oryx camera is only used in the final mole inspection process.
Via a complex calibration procedure, we are aware of all the rigid transformations between the different camera frames and each robot’s end-effector. This effectively allows us to translate the 3D measurements captured by the Helios2 camera to a common reference frame. Therefore, in a first step, the robots move to pre-defined positions and take measurements from both the Triton and Helios2 cameras. The 3D measurements of each camera are then transformed to the common reference frame, such that the local 3D views are gathered together in a single 3D point cloud map (see Figure. 1 (a/b)). Moreover, since the Triton and Helios2 cameras are mounted in a rigid configuration within the end-effector, they form a stereo pair. Thus, the know transformation between both camera frames, coupled with the known intrinsic parameters of each camera, allows us to colorize the 3D point set from the Helios2 camera by projecting it into the Triton image plane and taking the colour of the pixel where each 3D point projects to. All the processes above involve a low computational effort, and can therefore be executed in real time, while the patient is being scanned.
However, this 3D point cloud does not represent the surface of the skin of the patient. Actually, these points are just measurements taken from such surface. This means that in case we want to take new measurements on this surface (such as locating the moles on the patient’s body) we must recover the underlying surface that these points represent.
Therefore, we need to convert this discrete point set into a more convenient continuous surface, and the most commonly used representation for a surface in computer science is a triangle mesh. In order to retrieve a surface triangle mesh from the point cloud map we apply the Screened Poisson surface reconstruction method [1] (see Figure. 1 (c)). It is worth noting that, while not required in this project, the colour on the Triton image could also be used to provide a texture mapping to the reconstructed surface [2].
As previously mentioned, having the patient’s skin in this surface form allows us to take measurements on it, and this is important for the next step of the scanning process, where we want to retrieve the precise 3D location of the skin lesions
Figure 1. Sample 3D reconstruction of a manikin obtained from single shots taken from each of the four robots in the iToBoS scanner. You can see in (a) the global map joining all the point sets, in (b) the same map but colouring each point according to the robot it comes from, and finally in (c) the reconstructed surface triangle mesh.
[1] Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson Surface Reconstruction. ACM Trans. Graph. 32, 3, Article 29 (June 2013), 13 pages. https://doi.org/10.1145/2487228.2487237.
[2] Waechter, M., Moehrle, N., Goesele, M. (2014). Let There Be Color! Large-Scale Texturing of 3D Reconstructions. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8693. Springer, Cham. https://doi.org/10.1007/978-3-319-10602-1_54.