This page describes the process of converting mesh files to specific formats in more detail. There are three main types of input used for processing 3D models by neural networks - voxels, point clouds and 2D images taken from multiple views. Although the conversion is implemented in multi-threaded fashion it still can be quite time consuming on big datasets, it can take up to ten or more hours in some cases.
Docker: dockers/data_conversion/vrn_data
Networks: dockers/vrn
Details: Voxels are natural extension of pixels to three dimensions. Instead of square lattice we have a cube occupancy grid. For voxelization of meshes we use the openvdb library, which is written in c++ but offers Python bindings for some basic functionality including voxelization. Only one of the supported networks uses directly voxels as its input and for the purposes of this network the data is stored in .npz format. Parameters include num_voxels which is the resolution of the voxel grid and num_rotations which is the number of rotations of a single 3D model to be voxelized.
Docker: dockers/data_conversion/octree_data
Networks: dockers/octree, dockers/octree_adaptive
Octree is a data structure for storing 3D data efficiently. This script uses tools provided by the authors of the original paper. The data is augmented by creating more than one rotation of the 3D model - this number can be controlled by setting the num_rotations parameter. If you want to create data for adaptive octree network set adaptive = True. There is also an option clean_off_files = True to correct invalid .off file headers (missing newline or space after “OFF”) in the original ModelNet40; alternatively the dataset can be converted to .obj format (e.g. by running the mvcnn_data_pbrt script with remove_obj = False option in config) before running this conversion.
Multi view neural networks get multiple rendered images of a single 3D model as their input. This rendering is the computationally most demanding of all types of data conversions. Fortunately all the multi view networks accept the same format.
Docker: dockers/data_conversion/mvcnn_data_pbrt
Networks: dockers/mvcnn, dockers/mvcnn2, dockers/vgg, dockers/rotnet
This is our implementation of rendering using pbrt. It can be very slow, but the performance could be improved. You can set the number of rendered views as parameter num_views. If the dodecahedron parameter is set to True, twenty views from the vertices of a regular dodecahedron will be rendered. This is useful when working with a dataset without canonical rotation. You can also use more rotations of the camera from a single viewpoint by setting the camera_rotations parameter. Pbrt can use only .obj files so for ModelNet .off files are converted to this format and saved. Set remove_obj = False if you want to keep these files after rendering is done.
Docker: dockers/data_conversion/mvcnn_data_blender
Networks: dockers/mvcnn, dockers/mvcnn2, dockers/vgg, dockers/rotnet
With better success we used scripts for Blender provided by team around this paper. We just connected these into our framework, the parameters are same as above. There are following options: shaded images, depth images and Phong-shaded images. For details consult the original paper. You can set the mode of rendering by setting the render variable in run.sh to shaded, depth (or phong - this may produce some out-of-frame and “z-fighting” images).
Point cloud is simply set of points in three-dimensional space. It can be obtained fairly easily from meshes by sampling random points from the faces. The probability of the face being selected is weighted by its area.
Docker: dockers/data_conversion/pnet_data
Networks: dockers/pointnet, dockers/pointnet2
For PointNet and its successor PointNet++ we save the sampled point data in several .h5 files which are listed in text files. With all parameters unchanged, all paths will be valid, but make sure that the paths in these text files are correct inside the docker container. You can set the number of sampled points in the num_points parameter. You can also set normal = True to sample the points with surface normals, but the PointNet implementation of this is currently not working. We have tried three different sampling methods, which can be set as parameter mode (lloyd, uniform and sobol)
Docker: dockers/data_conversion/sonet_data
Networks: dockers/sonet
SO-Net data is sampled in the same way as PointNet by uniform sampling, but with surface normals. Then it creates and learns a self organizing map to better represent the data. This map is created during the data conversion and therefore this script requires a GPU and the NVIDIA runtime. You can set the number of SOM nodes as parameter num_nodes and this should be a square integer.
Docker: dockers/data_conversion/kdnet_data
Networks: dockers/pointnet, dockers/kdnet
This script only loads the mesh data and saves it in a large .h5 file. Construction of kd-trees which are input of the kd-network is contructed during training.