Since the depth data acquired by LIDAR is very sparse, a three-dimensional reconstruction algorithm based on sparse laser point cloud data and single frame image is proposed so as to reconstruct the 3D depth map from the depth data and image data in this paper. The proposed algorithm firstly uses the point histogram feature to effectively select the point data corresponding to the target and eliminate the non-similar points in the voxels. Then, the local depth data is modeled by Gaussian process regression, and the 3D depth data is obtained by interpolation. The 3D depth points obtained by our algorithm are closer to the reference value and keep the local shape feature of the object. Compared with existing 3D reconstruction algorithms based on LIDAR data and image data, simulation results show that the algorithm proposed in this paper will greatly enhance the robustness and reconstruction accuracy, and can be used for the unmanned vehicle in complicated urban scenes.