Benchmark on Indoor Modeling
Up-to-date spatial models of indoor environments are needed in a growing number of applications, including navigation, emergency response and a range of location-based services. Automated generation of 3D indoor models from point cloud data has been a topic of intensive research in recent years. While results on various datasets have been reported in the literature, a comparison of the performance of different methods has not been possible due to the lack of benchmark datasets and a common evaluation framework.
The ISPRS Benchmark on Indoor Modelling aims to address this issue by providing a public benchmark dataset and an evaluation framework for performance comparison of indoor modelling methods.
The ISPRS Benchmark on Indoor Modelling is currently open for submission. If you are interested in participation, you can request access to the benchmark dataset by completing this questionnaire, apply your method to reconstruct 3D models from the data, and submit your models for evaluation. The results will be released within a few weeks and published on the Benchmark Results page.
Detailed information about the datasets and the benchmark test can be found in this paper.
Conditions of use
The download of the datasets is subjected to the following conditions:
- Datasets must be used for research purposes only. Any other use is prohibited.
- Datastes must not be distributed to third parties. Any person interested in the data may obtain them via ISPRS WG IV/5.
- If the datasets are used in any publication, please acknowledge the ISPRS WG IV/5 for the provision of the data, and cite the Benchmark paper: Khoshelham, K., Díaz Vilariño, L., Peter, M., Kang, Z., and Acharya, D.: THE ISPRS BENCHMARK ON INDOOR MODELLING, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-2/W7, 367-372, https://doi.org/10.5194/isprs-archives-XLII-2-W7-367-2017, 2017.
The ISPRS website for the benchmark dataset was set up in September 2017. Since then, the dataset has been downloaded by 154 researchers from 23 countries. The following figure shows the download statistics of the benchmark dataset (last updated April 29th, 2020)
The ISPRS Benchmark test on Indoor Modelling will continue to be open for submission until June 2020. If you are interested in participating in the test, please submit your models before this date. The evaluation results may be used by the participants for any scientific publication.
Submission of models
The quantitative evaluation is limited to walls, floors, ceilings, doors and windows. Other elements, such as stairs and columns, are excluded from the evaluation.
Participants can submit their models in any format. In addition to the reconstructed models please provide a submission name, a short explanation of the reconstruction method, and if applicable a link to any publication or open source implementation of the method. Please also include a readme file containing instructions on how to visualize the model and access its components. Please compress your files in a zip file and send it via email to L. Díaz-Vilariño.
When preparing your models for submission please note that for the evaluation only wall surfaces that have corresponding points in the point cloud are considered correct and the presence of wall surfaces without supporting points in the point cloud will result in low correctness scores in the evaluation results. To improve the correctness for your submissions you can modify your algorithm to include a binary (true or false) attribute for each surface indicating whether or not it has been reconstructed from points observed in the point cloud. Detailed information about the evaluation method can be found in this paper and this paper. The evaluation results will be published on the Benchmark Results page.