Semantic segmentation and sensor fusion in point cloud processing for estimation of bridge structural parameters
DOI:
https://doi.org/10.51094/jxiv.866Keywords:
Deep Learning, LiDAR, Point Cloud, Segment Anything Model, Sensor FusionAbstract
In this study, we implemented sensor fusion by combining point cloud data and image data obtained from a terrestrial laser scanner, and mapped the segmentation results from the images onto the point clouds. Semantic segmentation of the images was performed using DeepLabv3+ to classify into wheel guard and background. Also, the edge information was updated using the Segment Anything Model, then segmentation information was stored in the point clouds using the camera's external and internal parameters. Utilizing this information enabled the measurement of bridge widths. By leveraging the detailed information from the image data and the three-dimensional information from the point cloud data, we were able to achieve an analysis that extract detail and structural information, while also efficiently processing large files of point cloud data.
Conflicts of Interest Disclosure
There are no conflicts of interest to disclose.Downloads *Displays the aggregated results up to the previous day.
References
辻井純平，合田哲朗，中野雅章：土木構造物の点群解析に向けた局所形状の畳み込みを伴う深層学習手法の適用，AI・データサイエンス論文集，4巻3号，pp. 442-450，2023.
米山睦美，髙見澤拓哉，眞部達也，田尻大介：鉄筋結束マシンに取り付けたカメラ動画から生成した点群による配筋検査への実用性検証，AI・データサイエンス論文集，5巻1号，pp. 239-244，2024.
関和彦，山口愛加，窪田 諭：3次元点群データを用いた道路橋の損傷抽出とヒートマップ表示，土木学会論文集，79巻10 号，2023.
山崎文敬，前田幸祐，柿市拓巳：橋梁コンクリート床版平坦性自動検査システムの開発，AI・データサイエンス論文集，5巻1号，pp. 26-32，2024.
Kamiwaki, Y., and Fukuda, S.: A Machine Learning-Assisted Three-Dimensional Image Analysis for Weight Estimation of Radish, Horticulturae, Vol. 10, Issue 2, pp. 142, 2024.
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds, CVPR, pp. 12697-12705, 2019.
Qi, C. R., Su, H., Mo, K., and Guibas, L. J.: Pointnet: Deep learning on point sets for 3d classification and segmentation, CVPR, pp. 652-660, 2017.
Qi, C. R., Yi, L., Su, H., and Guibas, L. J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space, NeurIPS, 30. (2017).
Phan, A. V., Le Nguyen, M., Nguyen, Y. L. H., and Bui, L. T.: DGCNN: A convolutional neural network over large-scale labeled graphs, Neural Networks, Vol. 108, pp. 533-543, 2018.
Liu, H., Wu, C., and Wang, H.: Real time object detection using LiDAR and camera fusion for autonomous driving, Sci. Rep., Vol. 13, pp. 8056, 2023.
Hosoi, F., Umeyama, S., and Kuo, K.: Estimating 3D chlorophyll content distribution of trees using an image fusion method between 2D camera and 3D portable scanning lidar, Remote Sens., Vol. 11, Issue 18, pp. 2134, 2019.
Narváez, F. J. Y., del Pedregal, J. S., Prieto, P. A., Torres-Torriti, M., and Cheein, F. A. A.: LiDAR and thermal images fusion for ground-based 3D characterisation of fruit trees, Biosys. Eng., Vol. 151, pp. 479-494, 2016.
Ronneberger, O., Fischer, P., and Brox, T.: U-net: Convolutional networks for biomedical image segmentation, MICCAI, pp. 234-241, 2015.
Chen, L. C., Papandreou, G., Schroff, F., and Adam, H.: Rethinking atrous convolution for semantic image segmentation, arXiv:1706.05587, 2017.
Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation, ECCV, pp. 801-818. 2018.
Badrinarayanan, V., Kendall, A., and Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation, TPAMI, Vol. 39, Issue 12, pp. 2481-2495, 2017.
劉佳明，党紀，全邦釘：DeepLabv3+を用いた橋梁腐食損傷とその精度の向上，AI・データサイエンス論文集，3巻J2号，pp. 802-810，2022.
Deng, L., Yuan, H., Long, L., Chun, P. J., Chen, W., Chu, H.: Cascade refinement extraction network with active boundary loss for segmentation of concrete cracks from high-resolution images, Autom. Constr., Vol. 162, pp. 105410, 2024.
Chu, H., and Chun, P. J.: Fine-grained crack segmentation for high‐resolution images via a multiscale cascaded network. Comput. Aided Civ. Infra. Eng., Vol. 39, No.4, pp. 575-594, 2024.
斎藤嘉人，板倉健太，山本一哉，二宮和則，近藤直：可視・近赤外画像のセマンティックセグメンテーションによるバレイショ塊茎表面の病害検出，AI・データサイエンス論文集，3巻J2号，pp. 175-181，2022.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W. Y., Dollar, P., Girshick, R.: Segment anything, ICCV, pp. 4015-4026, 2023.
板倉健太，林拓哉，上脇優人，全邦釘：LiDARとカメラのセンサーフュージョンによる点群からのノイズ除去，AI・データサイエンス論文集，受理済み，2024.
Jiao, J., Chen, F., Wei, H., Wu, J., and Liu, M.: LCE-Calib: Automatic lidar-frame/event camera extrinsic calibration with a globally optimal solution, IEEE/ASME Transactions on Mechatronics, Vol. 28, No. 5, pp. 2988-2999, 2023.
Wang, H., Xiao, C., Kossaifi, J., Yu, Z., Anandkumar, A., and Wang, Z.: Augmax: Adversarial composition of random augmentations for robust training, NIPS, Vol. 34, pp. 237-250, 2021.
Ho, Y., and Wookey, S.: The real-world-weight cross-entropy loss function: Modeling the costs of mislabeling, IEEE access, Vol. 8, pp. 4806-4813, 2019.
増田宏：大規模点群のための処理技術，計測と制御，60巻10号，pp. 716-720，2021.
Pan, X., Shi, J., Luo, P., Wang, X., and Tang, X.: Spatial as deep: Spatial CNN for traffic scene understanding, AAAI, Vol. 32, No. 1, pp. 7276-7283, 2018.
Itakura, K., and Hosoi, F.: Three-dimensional tree monitoring in urban cities using automatic tree detection method with mobile LiDAR data, AI Data Sci., Vol. 2, Issue 2, pp. 1-10, 2021.
Itakura, K., and Hosoi, F.: Automated tree detection from 3D lidar images using image processing and machine learning, Appl. Opt., Vol. 58, Issue 14, pp. 3807-3811, 2019.
板倉健太，細井文樹：画像処理や3次元深層学習を用いた航空機ライダー点群データからの樹木の検出，AI・データサイエンス論文集，1巻J1号，pp. 320-328，2020.
Itakura, K., and Hosoi, F.: Voxel-based leaf area estimation from three-dimensional plant images. J. Agri. Meteorol., Vol. 75, Issue 4, pp. 211-216, 2019.
Itakura, K., and Hosoi, F.: Automatic tree detection from three-dimensional images reconstructed from 360 spherical camera using YOLO v2, Remote Sens., Vol. 12, Issue 6, pp. 988, 2020.
Itakura, K., and Hosoi, F.: Simple and effective tool for estimating tree trunk diameters and tree species classification, Appl. Opt., Vol. 59, Issue 2, pp. 558-563, 2020.
Fischler, M. A., and Bolles, R. C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Com. ACM, Vol. 24, Issue 6, pp. 381-395, 1981.
Torr, P. H., and Zisserman, A., MLESAC: A new robust estimator with application to estimating image geometry, Comput. Vis. Image Und., Vol. 78, Issue 1, pp. 138-156, 2000.
渡邊祥庸，井上和真，池田隆明，志賀正崇，小林雅人，横山和佳奈：スマートフォンLiDARとネットワーク型RTK測位により作成した3次元地形モデルの小規模土工事への適用範囲の検討，AI・データサイエンス論文集，5巻1号，pp. 260-268，2024.
Yamane, T., Chun, P. J., Dang, J., and Honda, R. (2023). Recording of bridge damage areas by 3D integration of multiple images and reduction of the variability in detected results. Comput. Aided Civ. Infra. Eng., Vol. 38, Issue 17, pp. 2391-2407.
Yamane, T., Chun, P. J., and Honda, R.: Detecting and localizing damage based on image recognition and structure from motion, and reflecting it in a 3D bridge model. Struct. Infrastruct. Eng., Vol. 20, Issue. 4, pp. 594-606, 2024.
藤原圭哉，佐藤誠，山下千智，黒田直樹，亀田敏弘：3Dデータと河床変動解析を活用した河川分野におけるデジタルツインの実現に向けての提案，AI・データサイエンス論文集，5巻1号，pp. 126-133，2024.
Downloads
Posted
Submitted: 2024-08-29 07:55:53 UTC
Published: 2024-09-03 02:20:38 UTC
License
Copyright (c) 2024
Kenta Itakura
Takuya Hayashi
Yuto Kamiwaki
Pang-jo Chun
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
