Marine Organism Detection and Ecological Interpretation Using Vision–Language Models and Multimodal RAG

抄録

As the demand for marine ecological monitoring using visual information continues to increase, there is a growing need for the automatic identification of data-deficient organisms and automated ecological interpretation integrating environmental information. Despite this need, it remains poorly understood whether domain-specific AI systems possess such capabilities. Vision–language models (VLMs) offer a compelling advantage by enabling the identification of previously unseen objects without additional labeled training data, as well as ecological interpretation based on both visual and external information. To address these challenges, we combined VLMs with Retrieval-Augmented Generation (RAG) to detect data-deficient organisms and generate ecological interpretations from visual, information, literately knowledge, and environmental data. Task 1 compared the detection accuracy of the newly recorded shrimp Metabetaeus lapillicola using a VLM with and without RAG, in which the RAG system accessed zoological-taxonomic papers. Task 2 evaluated ecological interpretations generated by the VLM from environmental data, visual information, and scientific descriptions. In Task 1, although the VLM without RAG misidentified M. lapillicola as another arthropod, the RAG-enhanced VLM correctly identified the species by integrating visual features with literature-based morphological information. In Task 2, the customized model generated scientifically reasonable ecological interpretations by integrating these three sources of information. These results are the first to demonstrate the utility of multimodal RAG for classifying marine organisms independently of data availability and for providing ecological interpretations through the integration of quantitative environmental data. This framework may overcome the limitations of previous automatic monitoring approaches simply by updating the documents and environmental datasets stored in a vector database. Ultimately, this approach may lead to the development of a multimodal AI system capable of performing a complete workflow—from detecting diverse marine organisms to generating region-specific ecological interpretations—by integrating these two capabilities.