| About this Abstract |
| Meeting |
MS&T24: Materials Science & Technology
|
| Symposium
|
Materials Informatics for Images and Multi-dimensional Datasets
|
| Presentation Title |
Foundation Models for Multimodal Data Mining with Applications in Materials Science |
| Author(s) |
Aikaterini Vriza, Maria Chan, Henry Chan, Jie Xu |
| On-Site Speaker (Planned) |
Aikaterini Vriza |
| Abstract Scope |
In the rapidly evolving landscape of artificial intelligence (AI), Foundation Models (FMs) have emerged as transformative tools with potential applications across many scientific sectors. While there exist considerable efforts in the materials science field on using Large Language Models (LLMs) for text, research focusing on image data extraction related to spectroscopy and microscopy remains sparse.
This study explored the potential of FMs for data mining multimodal datasets related to materials characterization techniques. Our methodology involves augmenting a cutting-edge data mining toolkit designed to extract image-text pairs from scientific journals, which can be foundational for creating multimodal models and advancing semantic searches. A direct comparison of FMs to human evaluators highlighted the proficiency of these models in discerning scientific research, adeptly categorizing images, and allocating relevant tags.
Overall, the study suggests FMs hold promise for materials science but highlights the importance of ensuring accurate and safe applications. |