2026-06-01T08:40:10Zhttps://riuma.uma.es/rest/oai/request

oai:riuma.uma.es:10630/326302026-02-03T11:05:42Zcom_10630_2254col_10630_37953

Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework Cámara-Moreno, Javier Burgueño-Caballero, Lola Troya-Castilla, Javier Empresas - Gestión The integration of Large Language Models (LLMs) in software modeling tasks presents both opportunities and challenges. This Expert Voice addresses a significant gap in the evaluation of these models, advocating for the need for standardized benchmarking frameworks. Recognizing the potential variability in prompt strategies, LLM outputs, and solution space, we propose a conceptual framework to assess their quality in software model generation. This framework aims to pave the way for standardization of the benchmarking process, ensuring consistent and objective evaluation of LLMs in software modeling. Our conceptual framework is illustrated using UML class diagrams as a running example. 2024-09-18T12:15:35Z 2024-09-18T12:15:35Z 2024-09-18T12:15:35Z 2024-09-03 journal article Cámara, J., Burgueño, L. & Troya, J. Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework. Softw Syst Model (2024). https://doi.org/10.1007/s10270-024-01206-9 https://hdl.handle.net/10630/32630 10.1007/s10270-024-01206-9 eng http://creativecommons.org/licenses/by/4.0/ open access Atribución 4.0 Internacional Springer