RT Journal Article
T1 Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework
A1 Cámara-Moreno, Javier
A1 Burgueño-Caballero, Lola
A1 Troya-Castilla, Javier
K1 Empresas - Gestión
AB The integration of Large Language Models (LLMs) in software modeling tasks presents both opportunities and challenges.This Expert Voice addresses a significant gap in the evaluation of these models, advocating for the need for standardizedbenchmarking frameworks. Recognizing the potential variability in prompt strategies, LLM outputs, and solution space, wepropose a conceptual framework to assess their quality in software model generation. This framework aims to pave the wayfor standardization of the benchmarking process, ensuring consistent and objective evaluation of LLMs in software modeling.Our conceptual framework is illustrated using UML class diagrams as a running example.
PB Springer
YR 2024
FD 2024-09-03
LK https://hdl.handle.net/10630/32630
UL https://hdl.handle.net/10630/32630
LA eng
NO Cámara, J., Burgueño, L. & Troya, J. Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework. Softw Syst Model (2024). https://doi.org/10.1007/s10270-024-01206-9
NO Funding for open access charge: Universidad de Málaga / CBUA
DS RIUMA. Repositorio Institucional de la Universidad de Málaga
RD 12 abr 2026