RT Journal Article
T1 A framework for assessing the capabilities of code generation of constraint domain-specific languages with large language models
A1 Delgado, David
A1 Burgueño-Caballero, Lola
A1 Clarisó, Robert
K1 Lenguajes de programación lógicos
K1 Ingeniería del software
K1 Inteligencia artificial
K1 Calidad
AB Large language models (LLMs) can be used to support software development tasks, e.g., through code completionor code generation. However, their effectiveness drops significantly when considering less popular programminglanguages such as domain-specific languages (DSLs). In this paper, we propose a generic framework for evaluatingthe capabilities of LLMs generating DSL code from textual specifications. The generated code is assessed from theperspectives of well-formedness and correctness. This framework is applied to a particular type of DSL, constraintlanguages, focusing our experiments on OCL and Alloy and comparing their results to those achieved for Python,a popular general-purpose programming language. Experimental results show that, in general, LLMs have betterperformance for Python than for OCL and Alloy. LLMs with smaller context windows such as open-source LLMsmay be unable to generate constraint-related code, as this requires managing both the constraint and the domainmodel where it is defined. Moreover, some improvements to the code generation process such as code repair(asking an LLM to fix incorrect code) or multiple attempts (generating several candidates for each coding task)can improve the quality of the generated code. Meanwhile, other decisions like the choice of a prompt templatehave less impact. All these dimensions can be systematically analyzed using our evaluation framework, makingit possible to decide the most effective way to set up code generation for a particular type of task.
PB Elsevier
SN 0164-1212
YR 2026
FD 2026
LK https://hdl.handle.net/10630/46312
UL https://hdl.handle.net/10630/46312
LA eng
NO David Delgado, Lola Burgueño, Robert Clarisó, A framework for assessing the capabilities of code generation of constraint domain-specific languages with large language models, Journal of Systems and Software, Volume 238, 2026, 112871, ISSN 0164-1212, https://doi.org/10.1016/j.jss.2026.112871.
NO Ministerio de Ciencia, Innovación y Universidades
NO Funding for open access charge: Universidad de Málaga / CBUA
NO Universitat Oberta de Catalunya
DS RIUMA. Repositorio Institucional de la Universidad de Málaga
RD 5 may 2026