| Kurzfassung
|
Domain-specific model transformation languages such as the Reactions Language or ATL are vital in model-driven software development but remain largely absent from the training data of large language models (LLMs). As a result, generated code often contains severe syntactic and semantic errors. This thesis presents an evaluation pipeline, implemented in n8n and Docker, that systematically assesses LLM output on such languages. Metrics include syntax validity via parsing, syntactic closeness using ChrF, and semantic correctness through unit tests. A baseline across multiple LLMs (GPT, Claude, Gemini) is established and improvement strategies such as grammar prompting, few-shot prompting, and auxiliary descriptions of methods and variables is investigated. The pipeline enables reproducible experiments across languages and strategies. Results show that structured prompting and contextual aids can substantially increase correctness compared to baseline generation.
|
