Trabalho em anais de evento
LLM-based text-to-JQL Generation: an automated test using execution and structural indicators
LIMA, RAYFRAN ROCHA; DE BRITO, IRINEU EVANGELISTA CRUZ; NASCIMENTO, JARDEL DA CUNHA; CORDOVIL-JR, LUIZ
Resumo
Resumo
Evaluating natural language to query language (NL-to-QL) systems cannot rely on string matching alone, especially when queries involve dynamic data or complex query languages. We present APOLLO, an automated test framework that executes generated queries by Large Language Models (LLMs) against a live Jira instance, enabling execution-aware evaluation and detailed error diagnosis across syntax, unsupported features, and value inaccuracies. APOLLO introduces new metrics—field accuracy, value accuracy, and inference accuracy—that capture correctness beyond similarity. Using APOLLO, we evaluated two LLM-based conversion architectures: a retrieval-augmented generator and a specialized API-supported generation model that demonstrated both limitations and alternatives to address the drawbacks of generating dynamic non-pre-trained data. Future work will extend the framework to additional query languages, explore automated remediation strategies, and investigate richer semantic metrics for deeper diagnostic insight.
Autoria
Autores do grupo (1)
A lista completa de autores está no topo da página. Abaixo, apenas os pesquisadores do e-Controls vinculados a esta publicação.
palavras-chave
Palavras-chave
natural language to query translationlarge language modelstest automationerror diagnosisinference accuracyretrieval-augmented generationdynamic data
Status
Published
links
Links e arquivos
Trabalho em anais de evento