University of Cambridge Cambridge quick links
Study at Cambridge About the University Research at Cambridge
Logo

MENU

Back to publications

Prompt Variability Effects on LLM Code Generation

Andrei Paleyes, Radzim Sendyka, Diana Robinson, Christian Cabrera, Neil D. Lawrence
Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026, 2026.

Abstract

Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user’s background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM’s sensitivity to input variations, independent of specific programming tasks or models.

Links

Home Page arXiv version of this paper Edit this information on GitHub

Cite this Paper

BibTeX
@InProceedings{publications/prompt-variability-effects-on-llm-code-generation, title = {Prompt Variability Effects on LLM Code Generation}, author = {Paleyes, Andrei and Sendyka, Radzim and Robinson, Diana and Cabrera, Christian and Lawrence, Neil D.}, booktitle = {Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026}, year = {2026}, url = {/publications/prompt-variability-effects-on-llm-code-generation.html}, abstract = {Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user's background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM's sensitivity to input variations, independent of specific programming tasks or models.} }
Endnote
%0 Conference Paper %T Prompt Variability Effects on LLM Code Generation %A Andrei Paleyes %A Radzim Sendyka %A Diana Robinson %A Christian Cabrera %A Neil D. Lawrence %B Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026 %D 2026 %F publications/prompt-variability-effects-on-llm-code-generation %U /publications/prompt-variability-effects-on-llm-code-generation.html %X Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user's background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM's sensitivity to input variations, independent of specific programming tasks or models.
RIS
TY - CPAPER TI - Prompt Variability Effects on LLM Code Generation AU - Andrei Paleyes AU - Radzim Sendyka AU - Diana Robinson AU - Christian Cabrera AU - Neil D. Lawrence BT - Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026 DA - 2026/04/01 ID - publications/prompt-variability-effects-on-llm-code-generation UR - /publications/prompt-variability-effects-on-llm-code-generation.html AB - Large language models lower barriers to writing code, but the quality of generated programs is sensitive to prompt quality and the user's background. We propose a synthetic evaluation pipeline and a systematic persona-based evaluation approach to quantify an LLM's sensitivity to input variations, independent of specific programming tasks or models. ER -
APA
Paleyes, A., Sendyka, R., Robinson, D., Cabrera, C. & Lawrence, N.D.. (2026). Prompt Variability Effects on LLM Code Generation. Third International Workshop on Large Language Models for Code (LLM4Code 2026), co-located with ICSE 2026 Available from /publications/prompt-variability-effects-on-llm-code-generation.html.