Bryan Eikema

Bryan Eikema

I’m a Founding Member of Technical Staff at Parallax.

My research examines language generation through the lens of probabilistic modelling of sequences. From this perspective, I study models’ underlying beliefs, the uncertainty around those beliefs, and how to make decisions in the face of that uncertainty.

Before joining Parallax, I was a postdoctoral researcher at the Institute for Logic, Language and Computation (ILLC) at the University of Amsterdam. I also completed my PhD there under the supervision of Wilker Aziz. I previously interned at NAVER Labs Europe and Shopify.

Sample-Based Decision-Making Ancestral samples from a sequence model are used to estimate Bayes risk. The highlighted candidate minimizes estimated risk over the sampled candidate set.
What’s this?

This sketch illustrates decision-making under uncertainty with sampling-based minimum Bayes risk (MBR) decoding. We draw sequences from an autoregressive language model and treat each one as a candidate. Averaging a candidate’s loss over all the draws gives a Monte Carlo estimate of its expected loss, or risk. The highlighted sequence has the lowest estimated risk. In language generation, we often express the same decision as maximising expected utility.

Here, the loss between two sequences is the fraction of token positions where they differ (normalised Hamming loss). The point layout is illustrative.

Selected Research

All publications
2025

arXiv

Teaching Language Models to Faithfully Express their Uncertainty

Bryan Eikema, Evgenia Ilia, José G. C. de Souza, Chrysoula Zerva, Wilker Aziz

Paper Code
Abstract for Teaching Language Models to Faithfully Express their Uncertainty
Large language models (LLMs) often miscommunicate their uncertainty: repeated queries can produce divergent answers, yet generated responses are typically unhedged or hedged in ways that do not reflect this variability. This conveys unfaithful information about the uncertain state of the LLMs’ knowledge, creating a faithfulness gap that affects even strong LLMs. We introduce Faithful Uncertainty Tuning (FUT): a fine-tuning approach that teaches instruction-tuned LLMs to express uncertainty faithfully without altering their underlying answer distribution. We construct training data by augmenting model samples with uncertainty hedges (i.e. verbal cues such as ‘possibly’ or ’likely’) aligned with sample consistency, requiring no supervision beyond the model and a set of prompts. We evaluate FUT on open-domain question answering (QA) across multiple models and datasets. Our results show that FUT substantially reduces the faithfulness gap, while preserving QA accuracy and introducing minimal semantic distribution shift. Further analyses demonstrate robustness across decoding strategies, choice of hedgers, and other forms of uncertainty expression (i.e. numerical). These findings establish FUT as a simple and effective way to teach LLMs to communicate uncertainty faithfully.
Cite Teaching Language Models to Faithfully Express their Uncertainty
@misc{eikema2025teachinglanguagemodelsfaithfully,
      title={Teaching Language Models to Faithfully Express their Uncertainty}, 
      author={Bryan Eikema and Evgenia Ilia and José G. C. de Souza and Chrysoula Zerva and Wilker Aziz},
      year={2025},
      eprint={2510.12587},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.12587}, 
}
2025

EMNLP

Structure-Conditional Minimum Bayes Risk Decoding

Bryan Eikema, Anna Rutkiewicz, Mario Giulianelli

Paper Code
Abstract for Structure-Conditional Minimum Bayes Risk Decoding
Minimum Bayes Risk (MBR) decoding has seen renewed interest as an alternative to traditional generation strategies. While MBR has proven effective in machine translation, where the variability of a language model’s outcome space is naturally constrained, it may face challenges in more open-ended tasks such as dialogue or instruction-following. We hypothesise that in such settings, applying MBR with standard similarity-based utility functions may result in selecting responses that are broadly representative of the model’s distribution, yet sub-optimal with respect to any particular grouping of generations that share an underlying latent structure. In this work, we introduce three lightweight adaptations to the utility function, designed to make MBR more sensitive to structural variability in the outcome space. To test our hypothesis, we curate a dataset capturing three representative types of latent structure—dialogue act, emotion, and response structure (e.g., a sentence, a paragraph, or a list)—and we propose two metrics to evaluate the structural optimality of MBR. Our analysis demonstrates that common similarity-based utility functions fall short by these metrics. In contrast, our proposed adaptations considerably improve structural optimality. Finally, we evaluate our approaches on real-world instruction-following benchmarks, AlpacaEval and MT-Bench, and show that increased structural sensitivity improves generation quality by up to 13.7 percentage points in win rate.
Cite Structure-Conditional Minimum Bayes Risk Decoding
@inproceedings{eikema-etal-2025-structure,
    title = "Structure-Conditional Minimum {B}ayes Risk Decoding",
    author = "Eikema, Bryan  and
      Rutkiewicz, Anna  and
      Giulianelli, Mario",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1616/",
    doi = "10.18653/v1/2025.emnlp-main.1616",
    pages = "31706--31723",
    ISBN = "979-8-89176-332-6",
    abstract = "Minimum Bayes Risk (MBR) decoding has seen renewed interest as an alternative to traditional generation strategies. While MBR has proven effective in machine translation, where the variability of a language model{'}s outcome space is naturally constrained, it may face challenges in more open-ended tasks such as dialogue or instruction-following. We hypothesise that in such settings, applying MBR with standard similarity-based utility functions may result in selecting responses that are broadly representative of the model{'}s distribution, yet sub-optimal with respect to any particular grouping of generations that share an underlying latent structure. In this work, we introduce three lightweight adaptations to the utility function, designed to make MBR more sensitive to structural variability in the outcome space. To test our hypothesis, we curate a dataset capturing three representative types of latent structure{---}dialogue act, emotion, and response structure (e.g., a sentence, a paragraph, or a list){---}and we propose two metrics to evaluate the structural optimality of MBR. Our analysis demonstrates that common similarity-based utility functions fall short by these metrics. In contrast, our proposed adaptations considerably improve structural optimality. Finally, we evaluate our approaches on real-world instruction-following benchmarks, AlpacaEval and MT-Bench, and show that increased structural sensitivity improves generation quality by up to 13.7 percentage points in win rate."
}
2022

TMLR

An Approximate Sampler for Energy-based Models with Divergence Diagnostics

Bryan Eikema, Germán Kruszewski, Christopher R Dance, Hady Elsahar, Marc Dymetman

Paper
Abstract for An Approximate Sampler for Energy-based Models with Divergence Diagnostics
Energy-based models (EBMs) allow flexible specifications of probability distributions. However, sampling from EBMs is non-trivial, usually requiring approximate techniques such as Markov chain Monte Carlo (MCMC). A major downside of MCMC sampling is that it is often impossible to compute the divergence of the sampling distribution from the target distribution: therefore, the quality of the samples cannot be guaranteed. Here, we introduce quasi-rejection sampling (QRS), a simple extension of rejection sampling that performs approximate sampling, but, crucially, does provide divergence diagnostics (in terms of f-divergences, such as KL divergence and total variation distance). We apply QRS to sampling from discrete EBMs over text for controlled generation. We show that we can sample from such EBMs with arbitrary precision in exchange for sampling efficiency and quantify the trade-off between the two by means of the aforementioned diagnostics.
Cite An Approximate Sampler for Energy-based Models with Divergence Diagnostics
@article{
    eikema-etal-2022-approximate,
    title={An approximate sampler for energy-based models with divergence diagnostics},
    author={Bryan Eikema and Germ{\'a}n Kruszewski and Christopher R Dance and Hady Elsahar and Marc Dymetman},
    journal={Transactions on Machine Learning Research},
    issn={2835-8856},
    year={2022},
    url={https://openreview.net/forum?id=VW4IrC0n0M},
    note={}
}
2020

COLINGBest Paper Award

Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation

Bryan Eikema and Wilker Aziz

Paper
Abstract for Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
Recent studies have revealed a number of pathologies of neural machine translation (NMT) systems. Hypotheses explaining these mostly suggest there is something fundamentally wrong with NMT as a model or its training algorithm, maximum likelihood estimation (MLE). Most of this evidence was gathered using maximum a posteriori (MAP) decoding, a decision rule aimed at identifying the highest-scoring translation, i.e. the mode. We argue that the evidence corroborates the inadequacy of MAP decoding more than casts doubt on the model and its training algorithm. In this work, we show that translation distributions do reproduce various statistics of the data well, but that beam search strays from such statistics. We show that some of the known pathologies and biases of NMT are due to MAP decoding and not to NMT’s statistical assumptions nor MLE. In particular, we show that the most likely translations under the model accumulate so little probability mass that the mode can be considered essentially arbitrary. We therefore advocate for the use of decision rules that take into account the translation distribution holistically. We show that an approximation to minimum Bayes risk decoding gives competitive results confirming that NMT models do capture important aspects of translation well in expectation.
Cite Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
@inproceedings{eikema-aziz-2020-is,
    title = "Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation",
    author = "Eikema, Bryan  and
      Aziz, Wilker",
    booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
    month = dec,
    year = "2020",
    address = "Barcelona, Spain",
    publisher = "Association for Computational Linguistics",
}

Community & Teaching

Selected workshops, teaching, and shared tasks.

Selected Talks

Open-Source Software

GitHub

unctune

Teaching LLMs to faithfully express uncertainty in natural language.

scmbr

Structure-conditional adaptations to minimum Bayes risk decoding.

AEVNMT.pt

A PyTorch-based framework for deep generative models of text.