Watermarking in vLLM

What is text provenance, and why is it challenging?

Establishing the provenance of digital content is essential to build trust and accountability in the information we share. Watermarking offers a solution: it alters a work to embed a message about it [1], for instance its provenance. Images and audio can embed watermarks as a subtle perturbation, but these techniques are not applicable to text due to its discrete nature. Instead of trying to perturb the text itself, we influence the generation process in a way that is detectable but does not modify the expected output distribution.

Watermarking text is both a scientific and engineering challenge, and there are several requirements that we must aim to balance to have a practical solution:

Non-distortion

Adding a watermark should not modify the expected output distribution of a model [2]: it should not systematically favour certain words, writing styles, or solutions to problems.

Robustness

Generated text can be altered in many ways: shortened, edited, mixed with other corpora, or even rephrased by another LLM. A good watermarking system aims to preserve its provenance signal even under such modification.

Speed

Watermarking should add minimal latency and memory overhead to generation and detection. This is particularly important in a high-throughput serving engine such as vLLM. Slowdowns arise both when serving small models, sensitive to changes in scheduling latency, and models with large vocabulary sizes.

Minimal detection dependencies

Detecting watermarking on text should require as little information as possible: users seeking to determine the source of a piece of writing may not necessarily have additional information on its origin, such as its date of creation or what language model may have been used to generate it.

These desired qualities pull in different directions: a strong signal that is easily detectable may require us to significantly change a model’s outputs, and avoiding reliance on additional information limits the algorithms available to us.

For example:

• Metadata attached to a file can be stripped off, and they do not persist when copying the data into a new file.
• Red-green lists [3] and direct data watermarking can lead to a distortion of the output distribution.
• Unicode substitution is easily removed and tampered with.
• Text modeling techniques (e.g. looking at sentence length, unusual words and expressions) are unreliable.
• Tournament sampling [2] requires multiple rounds of processing that can be hard to implement efficiently without impacting inference speed

To satisfy these criteria, we exploit a natural property of language model generations: randomness.

Exploiting randomness in LLMs

At each generation step, a language model assigns a probability to every possible next token, then samples one. This process traces a path through a tree of possible continuations: some of these paths are more likely than others, but many are valid final outputs.

Ordinary stochastic sampling draws and discards fresh random values at every step. Watermarking instead derives reproducible keyed values from the recent context and each candidate token. Because the context and selected token remain in the output, a detector with the key can reconstruct these values after generation. Watermarked text will systematically align with these values, creating a signal that accumulates over a long enough sequence.

The next section discusses how we can implement this process while still preserving the initial sampling distribution of the model, effectively allowing us to reproducibly choose one of the possible final outputs of the model.

Gumbel-max Trick

The Gumbel-max trick is a technique for sampling from a categorical distribution. Suppose the model assigns a probability to each candidate token : for every token, we draw independent uniform random variables and transform them into Gumbel random variables:

The sampled token is then:

A remarkable result of this transformation is that:

In other words, adding Gumbel noise to the token log-probabilities and taking their argmax produces the exact same categorical distribution as ordinary random sampling: in fact, Model Runner v2’s standard sampling path already uses this process. In practice, we can apply Gumbel noise directly to the logits, avoiding a softmax and making the computation more parallelizable.

To make these random draws reproducible, we use a pseudorandom function (PRF) [4]. A PRF deterministically maps a key and an input to a value that behaves like a random uniform draw: since the same inputs always produce the same output, a detector can reconstruct the values used during generation. We replace each independent uniform draw with:

where is the secret watermark key, and is the recent watermarking context (the last tokens).

Including the token ID is important as it allows every token to receive different noise, the recent context (by default, we use the last 4 tokens) means that the random noise we apply is different at almost every generation step, and the secret key makes this pattern unpredictable for anyone who does not have it.

The generator computes these keyed values for each candidate token, and selects the token with the largest noised log-probability. Given the same key and context this choice can be reproduced, but without the key it looks like an ordinary random sample.

This gives the method a precise guarantee of non-distortion: in expectation over keys, the probability of selecting token remains exactly . However, this alone does not guarantee that complete sequences will be distortion-free. We discuss this limitation and its solutions further in “Maintaining output diversity”.

Detecting a watermark

At detection time, we reverse the process used during generation. It does not require access to the model’s weights or logits, but it does require knowledge about the secret key, and the tokenizer.

A user will provide a sequence of text, with unknown origin, to a detector. This detector will convert the sequence of text back into token IDs. For each token , the detector will use the preceding context and the secret key to recompute its pseudorandom value:

For unwatermarked text, the observed tokens are independent of their keyed values, so behaves like a uniform random variable. Watermarked generations will instead favour tokens with larger values. We transform each value into a token-level score:

Because repeated contexts reproduce the same keyed values, the detector scores each context only once. If the text is not watermarked, the scores follow an exponential distribution with mean . Their sum,

therefore follows a Gamma distribution with shape and scale . This lets us compute a one-sided p-value: the probability that unwatermarked text would produce a score at least as large as . A small p-value is evidence that the text is consistent with the watermark.

Choosing a threshold trades false positives against false negatives. For a calibrated test, will flag approximately 1% of unwatermarked sequences as positives. In practice, we might also test against several keys, tokenizers, or watermarking configurations. This increases the chance of finding a large score by accident, requiring us to apply a multiple-testing correction. As the number of candidates grows, the threshold becomes smaller and our detection power falls.

Detection becomes stronger as independent evidence accumulates: long outputs provide more tokens to score, and high entropy steps give the watermark more opportunities to influence token selection. Short or predictable outputs therefore provide less signal.

Since the signal is encoded into the token choices themselves, copying and pasting a sufficiently long passage will preserve its evidence. Modifying the text will disrupt scores around the modified tokens, but the signal remains intact once these tokens are no longer in the recent context of following tokens.

How this was implemented in vLLM

The implementation was initially presented in the watermarking RFC, which highlighted that watermarking must integrate with final token sampling and speculative decoding. We therefore integrated it directly into Model Runner v2’s sampling pipeline. The initial implementation (PR #54053) connects GPUWatermarkSampler to an algorithm-specific Watermarker, allowing different schemes to share the same request and batch handling.

A naïve implementation would materialize a [batch, vocabulary] noise tensor, creating substantial temporary storage and memory traffic. vLLM instead fuses pseudorandom-number generation, the Gumbel transformation, and argmax reduction into a single GPU kernel. Philox generates four values per invocation, allowing the kernel to process four consecutive token IDs together.

A per-row mask lets the same fused sampler use keyed noise for watermarked requests and ordinary randomness for unwatermarked requests or repeated contexts.

Across eight keys, mean matched throughput changes ranged from −1.1% to +2.0% across batch sizes, with no significant slowdown.

Challenges

There are some major challenges with watermarking that required particularly interesting solutions. We will discuss two of the most important below: interaction with speculative decoding, and reduction in output diversity.

Speculative decoding

Speculative decoding proposes tokens from a draft distribution and accepts them against a target distribution. Applying the same watermark to both distributions preserves them individually, but can reduce their overlap and lower the acceptance rate. vLLM instead uses a dual-key implementation (PR #56122): is used for accepted draft tokens, while is used for target residual and bonus tokens. Separating the keys preserves the ordinary acceptance rate and keeps target sampling independent of rejected draft proposals.

The final text will contain tokens watermarked by one key or the other, so we must use both keys during scoring and combine their scores. Unfortunately, this dilutes the signal from both keys, and leads to a weakening of the detection signal.

This weighting can be calibrated to reflect both the expected source of each token and how much signal it carries. For example, draft tokens may be accepted more often when they are “easy” and therefore lower entropy, meaning they carry less watermarking signal.

Maintaining output diversity

Single-token non-distortion only guarantees the distribution at each generation step, not over a complete sequence. If a context repeats, so does its keyed noise, correlating token choices that ordinary sampling would make independently (see example below). Single-sequence non-distortion is a stronger guarantee that the distribution over complete sequences is also unchanged.

For example, imagine a fixed key that makes a model generate the following sequence:

1 + 1 +       -> 1   context: (1, +, 1, +)
1 + 1 + 1     -> +   context: (+, 1, +, 1)
1 + 1 + 1 +   -> 1   context: (1, +, 1, +)

After generating 1 again, the context (1, +, 1, +) has repeated. The fixed key therefore produces the same pseudorandom values that previously favoured the generation of 1. If the model’s probabilities are also similar, or the bias is strong enough, it is likely to select 1 again: in this case, the generation could enter an infinite repetition loop.

1 + 1 + 1 + 1 + 1 + (...)

This is a worst-case scenario, but the impact of this correlation across contexts may generally lead to lower diversity within a sequence, increased rates of degenerate behaviour, longer generations, and reduced output quality.

vLLM addresses this with generation-time context deduplication (PR #56233), which skips watermarking whenever a context repeats and preserves single-sequence non-distortion. Despite checking the preceding generation for repeated contexts at every step, it reduced end-to-end throughput by at most 0.19% on Qwen3.5-27B.

We can also take advantage of the previously discussed dual-key watermarking scheme, since the randomness from whether a token will come from the target or draft distribution mitigates the impact of this bad behaviour, and improves diversity across sequences. More generally, we can use dual-key routing without speculative decoding [5] by randomly selecting which key to use during generation.

Getting started

You can enable Gumbel-max watermarking when starting a vLLM server:

vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --watermark-config '{"algorithm":"gumbel","key":42}'

vLLM also provides a minimal HTTP detection server example for checking text using the matching tokenizer and watermark key. See the watermarking documentation for configuration and detection examples.

Conclusion

vLLM now supports distortion-free watermarking using the Gumbel-max algorithm, with efficient GPU generation and speculative decoding support. Our evaluations show minimal performance overhead while preserving output quality and diversity.

Acknowledgements

We thank Wassim Bouaziz, Andy Lo, Nicolò Lucchesi, Victor Paltz, Leila Saidi, and Mickaël Seznec for their thoughtful review and feedback on the article, and Lucas Wilkinson for his contributions to the implementation and review.

Appendix

Throughput measurements

The measurements use Qwen3.5-27B on one H100 with MTP-3 and 512 output tokens. Throughput values are medians across eight watermarked keys, and four unwatermarked controls.

References

1. Ingemar J. Cox, Matthew L. Miller, Jeffrey A. Bloom, Jessica Fridrich, and Ton Kalker. “Digital Watermarking and Steganography.” 2nd ed., Morgan Kaufmann (2008).

2. Sumanth Dathathri et al. “Scalable watermarking for identifying large language model outputs.” Nature 634, 818–823 (2024). Supplementary information, §G.3.

3. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. “A Watermark for Large Language Models.” Proceedings of the 40th International Conference on Machine Learning (2023).

4. Scott Aaronson and Hendrik Kirchner. “Watermarking GPT outputs,” 2023.

5. Tom Sander et al. “TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection.” arXiv:2605.12456 (2026).