AI watermarking designed to identify machine-generated text can alter model behaviour and help harmful prompts succeed.
According to Ars Technica, research from AI security outfit Lasso Security found that Google’s SynthID-Text can change whether large language models refuse dangerous requests, particularly when attackers use prompt injection.
Anthropic recently said future Claude models will use SynthID-Text, which Google DeepMind developed and released as open source.
The technology is intended to provide provenance for generated text by embedding an invisible statistical signal in the words a model chooses.
This has gained extra relevance because Article 50 of the EU AI Act requires providers of generative AI systems to mark synthetic output in a machine-readable and detectable form.
SynthID-Text does not bolt a label onto finished text. Instead, it interferes politely with the model’s token-selection process while the answer is being generated.
Its tournament-sampling system considers competing next-token candidates and uses a secret key to influence which candidate advances. Someone possessing the key can later test whether the resulting token sequence carries the watermark.
The trick is supposed to leave the meaning essentially untouched while producing enough statistical evidence to identify AI-generated material.
Unfortunately, fiddling with token selection can have consequences beyond swapping one harmless word for another.
Lasso Security researcher Andrea Siposova tested the non-distortionary SynthID-Text configuration using Hugging Face’s unmodified SynthIDTextWatermarkLogitsProcessor.
She compared six open-weight models generating responses with and without watermarking while keeping the seed, batch composition and ordering identical.
The tests included 200 harmful behaviours from HarmBench and 100 benign controls from JailbreakBench. Researchers tested harmful requests both directly and with a prompt-injection technique.
“Watermarking changes refusal behaviour on bare harmful requests,” Siposova wrote.
The difference became more pronounced with prompt injection. Several models became more willing to answer harmful requests after researchers switched on watermarking.
The researchers found the changes didn’t stop at text responses. In agentic systems, token selection can determine which external tool an AI chooses and what arguments it passes to that tool.
Lasso tested tool calling using BFCL v4 single-turn tasks and found that watermarking could change individual calls from correct to incorrect, or push previously incorrect calls in the other direction.
The researchers call this effect “sampling drift”, where apparently minor changes to token sampling ripple into observable changes in model and agent behaviour.
Different secret watermarking keys produced different results too. Some increased harmful compliance compared with an unwatermarked model while others reduced it, suggesting the key itself can influence behaviour.
The work did not test Claude itself. Researchers used six open-weight models because they needed access to token sampling so they could turn watermarking on and off while keeping other conditions fixed. The experiments used Hugging Face’s implementation of SynthID-Text tournament sampling rather than whatever implementation Anthropic eventually deploys in Claude.
That means the research does not establish that watermarked Claude models will suddenly start cheerfully ignoring their safety rules. It does show that watermarking can change more than the provenance signal it was designed to introduce, particularly when models are exposed to hostile prompts or allowed to operate tools.
Siposova said the technology’s effect becomes especially significant “when we place it under adversarial conditions” or use models to power agents. For developers, that means watermarking needs to be included in red-team and safety testing rather than treated as a harmless final-stage provenance feature.
