Invisible Unicode characters: zero-width, bidi and tag code points

Why invisible characters exist

Unicode has to encode every writing system, and several of them cannot be rendered correctly from letters alone. U+200D ZERO WIDTH JOINER is the best-known example: Unicode Technical Standard #51 uses it between emoji so that a sequence such as a family or a person with a profession is shown as one glyph where a font supports it, and falls back to the separate emoji where it does not. Its counterpart U+200C ZERO WIDTH NON-JOINER does the opposite, keeping letters apart, and is ordinary in Persian and in several Indic scripts. U+00AD SOFT HYPHEN marks where a word may break, U+2060 WORD JOINER says where it may not, and U+FEFF is both a byte-order mark and a zero-width no-break space.

Bidirectional text needs its own controls. Arabic and Hebrew run right to left, numbers and Latin quotations inside them run left to right, and the Unicode bidirectional algorithm resolves the mix. Where it needs help, authors insert marks and isolates such as U+200F, U+2066 and U+2069, or the older embeddings and overrides in the range U+202A to U+202E. None of these characters has a glyph. All of them are legitimate, and any tool that simply deleted them would break real text in real languages.

How the same characters are abused

Because a model tokenises what is there rather than what is drawn, invisible characters can carry a message a reader never sees. The starkest case is the Unicode Tags block, U+E0000 to U+E007F, a set of code points that mirror the printable ASCII characters and were designed for language tagging. The Unicode Standard notes that implementations which do not understand tags display any sequence of them as invisible. In January 2024 Riley Goodside showed that an instruction written entirely in tag characters was read and followed by ChatGPT; Johann Rehberger documented the technique as ASCII smuggling and showed both directions of it, hidden instructions going in and invisible text coming out in a model’s answers, which defeats any review that relies on a person reading the output.

Zero-width joiners and bidirectional marks have been used the same way in the tools developers rely on. In March 2025 Pillar Security showed that rules files read by Cursor and GitHub Copilot could hide instructions inside those characters, steering the assistant to add code while suppressing any mention of it in the chat log. GitHub added a warning for hidden Unicode in files in May 2025. The common thread is that the human review step assumed the visible text was the whole text.

Bidirectional overrides and Trojan Source

The bidirectional controls have a longer history of abuse in source code. In November 2021 Nicholas Boucher and Ross Anderson disclosed Trojan Source, later presented at the USENIX Security Symposium in 2023: overrides placed inside comments and string literals reorder how a line of code is displayed without changing how a compiler reads it, so a reviewer sees one program and the machine runs another. The technique received CVE-2021-42574, and a companion attack using look-alike characters from other scripts in identifiers received CVE-2021-42694.

The defences they proposed apply directly to documents handed to a model: compilers and interpreters should warn on unterminated bidirectional controls in comments and strings, language specifications should forbid them, and editors and repository interfaces should make the controls visible with a symbol or a warning. Making the invisible visible is the whole of the defence. The scanner on this site follows the same principle, revealing default-ignorable, zero-width and bidirectional characters as named tokens at their exact positions rather than silently removing them.

Confusables: visible characters that lie

A related trick uses characters that are visible but look like something else. Cyrillic, Greek and Latin share dozens of near-identical letters, and Unicode Technical Standard #39 defines the machinery for comparing them: a table of confusable characters and a “skeleton” transformation that maps look-alikes to a common form so two strings can be compared for visual identity. An instruction written with a handful of substituted letters reads normally to a person and may pass a naive keyword filter. Comparing skeletons instead of raw strings is how a detector avoids being fooled, and how it can also avoid falsely accusing text that merely mixes scripts for legitimate reasons.

Telling legitimate text from a hiding place

A control character is a signal to inspect, not proof of intent. Persian and Indic text will contain zero-width non-joiners; emoji sequences will contain joiners; any right-to-left passage will contain bidirectional marks, and isolates are the modern, safe form of them. What is suspicious is context: joiners between Latin letters where no emoji is involved, overrides in a document that never changes direction, tag characters anywhere at all, or a run of zero-width characters long enough to spell something. This site’s detector keeps legitimate Arabic, Persian, Indic and bidirectional text in its hard-negative suite for exactly that reason.

For text you did not write and intend to hand to a model, the practical rule is to strip invisible characters unless the script needs them, to filter the Tags block unconditionally, which is Rehberger’s recommendation to anyone building on a model, and to look at anything the scanner reveals before deciding it is harmless. What the scanner cannot do is know the language you meant to preserve, so it shows you the characters and their positions and leaves the decision where it belongs.