HackerTube

THE VIDEO SHELF · llm tokenization

Glitch Tokens - Computerphile

Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.

Watch & explore the libraryYouTube ↗

Why HN readers recommended it

HN points to it as the accessible explanation of why glitch tokens exist whenever tokenizers or odd GPT behavior come up.

This is a summary from the source dataset, not an endorsement by Hacker News or the video creator.

Video & recommendation details

Creator / channel
Computerphile
Topic
llm tokenization
Format
explainer
Level
intro
HN recommenders in dataset
2
HN mentions in dataset
7
Recommendation years in dataset
2023–2023

These counts and years describe the included Hacker News records; they are not YouTube views or the video’s publication date.

From the original HN conversations

Excerpts from the recorded comments. Follow each source for the full comment and surrounding discussion.

  • I found this tool recently when it was linked from this Computerphile video[1] about "glitch tokens". tldw: Certain junk data was thrown out post-tokenization e.g. the /r/counting[2] community data and debug logs from Rocket League some tokens specific to those contexts stuck around, however, and …

    non- on Hacker News · 2023-04
    In OpenAI Tokenizer

  • Computerphile has an interesting video about Glitch Tokens, but as it goes with Computerphile videos it's also a good introduction to the topic. [VIDEO]

    weinzierl on Hacker News · 2023-06
    In Understanding GPT Tokenizers

Summaries, classifications and recommendation records are from the HackerTube source dataset. Read about this library and its limitations.

Keep exploring