HackerTube

THE VIDEO SHELF · llm tokenization

Let's build the GPT Tokenizer

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Watch & explore the libraryYouTube ↗

Why HN readers recommended it

Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.

This is a summary from the source dataset, not an endorsement by Hacker News or the video creator.

Video & recommendation details

Creator / channel
Andrej Karpathy
Topic
llm tokenization
Format
lecture
Level
intermediate
HN recommenders in dataset
5
HN mentions in dataset
8
Recommendation years in dataset
2024–2025

These counts and years describe the included Hacker News records; they are not YouTube views or the video’s publication date.

From the original HN conversations

Excerpts from the recorded comments. Follow each source for the full comment and surrounding discussion.

  • I finally understood the weirdness of tokenizers after watching the video Andrej Karpathy made: "Let's build the GPT Tokenizer" ( [VIDEO] ). He goes through why we need them instead of raw byte sequences (too expensive) and how the Byte Pair Encoding algorithm works. Worth …

    cranium on Hacker News · 2024-10
    In Probably pay attention to tokenizers

  • ChatGPT is specifically bad at these kinds of tasks because of tokenization. If you plug your query into [LINK] , you can see that"egregious" is a single token, so the LLM doesn't actually see any "e" characters -- to answer your question it would have …

    moozilla on Hacker News · 2024-03
    In Ask HN: Did you encounter any leap year bugs today?

  • This is just an artefact of tokenisation though. The model simply isn’t ever shown the letters that make up words, unless they are spelled out explicitly. It sees tokens representing groups of words. This is a little like saying a human isn’t intelligent because they …

    Chirono on Hacker News · 2024-08
    In Artificial intelligence is losing hype

Summaries, classifications and recommendation records are from the HackerTube source dataset. Read about this library and its limitations.

Keep exploring