Repository navigation

BPE_Tokenization

Public

About

Byte Pair Encoding (BPE) Tokenizer from Scratch | Custom GPT-style tokenizer with regex, comparison with tiktoken (GPT-4)

1 star

1 watching

Report repository

Name		Name	Last commit message	Last commit date
Models		Models
Test		Test
assets		assets
.gitignore		.gitignore
License		License
README.md		README.md
requirements.txt		requirements.txt
train.py		train.py

Repository files navigation

🧠 BPE Tokenizer from Scratch

About

Byte Pair Encoding (BPE) Tokenizer from Scratch | Custom GPT-style tokenizer with regex, comparison with tiktoken (GPT-4)

1 star

1 watching

Report repository

Releases

Packages

Used by

Contributors

Languages