tiktoken — Easy Playground

Fast BPE tokenizer used by OpenAI models

Getting started with tiktokenRun locally

Install

pip install tiktoken

Python CodeRun locally

# Install: pip install tiktoken
import tiktoken

# Basic tiktoken usage
print(f"Using tiktoken")
# See documentation for detailed examples

# Install: pip install tiktoken
import tiktoken

# Basic tiktoken usage
print(f"Using tiktoken")
# See documentation for detailed examples

Expected Output

# Expected output shown below
# (Run locally with: tiktoken)

tiktoken is a third-party package. Fast BPE tokenizer used by OpenAI models. Install with: pip install tiktoken

Try modifying the code above to explore different behaviors. Can you extend the example to handle a new use case?