# Welcome to transformers

Your guide to understanding the codebase

## Learning Path

**5 levels**  •  **15 learning units**

### Orchestration & APIs
- Pipelines, workflows, public interfaces  •  **3 units**
  - Pipeline Abstraction and Basic Usage  
  - Model Configuration and Auto Classes  
  - Tokenization and Data Processing

### Core Logic & Data
- Business rules, schemas, models  •  **3 units**
  - Transformer Architecture Fundamentals  
  - Feature Extraction and Generation  
  - Training Framework and Callbacks

### Interaction & Integration
- UI components, external connectors  •  **3 units**
  - Multimodal and Specialized Architectures  
  - Distributed Training and Model Sharding  
  - Model Hub and Integration Ecosystem

### Cross-Cutting Concerns
- Auth, logging, config, testing  •  **2 units**
  - Model Optimization and Quantization  
  - Testing, Benchmarking, and Deployment

### Edge Cases & Resilience
- Error handling, fault tolerance  •  **1 unit**

## Test Your Knowledge
24 questions across all levels

### Hands-On Assignment
**2-3 hours**

### Your Challenge
Implement a custom tokenizer that supports a new special token type called 'ENTITY' tokens for named entity recognition tasks. Your tokenizer should extend the existing PreTrainedTokenizer class, handle entity markers (e.g., <ENTITY:PERSON>, <ENTITY:ORG>) during encoding/decoding, and properly integrate with the AutoTokenizer registry so it can be loaded via AutoTokenizer.from_pretrained(). The implementation should maintain compatibility with existing tokenization pipelines while adding entity-aware vocabulary management.

### Starting Points
- `src/transformers/tokenization_utils_base.py`  
  Explore the PreTrainedTokenizerBase class to understand the core tokenization interface, special token handling, and encoding/decoding methods
- `src/transformers/tokenization_utils.py`  
  Study PreTrainedTokenizer implementation - this is the class you'll extend for your custom tokenizer
- `src/transformers/models/bert/tokenization_bert.py`  
  Reference this as a concrete example of how tokenizers are implemented, including __init__, vocab loading, and special methods
- `src/transformers/models/auto/tokenization_auto.py`  
  Understand how AutoTokenizer registry works - you'll need to register your custom tokenizer here
- `src/transformers/models/`  
  Create a new directory here for your custom tokenizer implementation (e.g., 'entity_aware/')
- `tests/models/bert/test_tokenization_bert.py`  
  Review tokenizer testing patterns - you'll create similar tests for your implementation

### Success Criteria
- Your tokenizer can encode text with entity markers like 'John <ENTITY:PERSON> works at Google <ENTITY:ORG>' and preserve entity information
- Decoding returns the original text with entity markers intact
- The tokenizer can be saved with save_pretrained() and loaded with AutoTokenizer.from_pretrained()
- Entity type vocabulary persists across save/load cycles
- At least 5 unit tests pass covering: basic tokenization, special token handling, save/load, entity type management, and edge cases
- The tokenizer works with a simple text classification pipeline without errors

### Hints
- **Hint 1:** Understanding special tokens architecture
- **Hint 2:** Key methods to override
- **Hint 3:** Registry integration pattern
- **Hint 4:** Handling dynamic entity types
- **Hint 5:** Testing your implementation

### Prerequisites
- Python class inheritance
- Tokenization concepts (vocab, encoding, decoding)
- JSON serialization
- Basic understanding of transformer model inputs