Phase 8 · weeks 54-61 · 8 weeks
NLP, Transformers & Building LLMs (Core)
- NLP fundamentals: tokenization, TF-IDF, embeddings (Word2Vec, GloVe)
- Attention mechanism derivation; seq2seq
- Transformer architecture from scratch: self-attention, multi-head attention, positional encoding, encoder-decoder, layer norm, residuals
- Tokenizers in depth: BPE, WordPiece, SentencePiece — build a custom tokenizer
- Build a GPT-style decoder-only LLM from scratch in PyTorch: embeddings → transformer blocks → output head, causal masking, training loop, cross-entropy loss, perplexity
- Train a small LLM end-to-end on a custom dataset
- BERT / encoder-only models (masked language modeling)
- Fine-tuning with HuggingFace Transformers: classification, NER, summarization, Q&A
- LoRA / QLoRA parameter-efficient fine-tuning
- RAG: embeddings, vector DBs (FAISS/Chroma/Pinecone), building a RAG pipeline
- Prompt engineering, chain-of-thought, AI agents/tool use (LangChain/LlamaIndex)
- Evaluating LLMs: perplexity, BLEU/ROUGE, benchmarks
Project: Small GPT trained from scratch on a custom corpus; LoRA fine-tune of an open model (Llama/Mistral/Qwen family)