Representation Learning for NLP - 2024/2025 course

This course provides a general overview of natural language processing. We first review seminal works regarding vector space representation of words and deep neural network. We then study the foundations of modern NLP, i.e. principles of large language models and how to adapt them to diverse tasks.

Pre-LLM Era

Learning vector space representations of words

• Skip-Gram with Negative Sampling: Distributed Representations of Words and Phrases and their Compositionality
• Glove: Global Vectors for Word Representation
• Link between Skip-Gram with Negative Sampling, GloVe & PMI: Neural Word Embedding as Implicit Matrix Factorization

Leveraging pre-trained word representations for supervised learning

Basic architectures for text classification

• Basic Convolutional Architecture: Convolutional Neural Networks for Sentence Classification
• Gated Recurrent Unit: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Advanced architecture for text generation

• Encoder-Decoder Architecture: Sequence to Sequence Learning with Neural Networks
• Attention Mechanism: Effective Approaches to Attention-based Neural Machine Translation

LLM Era

Foundation models

• Attention-based Encoder-Decoder (Transformer): Attention is All You Need
• Pre-trained Encoder (BERT): Bidirectional Encoder Representations from Transformers
• Pre-trained Decoder (GPT): Language Models are Unsupervised Multitask Learners
• Pre-trained Encoder-Decoder (T5): Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Adapting foundation models

• Reinforcement Learning from Human Feedback (RLHF): Training language models to follow instructions with human feedback
• Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model
• Retrieval-Augmented Generation (RAG): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Low-Rank Adaptation (LoRA): Low-Rank Adaptation of Large Language Models