Teaching

Representation Learning for NLP (Master Machine Learning & AI @ U Lyon 2)

Here is a list of suggested readings for this course:

Pre-LLM Era

Learning vector space representations of words

• SGNS: Distributed Representations of Words and Phrases and their Compositionality
• Glove: Global Vectors for Word Representation
• Link between SGNS, GloVe & PMI: Neural Word Embedding as Implicit Matrix Factorization

Leveraging pre-trained word representations for supervised learning

Basic architectures for text classification

• CNN: Convolutional Neural Networks for Sentence Classification
• RNN & GRU: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Advanced architecture for text generation

• Seq2Seq: Sequence to Sequence Learning with Neural Networks
• Attention: Effective Approaches to Attention-based Neural Machine Translation

LLM Era

Foundation models

• Transformer: Attention is All You Need
• BERT: Bidirectional Encoder Representations from Transformers
• GPT: Language Models are Unsupervised Multitask Learners
• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Adapting foundation models

• RLHF: Training language models to follow instructions with human feedback
• DPO : Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• LoRA: LoRA: Low-Rank Adaptation of Large Language Models

Machine Learning with Graphs (Master Complex Systems @ ENS Lyon)

Here is a list of suggested readings for this course:

Pre-trained Node and Graph Representations

• DeepWalk: Online Learning of Social Representations
• Node2Vec: Scalable Feature Learning for Networks

Graph Neural Networks

• GCN: Semi Supervised Node Classification with Graph Convolutional Networks
• GAT: Graph Attention Networks