COMP3361: Natural Language Processing

Course Information

Instructor

Tao Yu
Office Hour: Wednesday 4 - 5pm @IDS P307H

Lecture

Tuesday 9:30 am - 10:20 am and Friday 9:30 am - 11:20 am @KB132
Course syllabus

Yiheng Xu (Office Hour: Thursday 9 am - 10:15 am. Book here)

Course Description

Natural language processing (NLP) is the study of human language from a computational perspective. Over the past 20 years, the field of NLP has evolved significantly, primarily driven by advancements in statistical machine learning and deep learning. A notable recent breakthrough is the development of “pre-trained” language models, such as ChatGPT, which have substantially enhanced capabilities across various applications. This course is an introductory undergraduate-level course on natural language processing. In this class, we will cover recent developments on core techniques and modern advances in NLP, especially in the era of large language models. Students will gain the necessary skills and experience to understand, design, implement, and test large language models through lectures and assignments. We will potentially also host invited speakers for talks. 

Prerequisites

COMP3314 or COMP3340; and MATH1853
We require students to have prior knowledge undergraduate linear algebra, probability and statistics, machine learning, or deep learning. Familiarity with Python programming is required.

Course Materials

There is no required textbook for this course. Speech and Language Processing (3rd ed. draft) by Dan Jurafsky and James H. Martin (2023) and Natural Language Processing by Jacob Eisenstein are recommended if you would like to read more about NLP. Readings from papers, blogs, tutorials, and book chapters will be posted on the course website. Textbook readings are assigned to complement the material discussed in lecture. You may find it useful to do these readings before lecture as preparation or after lecture to review, but you are not expected to know everything discussed in the textbook if it isn’t covered in lecture. Paper readings are intended to supplement the course material if you are interested in diving deeper on particular topics.

Grading

Assignments: 40%
- Assignment 1 (A1): 20%
- Assignment 2 (A2): 20%
Course project: 30%
In-class final exam: 25% (May 8, Wed, 9:30 am - 11:30 am @Rm 3 Library Ext.)
Class participation: 5%
- Filling out course feedback surveys
- Participating in group discussions on Slack or in the class
- Attending potential guest speakers’ lectures, and so on

Course Schedule

Date	Topic	Material	Event	Due
Week 1 Jan 16	Introduction [slides]	Readings Human Language Understanding & Reasoning
Jan 19	Language modeling (n-gram language models) [slides]	Readings M. Collins, Notes 1 J&M Ch. 7
Week 2 Jan 23	Text classification [slides]	Readings J&M Ch. 4.1-4.6 J&M Ch. 5.1-5.8 M. Collins, Notes 2 J&M Ch. 6 Others Working with Google Colab	A1 out
Jan 26	Word embeddings 1 [slides]	Readings J & M 6.2-6.4, 6.6 word2vec paper GloVe paper
Week 3 Jan 30	Word embeddings 2 [slides]	Readings J & M 6.8, 6.10-6.12
Feb 2	Neural language models: Overview, tokenization [slides]	Readings J & M 7.3-7.5 Running example of using BERT Byte-pair encoding paper Others Talk on the GPT Tokenizer by Andrej Karpathy Summary of the tokenizers with HuggingFace
Week 4 Feb 6	Neural language models: RNNs [slides]	Readings J & M 7.3-7.5
Feb 9	Tutorial on PyTorch and Transformers [slides]	Other resources Stanford CS224N PyTorch Tutorial HuggingFace NLP Course Transformers Notebooks
Week 5 Feb 13	No class
Feb 16	No class
Week 6 Feb 20	Neural language models: RNNs and LSTM [slides]	Readings J & M 9.1-9.3, 9.5 The Unreasonable Effectiveness of Recurrent Neural Networks Understanding LSTM Networks	A2 out	A1 due
Feb 23	Neural language models: Transformers [slides]	Readings J & M 10.1 Attention Is All You Need The Annotated Transformer The Illustrated Transformer
Week 7 Feb 27	Neural language models: Pretraining 1 [slides]	Readings BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding The Illustrated BERT, ELMo, and co.
Mar 1	Neural language models: Pretraining 2 [slides]	Readings Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) Language Models are Few-Shot Learners (GPT-3) Llama 2: Open Foundation and Fine-Tuned Chat Models Others Colab on building GPT: from scratch, in code, spelled out by Andrej Karpathy The Illustrated GPT-2
Week 8 Mar 5	No class
Mar 8	No class
Week 9 Mar 12	LLM prompting, in-context learning, scaling laws, emergent capacities 1 [slides]	Readings Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Others State of GPT by Andrej Karpathy
Mar 15	LLM prompting, in-context learning, scaling laws, emergent capacities 2 [slides]	Readings Scaling Laws for Neural Language Models Emergent Abilities of Large Language Models Others Are Emergent Abilities of Large Language Models a Mirage?
Week 10 Mar 19	Coding tutorial [slides]	Readings The Annotated Transformer Talk on building GPT: from scratch, in code, spelled out by Andrej Karpathy
Mar 22	Natural language generation with LLMs 1 [slides]	Others The Curious Case of Neural Text Degeneration		A2 due
Week 11 Mar 26	Natural language generation with LLMs 2 [slides]		A3 (originally the course project) out
Mar 29	No class
Week 12 Apr 2	Intro to advanced topics [slides]
Apr 5	Code language models (by Ansong Ni, Yale) [slides] Retrieval-augmented LMs (by Weijia Shi, UW) [slides]	Readings One Embedder, Any Task: Instruction-Finetuned Text Embeddings REPLUG: Retrieval-Augmented Black-Box Language Models Evaluating Large Language Models Trained on Code Code Llama: Open Foundation Models for Code Others Tutorial: Retrieval-based Language Models and Applications
Week 13 Apr 9	LLMs/VLMs as agents [slides]	Readings LLM Powered Autonomous Agents OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Apr 12	Solving Real-World Tasks with AI Agents (by Shuyan Zhou, CMU) [slides] Instruction tuning for LLMs (by Yizhong Wang, UW) [slides]	Readings Finetuned Language Models Are Zero-Shot Learners Self-Instruct: Aligning Language Model with Self-Generated Instructions Training language models to follow instructions with human feedback PAL: Program-aided Language Models DocPrompting: Generating Code by Retrieving the Docs Others WebArena: A Realistic Web Environment for Building Autonomous Agents
Week 14 Apr 16	Efficient LM methods (by Bailin Wang, MIT) [slides]	Readings LoRA: Low-Rank Adaptation of Large Language Models (A) FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (B)
Apr 19	Principles of Reasoning: Designing Compositional and Collaborative Generative AIs (by William Wang, UCSB, on Apr 18) LLM alignment (by Ruiqi Zhong, UC Berkeley) [slides]	Readings Measuring Progress on Scalable Oversight for Large Language Models Self-critiquing models for assisting human evaluators Recursively Summarizing Books with Human Feedback Others Explaining AI Alignment as an NLPer and Why I am Working on It Core Views on AI Safety: When, Why, What, and How
Week 15 Apr 23	Multimodal language models/VLMs (by Yushi Hu, UW) [slides]	Readings CLIP: Connecting text and images A Dive into Vision-Language Models
Apr 26	Robotics in the era of LLM/VLMs (by Ted Xiao, Google DeepMind) [slides]	Readings Do As I Can, Not As I Say: Grounding Language in Robotic Affordances RT-1: Robotics Transformer for Real-World Control at Scale Others RT-2: Vision-Language-Action Models Robot Learning in the Era of Foundation Models: A Survey
Week 16 Apr 30	No class (Revision)			Course project due