Project
Table of Contents
Overview
This project implements and compares sequential language models for natural language processing tasks using the Penn Treebank (PTB) dataset. It provides scripts and models to train, validate, and test language models on both word-level and character-level PTB data.
What this project does:
- Loads and preprocesses the PTB dataset.
- Builds and trains RNN, GRU, and Transformer models for language modeling.
- Evaluates model performance using perplexity and loss metrics.
- Allows experimentation with different architectures and hyperparameters.
Input:
- Pre-tokenized text files from the PTB dataset (train, validation, test splits).
- User-specified model type and training parameters (via command-line arguments).
Expected Output:
- Trained model parameters (can be saved and loaded).
- Evaluation metrics: training/validation loss and perplexity per epoch.
- Learning curves and logs for analysis.
- Optionally, generated text samples from the trained model.
Key Features
- Language Modeling: Supports training and evaluation of language models on PTB datasets.
- Data Handling: Includes scripts for loading and preprocessing PTB data (word and character level).
- Model Training: Offers configurable training routines for sequential models.
- Evaluation: Provides validation and test scripts to measure model performance.
Detailed Structure
1
2
3
4
5
6
7
8
9
10
11
| models.py # Model definitions (RNN, GRU, Transformer, utilities)
ptb-lm.py # Main script for training/evaluation
README.md # Basic project info
requirements.txt # Python dependencies
/data/ # Penn Treebank dataset files
ptb.char.test.txt
ptb.char.train.txt
ptb.char.valid.txt
ptb.test.txt
ptb.train.txt
ptb.valid.txt
|
Architectures
- RNN: Basic recurrent neural network for sequence modeling.
- GRU: Gated recurrent unit, improves on RNN by mitigating vanishing gradients.
- Transformer: Uses self-attention for parallel sequence processing, state-of-the-art for NLP.
Steps of the Project
- Data Loading: Reads PTB files, builds vocabulary, prepares batches.
- Model Setup: Instantiates chosen model (RNN, GRU, Transformer).
- Training Loop: Runs epochs, computes loss, updates parameters.
- Validation: Evaluates model on validation set, tracks best performance.
- Logging & Saving: Logs metrics, saves best model, stores learning curves.
Dataset and Preprocessing
Dataset
The project uses the Penn Treebank (PTB) dataset, a standard benchmark for language modeling. It contains pre-tokenized text files for training, validation, and testing at both word and character levels.
Preprocessing & Data Loading
- Reading Data:
1
2
3
4
| def _read_words(filename):
"""Reads words from a file, replaces newlines with <eos>."""
with open(filename, "r") as f:
return f.read().replace("\n", " <eos> ").split()
|
- Building Vocabulary:
1
2
3
4
5
6
7
8
9
| def _build_vocab(filename):
"""Builds vocabulary mapping from training data."""
data = _read_words(filename)
counter = collections.Counter(data)
count_pairs = sorted(counter.items(), key=lambda x: (-x[1], x[0]))
words, _ = list(zip(*count_pairs))
word_to_id = dict(zip(words, range(len(words))))
id_to_word = dict((v, k) for k, v in word_to_id.items())
return word_to_id, id_to_word
|
- Explanation:
_read_words loads the text and splits it into tokens, replacing newlines with <eos> (end of sentence)._build_vocab counts word frequencies, sorts them, and assigns each word a unique integer id for efficient processing.
- Purpose: Masks are used to prevent the model from attending to future tokens during training (autoregressive property).
- Code:
1
2
3
4
5
| def subsequent_mask(size):
"""Creates a mask to hide future tokens in sequence."""
attn_shape = (1, size, size)
subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')
return torch.from_numpy(subsequent_mask) == 0
|
- Explanation:
- The mask is an upper-triangular matrix that blocks attention to future positions, ensuring predictions depend only on current and previous tokens.
Loss Function and Optimizer
- Loss:
1
| loss_fn = nn.CrossEntropyLoss() # Cross-entropy loss for classification
|
- Measures the difference between predicted and true token distributions.
- Optimizer:
1
| optimizer = torch.optim.Adam(model.parameters(), lr=args.initial_lr)
|
- Updates model parameters to minimize the loss.
Helper Function: repackage_hidden
- Purpose: Detaches hidden states from their computation history to prevent backpropagation through previous batches (important for RNN/GRU).
- Code:
1
2
3
4
5
6
| def repackage_hidden(h):
"""Detaches hidden states from their history."""
if isinstance(h, Variable):
return h.detach_()
else:
return tuple(repackage_hidden(v) for v in h)
|
Model Structures and Implementation
RNN Model
1
2
3
4
5
6
7
| class RNN(nn.Module):
def __init__(self, ...):
# Initialize layers and parameters
def forward(self, inputs, hidden):
# Compute outputs and new hidden states
def init_hidden(self):
# Return initial hidden state
|
GRU Model
1
2
3
4
5
6
7
| class GRU(nn.Module):
def __init__(self, ...):
# Initialize layers and parameters
def forward(self, inputs, hidden):
# Compute outputs and new hidden states using GRU gates
def init_hidden(self):
# Return initial hidden state
|
1
2
3
4
5
6
7
8
9
| class MultiHeadedAttention(nn.Module):
def forward(self, query, key, value, mask=None):
# Compute attention scores and weighted context
class FullTransformer(nn.Module):
def __init__(self, ...):
# Stack transformer blocks and embedding
def forward(self, input_sequence, mask):
# Forward pass through transformer stack
|
Function Explanations and Relationships
models.py
1
2
3
4
5
6
7
8
| class RNN(nn.Module):
"""Implements a vanilla recurrent neural network for sequence modeling."""
def __init__(self, emb_size, hidden_size, seq_len, batch_size, vocab_size, num_layers, dp_keep_prob):
# Initializes model parameters and layers
def forward(self, inputs, hidden):
# Computes outputs and new hidden states for each time step
def init_hidden(self):
# Returns initial hidden state for each layer
|
1
2
3
4
5
6
7
8
| class GRU(nn.Module):
"""Implements a gated recurrent unit for improved sequence modeling."""
def __init__(self, ...):
# Initializes model parameters and layers
def forward(self, inputs, hidden):
# Computes outputs and new hidden states using GRU gates
def init_hidden(self):
# Returns initial hidden state for each layer
|
1
2
3
4
| class MultiHeadedAttention(nn.Module):
"""Implements multi-headed self-attention for the Transformer."""
def forward(self, query, key, value, mask=None):
# Computes attention scores and weighted context
|
Utility Functions
1
2
3
| def subsequent_mask(size):
"""Creates a mask to hide future tokens in sequence."""
# Used in Transformer for autoregressive modeling
|
ptb-lm.py
Data Loading Functions
1
2
3
4
5
6
7
8
9
10
| def _read_words(filename):
"""Reads words from a file, replaces newlines with <eos>."""
def _build_vocab(filename):
"""Builds vocabulary mapping from training data."""
def _file_to_word_ids(filename, word_to_id):
"""Converts file words to their integer ids."""
def ptb_raw_data(data_path=None, prefix="ptb"):
"""Loads train, validation, and test data, builds vocab."""
def ptb_iterator(raw_data, batch_size, num_steps):
"""Creates minibatches for training/validation."""
|
Training Functions
1
2
3
4
5
6
| def run_epoch(model, data, is_train=False, lr=1.0):
"""Runs one epoch of training or validation.
- Calls model.forward for each batch
- Computes loss and updates parameters
- Returns perplexity and loss list
"""
|
Helper Functions
1
2
| def repackage_hidden(h):
"""Detaches hidden states from their history to prevent backpropagation through previous batches."""
|
Function Relationships
- ptb-lm.py loads data using
_read_words, _build_vocab, _file_to_word_ids, and ptb_raw_data.
- Batches are created with
ptb_iterator and passed to run_epoch.
run_epoch calls the model’s forward method (RNN, GRU, Transformer) for each batch.- Loss is computed and backpropagated; optimizer updates parameters.
repackage_hidden is used to detach hidden states between batches for RNN/GRU.- Transformer uses
MultiHeadedAttention and masking utilities from models.py.
Example: Training Flow
1
2
3
4
5
6
| # In ptb-lm.py
train_data, valid_data, test_data, word_to_id, id_2_word = ptb_raw_data(...)
model = RNN(...)
for epoch in range(num_epochs):
train_ppl, train_loss = run_epoch(model, train_data, True, lr)
val_ppl, val_loss = run_epoch(model, valid_data)
|
Quick Notes
- Why use a random seed?
- Ensures reproducibility of experiments by fixing the random number generator state.
- Code:
1
| torch.manual_seed(args.seed)
|
- Why mask future tokens?
- Prevents information leakage during training, ensuring the model predicts each token using only past and current context.
Resources
Project repository
GitHub Code: Sequential Language Models (Penn Treebank)