Distributed and Multi-Node Training in Sequifier

Sequifier natively supports multi-GPU and multi-node training using PyTorch’s DistributedDataParallel (DDP) and FullyShardedDataParallel (FSDP).

1. Prerequisites: Preprocessing for Distributed Training

To use distributed training, your data must be sharded into multiple files so that different GPUs can read different chunks simultaneously without memory bottlenecks.

In your preprocess.yaml, you must write sharded output:

merge_output: false

For production multi-GPU training, use PyTorch tensor shards:

write_format: pt

Note: Distributed training is not supported if your data is kept as a single csv or parquet file. You must use merge_output: false to generate a folder of sharded files.

Beta Notice for Parquet in Distributed Training: While write_format: parquet is supported for distributed training, it is currently considered Beta. Because Parquet chunk reading relies on Polars’ multi-threading, using it alongside PyTorch’s multiprocess DataLoader in heavy multi-GPU environments can lead to CPU thread contention, high RAM usage, or NCCL timeouts. Recommendation: For production multi-GPU runs, use write_format: pt. It relies on native PyTorch serialization and is significantly more stable under heavy hardware loads.

2. Configuration: train.yaml

Once your data is preprocessed into .pt shards, or beta .parquet shards, you need to tell the Sequifier training engine to expect a distributed environment.

In your train.yaml, configure the canonical global_training block:

global_training:
  read_format: pt             # or parquet for beta sharded Parquet loading
  distributed: true
  data_parallelism: fsdp # or ddp
  fsdp_cpu_offload: false   # omit if using ddp; set true to offload FSDP parameters to CPU RAM
  layer_type_dtypes: null    # required for FSDP; use layer_autocast for mixed precision
  torch_compile: inner       # use inner or none for FSDP; use outer or none for DDP
  world_size: 32       # The TOTAL number of GPUs across all nodes (e.g., 8 nodes * 4 GPUs = 32)
  backend: nccl        # 'nccl' is the standard and most efficient backend for NVIDIA GPUs

When shards do not divide evenly across ranks, Sequifier automatically pads shorter ranks with repeated samples for step alignment. Those repeats are masked out of loss calculation, so each real sample contributes once.

3. Launching the Training Job

How you launch the training depends on whether you are using a single machine with multiple GPUs, or multiple machines (nodes) connected over a network.

Scenario A: Single-Node, Multi-GPU

If you are running on a single machine that has multiple GPUs (e.g., an AWS EC2 instance with 4x A100s), Sequifier can handle process generation internally using torch.multiprocessing.spawn.

You simply run the standard command:

sequifier train --config-path configs/train.yaml

Sequifier will read the world_size config parameter and automatically spawn that exact number of worker processes.

Scenario B: Multi-Node, Multi-GPU (HPC / Slurm)

Sequifier cannot automatically spawn Python processes across physical network boundaries. For multi-node training, you must use an external cluster manager (like Slurm) combined with PyTorch’s torchrun utility.

When sequifier detects torchrun environment variables (like RANK and WORLD_SIZE), it bypasses its internal spawner and attaches to the distributed network established by the cluster. In that mode, the environment WORLD_SIZE is used.

Here is a standard sbatch script template for launching Sequifier across multiple nodes:

#!/bin/bash
#[SBATCH COMMANDS]

MASTER_NODE=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)

srun torchrun \
    [-- torchrun args]...
    $(which sequifier) train --config-path=configs/train.yaml

Important Considerations for Multi-Node

  • Batch Size: The batch_size in your train.yaml is the per-process batch size. If batch_size is 100 and world_size is 32, each synchronized backward pass covers 3,200 samples. With gradient accumulation, the samples per optimizer update are batch_size * world_size * accumulation_steps (apart from a final partial accumulation window).

  • Learning Rate: You may need to scale your learning_rate up if you drastically increase your global batch size via distributed training.

  • Data Access: All nodes must have access to the same shared filesystem (e.g., NFS, GPFS) where the project_root and the sharded preprocessing output are stored.