Train Command Guide

sequifier train trains one shared transformer backbone through one or more named model interfaces. An interface is an ingestion module, its generated adapter, and a decoder. Datasets own data and training policy; model owns architecture.

sequifier train --config-path configs/train.yaml

Singleton configuration

When a run has one model interface, one dataset, one part, and one phase, those values can be authored directly without routing names or references:

project_root: .
model_name: event-model
device: cuda
seed: 1010

global_training:
  read_format: parquet
  training_objective: causal
  context_length: 128
  inference_batch_size: 256
  batch_size: 64
  learning_rate: 0.0001

model:
  backbone:
    architecture:
      dim_model: 128
      max_context_length: 512
      num_layers: 6
      attention: {type: mha, n_heads: 8, n_kv_heads: 8, output_projection: true}
      feed_forward: {dim: 512, activation: swiglu}
      normalization: {type: rmsnorm, norm_first: true}
      position_encoding: {type: rope, theta: 10000}
      dropout: 0.1
      shared_layer_groups: []
  interface:
    input_columns: [event]
    target_columns: [event]
    ingestion: {type: embedding, output_dim: 128}
    decoder: {type: linear, prediction_length: 1, support: 1}

dataset:
  part: {metadata_config_path: configs/metadata_configs/events.json}
  criterion: {event: CrossEntropyLoss}

training_plan:
  epochs: 5

evaluation: true

The concise form expands before validation:

Authored field	Canonical form
model.interface	model.interfaces.default
dataset.part	dataset_training.default.parts.default
training_plan.epochs	One sequential phase named train
evaluation: true	Evaluate the inferred single source

These shortcuts can be used independently. A unique interface and training source are inferred; multiple interfaces or datasets require explicit names and references. Singular and plural spellings cannot be combined. Because errors are reported after expansion, they may use the canonical paths above.

Canonical configuration

project_root: .
model_name: event-model
device: cuda
seed: 1010

global_training:
  read_format: parquet
  training_objective: causal
  context_length: 128
  target_offset: 1
  window_stride: 1
  inference_batch_size: 256
  batch_size: 64
  accumulation_steps: 4
  learning_rate: 0.0001
  optimizer: {name: AdamW, weight_decay: 0.01}
  scheduler: {name: StepLR, step_size: 1, gamma: 0.99}
  scheduler_step_on: epoch
  reset_optimization_on_phase: true
  gradient_clip: 1.0
  save_interval_epochs: 1

model:
  backbone:
    architecture:
      dim_model: 128
      max_context_length: 512
      num_layers: 6
      attention: {type: mha, n_heads: 8, n_kv_heads: 8, output_projection: true}
      feed_forward: {dim: 512, activation: swiglu}
      normalization: {type: rmsnorm, norm_first: true}
      position_encoding: {type: rope, theta: 10000}
      dropout: 0.1
      shared_layer_groups: []
  interfaces:
    event_prediction:
      input_columns: [event]
      target_columns: [event]
      categorical_decoder_special_tokens: {event: [other]}
      ingestion: {type: embedding, output_dim: 128}
      decoder: {type: linear, prediction_length: 1, support: 1}

dataset_training:
  events:
    model_interface: event_prediction
    parts:
      original: {metadata_config_path: configs/metadata_configs/events.json}
      increment: {metadata_config_path: configs/metadata_configs/events-increment.json}
    criterion: {event: CrossEntropyLoss}
    loss_weights: {event: 1.0}
    freeze:
      backbone: {freeze: [attention.qkv]}

training_plan:
  phases:
    - name: incremental_finetuning
      epochs: 2
      mode: sequential
      sources: [{source: events.increment}]
    - name: complete_retraining
      epochs: 5
      mode: interleaved
      selection: round_robin
      sources: [{source: events, batches_per_selection: 4}]

evaluation:
  sources: [{source: events}]

export_generative_model: true
export_embedding_model: false
export_onnx: true
export_pt: false
export_with_dropout: false

The historical flat training schema is not accepted. In particular, training_spec, top-level dataset paths/columns, model.ingestion, model.decoder, and architecture-owned freezing are not canonical fields.

Ownership and resolution

• global_training owns objective, window, optimizer, precision, distribution, compilation, checkpoint, and data-loader behavior. Phase entries own epochs.

• model contains exactly one backbone and one or more named interfaces. Different interface names create distinct ingestion and decoder weights; repeated references to one name share those weights.

• dataset_training owns parts, criterion/weights, class-share logging, freezing, and the interface reference.

• Preprocessing metadata owns split paths, data types, class counts, ID maps, special-token IDs, normalization facts, and stored-window layout.

Every part of a dataset must resolve to the same schema, categorical semantics, normalization contract, storage layout, and file/folder storage form. A source named events iterates all parts in declaration order; events.increment iterates only that part. Only parts selected by evaluation.sources require a validation split.

Categorical multi-hash embeddings and targets

Add categorical_hashing to a model interface, keyed by categorical column name. The setting applies to an input column’s embedding and, if the column is also a target, to its prediction heads. For example, add these fields to a training config with a backbone and dataset part:

model:
  interfaces:
    default:
      input_columns: [accountId, merchantId]
      target_columns: [accountId]
      categorical_hashing:
        accountId:
          type: multi_hash
          num_buckets: 50000
          num_hashes: 4
          seed: 1010
        merchantId:
          type: qr
          num_buckets: 1000
      ingestion:
        type: embedding
        output_dim: 128
      decoder:
        type: linear
        prediction_length: 1
        support: 1
dataset_training:
  default:
    model_interface: default
    criterion:
      accountId: CrossEntropyLoss
    loss_weights:
      accountId: 1.0

Here accountId uses both hashed input embeddings and hashed target heads; merchantId uses hashed input embeddings only. multi_hash creates num_hashes independently seeded tables of num_buckets rows and adds their outputs. num_hashes must be positive; seed defaults to 0. qr uses the quotient and remainder of the category ID as two table indices and multiplies their embeddings. Its hash count is always two. Do not add hashing under ingestion: the embedding ingestion reads the interface setting, including when used by temporal_conv with base_ingestion: embedding.

When feature_embedding_dims is omitted for a categorical-only embedding ingestion, each column receives at least two dimensions. The remaining output_dim positions are split in proportion to the logarithm of each column’s cardinality, rounding down first and assigning leftover positions by largest fractional remainder (ties follow column order). For multi_hash, the effective cardinality is the smaller of the original cardinality and num_buckets per table. Other categorical columns use their original cardinality. The output width must be at least twice the number of categorical columns. Explicit feature_embedding_dims still controls each column’s width directly.

A hashed target requires CrossEntropyLoss. Its loss weight is divided equally across the hash heads; class_weights cannot be used for that target. The stored data still contains the original categorical IDs, without extra hash columns. Hash codes for classes eligible for prediction must be unique; config resolution reports a collision if the chosen bucket count and hash count do not distinguish them.

In an autoregressive_transformer decoder branch, a hashed target must be the last target in that branch’s target_columns order. Earlier targets use a full-size feedback embedding inside the decoder.

Folder dataset parts accept file_order: shuffled (the default) or file_order: name. For curriculum training, these respectively reshuffle file blocks each epoch or keep them in lexicographic path order; curriculum order is local to each file. Without curriculum training, eager shuffled loading keeps the ordinary global sample shuffle. The setting has no effect on single-file parts. training_plan also accepts curriculum_training: false (the default). When set to true, curriculum_column must name one column available in the preprocessed data. Training visits lower values first inside each physical file and reshuffles equal-valued samples each epoch. Curriculum ordering applies only to training loaders for parts selected by the training plan; validation keeps its ordinary unshuffled order. Ordering is intentionally not synchronized across files, loader workers, or distributed ranks. curriculum_column defaults to null and cannot be set when curriculum training is disabled. When disabled, curriculum values are ignored and the ordinary sample shuffle is preserved.

dataset:
  part:
    metadata_config_path: configs/metadata_configs/events.json
    file_order: name
training_plan:
  curriculum_training: true
  curriculum_column: difficulty
  epochs: 5

loss_weights scales each target’s contribution to the training and reported aggregate loss. A weight of 0.0 disables that target’s backward-loss component while retaining its output and per-target accounting. At least one target in each dataset must have a positive weight.

Optimizer plans

The single-optimizer form above continues to apply one optimizer to all parameters and uses global_training.learning_rate. To select parameters or use multiple optimizers, replace it with an ordered plan. Each group has its own learning rate; omit the run-wide learning_rate in this form:

global_training:
  optimizer:
    groups:
      - id: hidden_matrices
        select:
          semantic_groups: ["attention.*", "feed_forward.*"]
          component: backbone
          parameter_kind: weight
          ndim: 2
        optimizer: {name: Muon, momentum: 0.95}
        learning_rate: 0.02
      - id: remainder
        select: otherwise
        optimizer: {name: AdamW, weight_decay: 0.01}
        learning_rate: 0.0003

Each supplied selector condition must match. semantic_groups matches the parameter catalog’s semantic names using shell-style * patterns; a parameter matches the list if it matches any pattern. Other available conditions are component (ingestion, backbone, or decoder), parameter_kind (weight, bias, or other), and ndim. At least one condition is required. The final group must be otherwise, which receives every parameter not selected above. Routes must be disjoint, all groups must receive parameters, and each trainable parameter is assigned once. Group IDs are available to integration directives.

The plan also supports different settings for one optimizer, for example two AdamW groups with different weight decay. The named optimizer must be available in Sequifier’s optimizer registry. Muon availability depends on the installed PyTorch version. The configured scheduler applies to every group; training metrics report the first group’s learning rate. With OneCycleLR, each group’s learning_rate is its peak rate. Omit scheduler.max_lr in plan mode; Sequifier supplies the group rates in plan order. OneCycleLR derives the starting rates from those peaks and div_factor.

With CyclicLR, each group’s learning_rate is also its peak rate. Add a base_learning_rate below that peak to every optimizer group, and omit scheduler.base_lr and scheduler.max_lr. For the plan above, the two base rates could be 0.002 and 0.00003, respectively. Configure the scheduler with name: CyclicLR, step_size_up: 2000, and scheduler_step_on: batch. Momentum cycling defaults to off for both cycle schedulers in plan mode; explicit cycle_momentum: true is unsupported. base_learning_rate is only valid with CyclicLR.

Optimization across phases

By default, every training phase starts with a new optimizer, scheduler, and gradient scaler. Model weights and global progress counters continue across phase boundaries, while optimizer moments and scheduler position do not. Set global_training.reset_optimization_on_phase: false to keep the optimizer, scheduler, and gradient scaler operating continuously across all phases. A checkpoint restores the active optimization lifecycle exactly in either mode.

For an epoch-stepped scheduler in reset mode, do not provide global_training.scheduler.total_steps because Sequifier owns each phase’s scheduler duration. When a scheduler such as OneCycleLR accepts total_steps, Sequifier sets it to the active phase’s epochs in reset mode, or the sum of all phase epochs in continuous mode. A continuous epoch-stepped scheduler may still provide total_steps for compatibility, but it must equal the sum of all phase epochs. Other scheduler arguments remain shared across phases. This example uses a single optimizer:

global_training:
  scheduler:
    name: OneCycleLR
    max_lr: 0.003
    pct_start: 0.2
    cycle_momentum: false
  scheduler_step_on: epoch
  reset_optimization_on_phase: false

Batch-stepped schedulers may configure total_steps explicitly because phase update counts can depend on source interleaving, dataset transitions, and gradient accumulation. In reset mode, that value applies independently to each phase; configuration emits a warning to make that scope explicit. In continuous mode, it applies once across the full run.

Training plans

A sequential phase exhausts each source in listed order. An interleaved phase uses round_robin or weighted_random. Each selection consumes at most batches_per_selection; every source is still exhausted once per phase epoch. Weights affect order and burst frequency, not the amount of data consumed.

Gradient accumulation may cross part or source boundaries only while the dataset stays the same. A dataset transition flushes a partial window using its actual microbatch count. Dataset-specific frozen gradients are removed before the optimizer step, preventing momentum and weight decay from changing frozen parameters.

With several evaluation sources, configure an explicit monitor when validation-based saving or early stopping is enabled:

evaluation:
  sources: [{source: events}, {source: telemetry.main}]
  monitor: {source: events, metric: loss, mode: min}

Checkpoint interval fields retain distinct behavior. save_interval_epochs, save_interval_batches, and save_interval_minutes create persistent epoch/batch snapshots and refresh the rolling latest checkpoint; save_latest_interval_minutes refreshes only latest. save_interval_val_loss controls whether timed or batch snapshots also run validation and record its monitored loss. Every checkpoint is written after flushing any partial gradient-accumulation window.

Composable YAML files

An entry file may declare complementary fragments with additional_config_paths. Relative paths resolve against the entry file’s project_root. Fragments can contribute disjoint children under containers such as global_training, model.interfaces, and dataset_training; duplicate fields are rejected. CLI overrides are applied after composition and before metadata resolution.

Singleton fragments may likewise contribute disjoint fields under model.interface or dataset. Normalization occurs after fragment composition, so all fragments in one training config must consistently use the singular or named spelling at each level.

The training command accepts --model-name, --seed, and --skip-metadata as configuration overrides. Dataset paths, columns, metadata paths, and device selection must use their canonical YAML locations.

Artifacts

ONNX is exported by default; PT inference bundles are opt-in. ONNX favors a portable deployment runtime, while PT embeds its execution contract and retains PyTorch behavior. See the “ONNX or PT?” section of the inference guide for the trade-offs.

Single-dataset filenames use <model>, while multi-dataset logs, metrics, and ONNX files use <model>-<dataset>. Part names are metric-row fields, not filename components. PT inference bundles and exact-resume checkpoints remain run-wide. Generated filenames do not use a sequifier- prefix.

The PT inference bundle contains artifact_type, format_version, model_state_dict, an execution-only model_config, and export metadata (trace-site names and provenance). Optimizers, paths, parts, training plans, evaluation policy, and dataset bindings remain outside that bundle. export_with_dropout affects ONNX export only: enabling it exports the ONNX graph in training mode and disables constant folding so dropout remains active.

Depth encoders and nested composites

Use global_training.read_format: pt for preprocessed depth inputs. Layout capacities come from dataset metadata. The following singleton model fragment combines one shallow branch with a depth encoder; its metadata comes from the preprocessing example in preprocess.md.

At every outer time step, a depth_transformer branch embeds the valid child slots from its named layout, applies a masked transformer across the child dimension, and uses the CLS output as that collection’s single temporal token. The shared backbone then models relationships across outer time steps.

model:
  backbone:
    initialization_seed: 20260910
    architecture:
      dim_model: 128
      max_context_length: 128
      num_layers: 4
      attention: {type: mha, n_heads: 8}
      feed_forward: {dim: 512, activation: swiglu}
      position_encoding: {type: rope}
  interface:
    input_columns: [accountType, subitemType, subitemAmount]
    target_columns: [nextAction]
    ingestion:
      type: composite
      branches:
        item:
          type: embedding
          columns: [accountType]
          output_dim: 32
        subitems:
          type: depth_transformer
          layout: subitems
          columns: [subitemType, subitemAmount]
          feature_embedding_dims: {subitemType: 48, subitemAmount: 16}
          output_dim: 96
          dropout: 0.1
          initialization_seed: 12345
          initialization:
            attention.qkv:
              weight: {method: xavier_uniform, gain: 1.0}
            free_parameter:
              weight: {method: normal, mean: 0.0, std: 0.02}
          architecture:
            dim_model: 64
            num_layers: 2
            attention: {type: gqa, n_heads: 4, n_kv_heads: 2}
            feed_forward: {dim: 256, activation: swiglu}
            normalization: {type: rmsnorm, norm_first: true}
            position_encoding: {type: learned}
            dropout: 0.1
      merge: {type: concat}
    decoder: {type: linear, prediction_length: 1}

Depth encoders support learned, sinusoidal, or rotary positions; MHA, MQA, or GQA; LayerNorm or RMSNorm; the shared feed-forward activations and layer-sharing configuration. architecture.dropout controls depth position and transformer sites. The ingestion-level dropout controls the pooled output. Mixed categorical/real features require explicit feature widths; homogeneous features can divide architecture.dim_model using the ordinary ingestion width rules.

For categorical-only depth features, automatic allocation gives each column at least two dimensions, so architecture.dim_model must be at least twice the number of columns. To use smaller shares, set positive widths for every column in feature_embedding_dims. For example, with two categorical columns, feature_embedding_dims: {first: 1, second: 1} permits one dimension each. Explicit widths may sum to a value different from architecture.dim_model; the depth input projection handles the difference. The pooled projection handles differences between architecture.dim_model and output_dim.

CLS occupies position zero, and physical slot s occupies position s+1. Empty collections have a learnable CLS-only representation. Deep targets, deep BERT objectives, and deep autoregressive inference are excluded.

A composite branch may itself be a composite. Every nested composite requires output_dim; an omitted root composite width retains the existing backbone input width. Child widths resolve recursively before merge projections are built. allow_shared_columns applies to the immediate children of the node where it appears, including overlaps between their descendant features. allow_unused_input_columns and auxiliary_input_columns belong at the root; non-default child policies are rejected. Temporal per-feature positions occur at leaf outputs only, and global temporal positions occur in the backbone. Composite merges do not add another temporal position stage.

Ordered autoregressive target decoding

Use type: autoregressive_transformer when outputs at the same temporal position depend on one another. target_columns is an ordered contract: each target is conditioned on the backbone state and all targets before it. Training and validation use parallel teacher forcing; inference generates that prefix greedily inside the decoder. Categorical and real targets can be mixed.

decoder:
  type: autoregressive_transformer
  prediction_length: 1
  support: 1
  target_columns: [patch_length, byte_0, byte_1, magnitude]
  architecture:
    dim_model: 256
    num_layers: 2
    attention: {type: mha, n_heads: 4}
    feed_forward: {dim: 768, activation: swiglu}
    normalization: {type: rmsnorm, norm_first: true}
    position_encoding: {type: rope, theta: 10000}
    dropout: 0.0
  shared_categorical_target_groups: [[byte_0, byte_1]]
  tie_input_output_embeddings: true

Shared groups must contain categorical targets with identical decoder-ID mappings. Weight tying aliases each categorical input table with its output projection. A composite decoder may mix this branch with ordinary linear or MLP branches. For variable-length byte patches, predict length first and encode unused slots with an explicit padding category.

For BERT objectives, autoregressive transformer targets must not include mask in categorical_decoder_special_tokens. Inference excludes mask predictions, which would invalidate the prefix used to generate later targets. The default decoder vocabulary includes other and excludes mask. Configure an explicit empty token list for a categorical target to exclude other as well.

Initialization overrides inherit per semantic group and per weight/bias target. A child overrides only the targets it specifies; preserve keeps the constructed value. Parameters are initialized once per identity. The ingestion adapter inherits the root ingestion policy. Branch overrides that older versions ignored now take effect; exact resume of those older runs is rejected.

initialization_seed accepts integers from zero through 2**63-1, excluding booleans. An explicit seed isolates constructor and custom-initializer randomness from the run stream. Descendants derive seeds using SHA-256 over version 1, namespace seed, relative branch path, and phase; explicit child seeds start a new namespace. Omitted seeds preserve the existing RNG stream and initialization traversal. Initialization seeds do not control runtime dropout. Dropout continues using the run RNG and checkpointed per-rank state.

Dataset freezing can follow the same tree:

dataset:
  part: {metadata_config_path: configs/metadata_configs/raw-items.json}
  criterion: {nextAction: CrossEntropyLoss}
  freeze:
    backbone:
      freeze: [attention.qkv]
    ingestion:
      branches:
        subitems:
          freezing_except: [free_parameter, ingestion.output_projection]
        item:
          freeze: []

A local child selector replaces the inherited decision, including selective unfreezing. freeze: [] unfreezes the scope; freezing_except: [] freezes it. A node containing only branches delegates without changing inherited decisions. Merge parameters retain their parent’s policy; the adapter has its own dataset selector. Unknown branches, policies below leaves, and contradictory alias selections fail. Freezing leaves dropout active and permits gradients through frozen modules. At optimizer boundaries, frozen gradients are removed before AMP unscaling/overflow detection, so a frozen-only overflow cannot suppress a healthy active update. Momentum and weight decay cannot update parameters whose gradients are absent.