Preprocess Command Guide¶

The sequifier preprocess command transforms raw tabular data (CSV or Parquet) into the specific sequence format required for training transformer sequence models. It handles windowing, data splitting (train/validation/test), categorical encoding, and optional numerical standardization.

Usage¶

sequifier preprocess --config-path configs/preprocess.yaml

CLI Overrides¶

Values passed on the command line override the YAML before validation.

Flag	Overrides / Action
`-r`, `--randomize`	Generates a random `seed`. The seed affects `between_sequence` split assignment.
`-dp`, `--data-path`	Overrides `preprocessing_data_path`.
`-sc`, `--selected-columns`	Overrides `selected_columns` with a space-separated list. Use `None` to process all columns.

Composable Configuration Files¶

A preprocessing entry config may set additional_config_paths to one non-empty string, a list of non-empty strings, or null. Relative paths resolve against the entry config’s project_root; absolute paths are used directly. Fragments are direct only and cannot include further fragments. They may share nested containers when their child fields are disjoint, but duplicate fields are errors. CLI values override the completed file composition.

Configuration Fields¶

The configuration is defined in a YAML file (e.g., preprocess.yaml). Below are the available fields, their requirements, and their functions.

1. File System & Input/Output¶

Field	Type	Mandatory	Default	Description
`project_root`	`str`	Yes	-	The root directory of your Sequifier project. Usually `.`
`additional_config_paths`	`str`, `list[str]`, or `null`	No	`null`	Direct complementary YAML fragments. Relative paths resolve against `project_root`; recursive composition and duplicate fields are rejected.
`preprocessing_data_path`	`str`	Yes	-	Path to the raw input file or folder.
`read_format`	`str`	No	`csv`	Format of input data (`csv`, `parquet`).
`write_format`	`str`	No	`parquet`	Format of output data (`csv`, `parquet`, `pt`).
`merge_output`	`bool`	No	`true`	Whether to merge split files into single files or keep them sharded.
`continue_preprocessing`	`bool`	No	`false`	If `true`, resumes from an existing preprocessing temp folder created by an interrupted run.

Important Constraint on write_format:

If write_format is pt (PyTorch tensors), merge_output must be false.

If write_format is parquet, merge_output can be false or true.

If write_format is csv, merge_output must be true. For distributed training, merge_output must be set to false.

2. Column Selection & Filtering¶

Field	Type	Mandatory	Default	Description
`selected_columns`	`list[str]`	No	`null`	A specific list of columns to process. If `null`, all columns (except metadata) are processed.
`column_data_types`	`dict[str, str]`	No	`null`	Optional output dtype map for processed columns, such as `Float32`, `Float64`, `Int32`, or `Int64`. If set, every processed column must be included. Parquet uses one unified sequence dtype; `pt` writes each variable to its configured tensor dtype.
`normalize_real_columns`	`bool`	No	`true`	If `true`, Z-score normalizes real-valued columns. Set to `false` to preserve their original values. Statistics are still recorded in metadata.
`max_rows`	`int`	No	`null`	Limits processing to the first N rows. Useful for rapid debugging.
`metadata_config_path`	`Optional[str]`	No	`null`	Use a preexisting metadata config for tokenizing discrete columns and, when enabled, standardizing real-valued columns.
`mask_column`	`Optional[str]`	No	`null`	Optional input column used as a row-level mask. If set, `metadata_config_path` must also be set.
`use_precomputed_maps`	`list[str]`	No	`null`	If not `null`, enforces the use of precomputed maps for the variables in the list.

3. Sequence Logic & Splitting¶

Field	Type	Mandatory	Default	Description
`window_length`	`int`	Yes	-	The physical serialized window width written to preprocessed data.
`max_target_offset`	`int`	No	`1`	Number of future items retained after the model input window. Use `0` for BERT-style same-width inputs and targets; use `1` for causal next-item training.
`split_ratios`	`list[float]`	Yes	-	Ordered train/validation/test proportions. Must sum to 1.0.
`split_method`	`str`	No	`within_sequence`	How rows are assigned to splits (`within_sequence` or `between_sequence`).
`window_strides`	`list[int]`	No	`[window_length]*N`	Window stride for each split; entry `i` corresponds to `split_ratios[i]`.
`window_placement`	`str`	No	`distribute`	Strategy for selecting start indices (`distribute` or `exact`).
`allow_sequence_splitting`	`bool`	No	`false`	If `false`, a single sequence is kept within one preprocessing batch.

4. Performance & System¶

Field	Type	Mandatory	Default	Description
`seed`	`int`	No	`1010`	Random seed for reproducibility.
`n_cores`	`int`	No	Max Cores	Number of CPU cores to use for parallel processing.
`batches_per_file`	`int`	No	`1024`	Only used when `write_format: pt`. Controls how many sequences are packed into one `.pt` file.
`process_by_file`	`bool`	No	`true`	Memory optimization. If `true`, processes one input file at a time.

Key Trade-offs and Decisions¶

1. `write_format`: `parquet` vs. `pt`¶

Choose parquet (default): Unless you have a specific reason, use parquet. Note: If you are doing distributed training, Parquet support is currently in Beta.
Choose pt: Use pt data loading if speed and CPU overhead are your primary bottlenecks, or if you are running multi-GPU distributed training. This format is the most stable choice for high-throughput scaling.

2. `window_strides` configuration¶

window_length: non-overlapping windows and less data.
1: maximum overlap, coverage, storage, and training time.
A common compromise is a larger train/validation stride and test stride 1, for example window_strides: [24, 24, 1].

3. `window_placement`: `distribute` vs `exact`¶

distribute (Default): The algorithm adjusts the start indices slightly to minimize the overlap of the final subsequence with the previous one, ensuring the data covers the full sequence length as evenly as possible. Recommended for most use cases.
exact: Strictly enforces the stride. If the sequence length minus the window size isn’t perfectly divisible by the stride, this will raise an error. Use this only if mathematical precision of the sliding window is strictly required by your downstream application or evaluation code.

4. Advanced: Static Vocabularies (Custom ID Maps)¶

By default, Sequifier dynamically builds ID maps from the data found in the input file. However, in production systems, you often need a fixed vocabulary to ensure that ID “105” always maps to “Item_X”, regardless of the daily training batch.

To use a static vocabulary:

Create a folder configs/id_maps/ in your project root.
Add JSON files named {COLUMN_NAME}.json.
The format must be a dictionary mapping ordinary data values to integers starting at 3. Reserved labels may be included only with their fixed IDs.

Reserved Indices:

0: Reserved for [unknown] (padding/missing).

1: Reserved for [other] (unseen values not in your map).

2: Reserved for [mask].

3+: Your data.

Example configs/id_maps/itemId.json:

{
    "apple": 3,
    "banana": 4,
    "cherry": 5
}

Outputs¶

After running preprocess, the following are generated:

Data Files: Located in data/. Depending on your configuration, these will be merged files such as [NAME]-split0.parquet (Training), [NAME]-split1.parquet (Validation), etc., or split folders such as [NAME]-split0/ containing .pt or .parquet shards.
Metadata Config: Located in configs/metadata_configs/[NAME].json.
- Crucial: This file contains the integer mappings for categorical variables (id_maps), statistics for real variables (selected_columns_statistics), and whether those variables were normalized (normalize_real_columns).
- Next Step: Reference this file from dataset.part.metadata_config_path in a singleton training config, or from dataset_training.<dataset>.parts.<part>.metadata_config_path in a named training config. In inference, either preprocessing_data_path or metadata_config_path can locate the metadata and its split paths.

Named depth layouts¶

Repeated rows can describe one outer item with a fixed-capacity child collection. Depth behavior is explicit; a column named subItemPosition alone remains an ordinary flat feature.

project_root: .
preprocessing_data_path: data/raw-items
read_format: parquet
write_format: pt
merge_output: false
selected_columns: [accountType, subitemType, subitemAmount, nextAction]
depth_layouts:
  subitems:
    position_column: subItemPosition
    columns: [subitemType, subitemAmount]
    context_length: 16
    position_base: 0
    allow_gaps: false
window_length: 129
max_target_offset: 1
split_ratios: [0.8, 0.1, 0.1]
window_strides: [128, 128, 128]

Every file must contain the position column, which is read automatically and excluded from feature statistics and output types. Configure at most one raw layout, use PT output without merging, and omit mask_column. Reused metadata must have the same complete layout definition, output types, and normalization policy. String identifiers must be convertible to signed Int64; item and depth positions must have integer source types.

The adapter indexes raw fragments on disk before grouping them. max_rows counts complete outer items ordered by (sequenceId, itemPosition), including children found in later files. Shallow features must agree across every child row before casting or mapping. Shallow statistics count each item once; deep statistics count occupied child slots. Both populations are selected before split extraction. Materialization uses bounded windows and output batches; batches_per_file bounds the number of windows accumulated per split on this path. This adapter is currently sequential; n_cores does not parallelize it.

Child positions map to physical slots by subtracting position_base. Without allow_gaps, occupied slots must be a prefix starting at zero. Tail padding is always allowed. With gaps enabled, physical slots remain unchanged. Outer item positions must be continuous within each selected sequence. An item in this raw format must have at least one child; null child rows do not encode emptiness.

Flat PT files retain the five-element tuple. Depth files use version 2 of sequifier_tensor_batch, with shallow [N,W], deep [N,W,D], and boolean masks under metadata.depth_valid_masks.<layout>. Metadata records depth_layouts and tensor_payload_version, separately from the stored window version. Categorical padding is the existing unknown-token ID (zero); real padding is finite zero after normalization. Temporal padding has false depth masks.

External tensor payloads may contain several layouts with different capacities and independently empty collections. Use StoredTensorBatch, save_pt_payload, and load_pt_payload in sequifier.io.pt_payload, supplying the complete layout registry and categorical vocabulary sizes. All stored feature values, including masked slots, must have legal categorical indices and finite real values. Empty collections use an all-false mask. Missing masks and forbidden internal gaps are errors. Selected interfaces compare only relevant layout feature membership and layout properties, so unused stored layouts/features can be added independently.

To consume a named layout during training, configure a depth_transformer ingestion branch whose layout names this entry and whose columns are selected features from it. See the training guide.