Preprocess Command Guide

The sequifier preprocess command transforms raw tabular data (CSV or Parquet) into the specific sequence format required for training transformer sequence models. It handles windowing, data splitting (train/validation/test), categorical encoding, and optional numerical standardization.

Usage

sequifier preprocess --config-path configs/preprocess.yaml

CLI Overrides

Values passed on the command line override the YAML before validation.

Flag	Overrides / Action
-r, --randomize	Generates a random seed. The seed affects between_sequence split assignment.
-dp, --data-path	Overrides preprocessing_data_path.
-sc, --selected-columns	Overrides selected_columns with a space-separated list. Use None to process all columns.

Composable Configuration Files

A preprocessing entry config may set additional_config_paths to one non-empty string, a list of non-empty strings, or null. Relative paths resolve against the entry config’s project_root; absolute paths are used directly. Fragments are direct only and cannot include further fragments. They may share nested containers when their child fields are disjoint, but duplicate fields are errors. CLI values override the completed file composition.

Configuration Fields

The configuration is defined in a YAML file (e.g., preprocess.yaml). Below are the available fields, their requirements, and their functions.

1. File System & Input/Output

Field	Type	Mandatory	Default	Description
project_root	str	Yes	-	The root directory of your Sequifier project. Usually .
additional_config_paths	str, list[str], or null	No	null	Direct complementary YAML fragments. Relative paths resolve against project_root; recursive composition and duplicate fields are rejected.
preprocessing_data_path	str	Yes	-	Path to the raw input file or folder.
read_format	str	No	csv	Format of input data (csv, parquet).
write_format	str	No	parquet	Format of output data (csv, parquet, pt).
merge_output	bool	No	true	Whether to merge split files into single files or keep them sharded.
continue_preprocessing	bool	No	false	If true, resumes from an existing preprocessing temp folder created by an interrupted run.

Important Constraint on write_format:

• If write_format is pt (PyTorch tensors), merge_output must be false.

• If write_format is parquet, merge_output can be false or true.

• If write_format is csv, merge_output must be true. For distributed training, merge_output must be set to false.

2. Column Selection & Filtering

Field	Type	Mandatory	Default	Description
selected_columns	list[str]	No	null	A specific list of columns to process. If null, all columns (except metadata) are processed.
column_data_types	dict[str, str]	No	null	Optional output dtype map for processed columns, such as Float32, Float64, Int32, or Int64. If set, every processed column must be included. Parquet uses one unified sequence dtype; pt writes each variable to its configured tensor dtype.
normalize_real_columns	bool	No	true	If true, Z-score normalizes real-valued columns. Set to false to preserve their original values. Statistics are still recorded in metadata.
max_rows	int	No	null	Limits processing to the first N rows. Useful for rapid debugging.
metadata_config_path	Optional[str]	No	null	Use a preexisting metadata config for tokenizing discrete columns and, when enabled, standardizing real-valued columns.
mask_column	Optional[str]	No	null	Optional input column used as a row-level mask. If set, metadata_config_path must also be set.
use_precomputed_maps	list[str]	No	null	If not null, enforces the use of precomputed maps for the variables in the list.

3. Sequence Logic & Splitting

Field	Type	Mandatory	Default	Description
window_length	int	Yes	-	The physical serialized window width written to preprocessed data.
max_target_offset	int	No	1	Number of future items retained after the model input window. Use 0 for BERT-style same-width inputs and targets; use 1 for causal next-item training.
split_ratios	list[float]	Yes	-	Ordered train/validation/test proportions. Must sum to 1.0.
split_method	str	No	within_sequence	How rows are assigned to splits (within_sequence or between_sequence).
window_strides	list[int]	No	[window_length]*N	Window stride for each split; entry i corresponds to split_ratios[i].
window_placement	str	No	distribute	Strategy for selecting start indices (distribute or exact).
allow_sequence_splitting	bool	No	false	If false, a single sequence is kept within one preprocessing batch.

4. Performance & System

Field	Type	Mandatory	Default	Description
seed	int	No	1010	Random seed for reproducibility.
n_cores	int	No	Max Cores	Number of CPU cores to use for parallel processing.
batches_per_file	int	No	1024	Only used when write_format: pt. Controls how many sequences are packed into one .pt file.
process_by_file	bool	No	true	Memory optimization. If true, processes one input file at a time.

---

Key Trade-offs and Decisions

1. write_format: parquet vs. pt

• Choose parquet (default): Unless you have a specific reason, use parquet. Note: If you are doing distributed training, Parquet support is currently in Beta.

• Choose pt: Use pt data loading if speed and CPU overhead are your primary bottlenecks, or if you are running multi-GPU distributed training. This format is the most stable choice for high-throughput scaling.

2. window_strides configuration

• window_length: non-overlapping windows and less data.

• 1: maximum overlap, coverage, storage, and training time.

• A common compromise is a larger train/validation stride and test stride 1, for example window_strides: [24, 24, 1].

3. window_placement: distribute vs exact

• distribute (Default): The algorithm adjusts the start indices slightly to minimize the overlap of the final subsequence with the previous one, ensuring the data covers the full sequence length as evenly as possible. Recommended for most use cases.

• exact: Strictly enforces the stride. If the sequence length minus the window size isn’t perfectly divisible by the stride, this will raise an error. Use this only if mathematical precision of the sliding window is strictly required by your downstream application or evaluation code.

4. Advanced: Static Vocabularies (Custom ID Maps)

By default, Sequifier dynamically builds ID maps from the data found in the input file. However, in production systems, you often need a fixed vocabulary to ensure that ID “105” always maps to “Item_X”, regardless of the daily training batch.

To use a static vocabulary:

1. Create a folder configs/id_maps/ in your project root.

2. Add JSON files named {COLUMN_NAME}.json.

3. The format must be a dictionary mapping ordinary data values to integers starting at 3. Reserved labels may be included only with their fixed IDs.

Reserved Indices:

• 0: Reserved for [unknown] (padding/missing).

• 1: Reserved for [other] (unseen values not in your map).

• 2: Reserved for [mask].

• 3+: Your data.

Example configs/id_maps/itemId.json:

{
    "apple": 3,
    "banana": 4,
    "cherry": 5
}

---

Outputs

After running preprocess, the following are generated:

1. Data Files: Located in data/. Depending on your configuration, these will be merged files such as [NAME]-split0.parquet (Training), [NAME]-split1.parquet (Validation), etc., or split folders such as [NAME]-split0/ containing .pt or .parquet shards.

2. Metadata Config: Located in configs/metadata_configs/[NAME].json.

   • Crucial: This file contains the integer mappings for categorical variables (id_maps), statistics for real variables (selected_columns_statistics), and whether those variables were normalized (normalize_real_columns).

   • Next Step: Reference this file from dataset.part.metadata_config_path in a singleton training config, or from dataset_training.<dataset>.parts.<part>.metadata_config_path in a named training config. In inference, either preprocessing_data_path or metadata_config_path can locate the metadata and its split paths.