SpectraREML/docs/FORMAT.md

SpectraREML file contract

All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.

Shared sample order

The following matrices must use exactly the same sample order:

1. GRM;
2. common design matrix;
3. phenotype matrix;
4. extra-covariate matrix.

When supplied, --grm-id is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.

GRM

--grm-bin uses the GCTA lower-triangle packed float32 layout:

G[0,0],
G[1,0], G[1,1],
G[2,0], G[2,1], G[2,2], ...

For n samples, the exact file size is 4 * n * (n + 1) / 2 bytes.

Common design

--base-x is a row-major float64 matrix with shape

n_samples × n_base_covariates

It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.

Phenotypes

--phenotypes is a row-major float64 matrix with shape

n_phenotype_rows × n_samples

Each task selects one row through phenotype_row.

Extra fixed-effect covariates

--extra-covariates is an optional row-major float32 matrix with shape

n_extra_covariate_rows × n_samples

Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.

Task table

--tasks is a UTF-8 tab-separated file with exactly four columns:

task_index  task_id  phenotype_row  n_extra_covariates
0           trait_a  0              0
1           trait_b  1              2

Requirements:

• task_index is consecutive and zero based;
• task_id is nonempty and unique;
• phenotype_row is within the phenotype matrix;
• n_extra_covariates agrees with the CSR offsets.

CSR task-to-covariate mapping

--extra-offsets is an int64 array of length n_tasks + 1. It begins with zero and is nondecreasing.

--extra-indices is an int32 array of length offsets[-1]. For task i, its extra-covariate row indices are

indices[offsets[i]:offsets[i+1]]

An index must be in [0, n_extra_covariate_rows), and a task cannot reference the same row twice.

Block output

For block number KKKKKK:

block_KKKKKK.summary.tsv
block_KKKKKK.beta.f64.bin
block_KKKKKK.cov.f64.bin
block_KKKKKK.complete

The summary header is:

task_index
task_id
status
n_fixed
n_extra_covariates
beta_offset
cov_offset
sigma_g2
sigma_e2
h2
logL
iterations
line_search_steps
grad_inf
error

beta_offset and cov_offset count float64 elements, not bytes. A negative offset indicates that no estimates were emitted for that task.

The covariance array uses the row-wise packed lower triangle:

(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...

The .complete marker is written last and contains tab-separated key/value rows:

format  spectra-reml-block-v1
block   0
tasks   256
beta_elements  4096
cov_elements   34816

Consumers must ignore blocks without .complete.

Status values

converged
converged_boundary
max_iterations
line_search_failed
rank_deficient
invalid_input
non_positive_covariance
numerical_error

converged_boundary is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.

Generic finalized output

The Python CLI exports one TSV row per task. It includes the full summary plus:

beta_json
covariance_packed_lower_json

Project-specific software can attach coefficient names and derive contrasts without changing the numerical engine.