SpectraREML/docs/FORMAT.md

SpectraREML file contract

All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.

Shared sample order

The following matrices must use exactly the same sample order:

1. GRM;
2. common design matrix;
3. phenotype matrix;
4. extra-covariate matrix.

When supplied, --grm-id is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.

GRM

--grm-bin uses the GCTA lower-triangle packed float32 layout:

G[0,0],
G[1,0], G[1,1],
G[2,0], G[2,1], G[2,2], ...

For n samples, the exact file size is 4 * n * (n + 1) / 2 bytes.

Common design

--base-x is a row-major float64 matrix with shape

n_samples × n_base_covariates

It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.

Phenotypes

--phenotypes is a row-major float64 matrix with shape

n_phenotype_rows × n_samples

Each task selects one row through phenotype_row.

Extra fixed-effect covariates

--extra-covariates is an optional row-major float32 matrix with shape

n_extra_covariate_rows × n_samples

Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.

Task table

--tasks is a UTF-8 tab-separated file with exactly four columns:

task_index  task_id  phenotype_row  n_extra_covariates
0           trait_a  0              0
1           trait_b  1              2

Requirements:

• task_index is consecutive and zero based;
• task_id is nonempty and unique;
• phenotype_row is within the phenotype matrix;
• n_extra_covariates agrees with the CSR offsets.

CSR task-to-covariate mapping

--extra-offsets is an int64 array of length n_tasks + 1. It begins with zero and is nondecreasing.

--extra-indices is an int32 array of length offsets[-1]. For task i, its extra-covariate row indices are

indices[offsets[i]:offsets[i+1]]

An index must be in [0, n_extra_covariate_rows), and a task cannot reference the same row twice.

Block output

For block number KKKKKK:

block_KKKKKK.summary.tsv
block_KKKKKK.beta.f64.bin
block_KKKKKK.cov.f64.bin
block_KKKKKK.complete

The summary header is:

task_index
task_id
status
n_fixed
n_extra_covariates
beta_offset
cov_offset
sigma_g2
sigma_e2
h2
logL
iterations
line_search_steps
grad_inf
error

beta_offset and cov_offset count float64 elements, not bytes. A negative offset indicates that no estimates were emitted for that task.

The covariance array uses the row-wise packed lower triangle:

(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...

The .complete marker is written last and contains tab-separated key/value rows:

format  spectra-reml-block-v1
block   0
tasks   256
beta_elements  4096
cov_elements   34816

Consumers must ignore blocks without .complete.

Status values

converged
converged_boundary
max_iterations
line_search_failed
rank_deficient
invalid_input
non_positive_covariance
numerical_error

converged_boundary is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.

Phenotypes are scaled internally by their task-specific OLS residual RMS before optimization. Reported fixed effects, fixed-effect covariance, variance components, and restricted likelihood are transformed back to the original input phenotype units. Input binary files are never modified.

The strong-Wolfe zoom uses safeguarded quadratic interpolation with a 2% endpoint margin and falls back to bisection. At an evaluation limit or a collapsed bracket, the last valid likelihood-improving point encountered is accepted. line_search_failed therefore means that the search found no valid point that improved the starting likelihood.

Generic finalized output

The Python CLI exports one TSV row per task. It includes the full summary plus:

beta_json
covariance_packed_lower_json

Project-specific software can attach coefficient names and derive contrasts without changing the numerical engine.