SpectraREML/docs/FORMAT.md

SpectraREML file contract

All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.

Shared sample order

The following matrices must use exactly the same sample order:

1. GRM;
2. common design matrix;
3. phenotype matrix;
4. extra-covariate matrix.

When supplied, --grm-id is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.

GRM

--grm-bin uses the GCTA lower-triangle packed float32 layout:

G[0,0],
G[1,0], G[1,1],
G[2,0], G[2,1], G[2,2], ...

For n samples, the exact file size is 4 * n * (n + 1) / 2 bytes.

Common design

--base-x is a row-major float64 matrix with shape

n_samples × n_base_covariates

It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.

Phenotypes

--phenotypes is a row-major float64 matrix with shape

n_phenotype_rows × n_samples

Each task selects one row through phenotype_row.

Extra fixed-effect covariates

--extra-covariates is an optional row-major float32 matrix with shape

n_extra_covariate_rows × n_samples

Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.

Task table

--tasks is a UTF-8 tab-separated file with exactly four columns:

task_index  task_id  phenotype_row  n_extra_covariates
0           trait_a  0              0
1           trait_b  1              2

Requirements:

• task_index is consecutive and zero based;
• task_id is nonempty and unique;
• phenotype_row is within the phenotype matrix;
• n_extra_covariates agrees with the CSR offsets.

CSR task-to-covariate mapping

--extra-offsets is an int64 array of length n_tasks + 1. It begins with zero and is nondecreasing.

--extra-indices is an int32 array of length offsets[-1]. For task i, its extra-covariate row indices are

indices[offsets[i]:offsets[i+1]]

An index must be in [0, n_extra_covariate_rows), and a task cannot reference the same row twice.

Named shared fixed-effect contrasts

Optional arbitrary hypotheses use two files. --contrast-matrix is a row-major float64 matrix with shape n_contrasts x n_base_covariates; its columns follow base_x exactly. Task-specific extra fixed-effect columns are assigned zero weights automatically. --contrast-metadata is a UTF-8 TSV:

contrast_index  contrast_id  rhs
0               group_a_b    0

Indices are consecutive and zero based, IDs are unique, and every matrix row must contain a nonzero weight. Each row tests L beta = rhs with the method selected by --fixed-effect-test (Satterthwaite or Kenward-Roger).

Block output

For block number KKKKKK:

block_KKKKKK.summary.tsv
block_KKKKKK.beta.f64.bin
block_KKKKKK.cov.f64.bin
block_KKKKKK.fixed_se.f64.bin
block_KKKKKK.fixed_stat.f64.bin
block_KKKKKK.fixed_ddf.f64.bin
block_KKKKKK.fixed_p.f64.bin
block_KKKKKK.contrast_estimate.f64.bin
block_KKKKKK.contrast_se.f64.bin
block_KKKKKK.contrast_stat.f64.bin
block_KKKKKK.contrast_numdf.f64.bin
block_KKKKKK.contrast_ddf.f64.bin
block_KKKKKK.contrast_p.f64.bin
block_KKKKKK.complete

The summary header is:

task_index
task_id
status
n_fixed
n_extra_covariates
beta_offset
cov_offset
sigma_g2
sigma_e2
h2
logL
iterations
line_search_steps
grad_inf
fixed_test_method
fixed_test_status
fixed_test_offset
contrast_test_offset
extra_joint_num_df
extra_joint_den_df
extra_joint_f
extra_joint_p
fixed_test_error
error

beta_offset, cov_offset, and fixed_test_offset count float64 elements, not bytes. A negative beta/covariance offset means that no estimates were emitted. A negative fixed-test offset means that no valid coefficient-wise fixed-effect tests were emitted; successful REML estimates are retained even when inference fails.

The covariance array uses the row-wise packed lower triangle:

(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...

The .complete marker is written last and contains tab-separated key/value rows:

format  spectra-reml-block-v4
block   0
tasks   256
beta_elements  4096
cov_elements   34816
fixed_test_elements  4096
contrast_test_elements  1280

Consumers must ignore blocks without .complete.

Status values

converged
converged_boundary
max_iterations
line_search_failed
rank_deficient
invalid_input
non_positive_covariance
numerical_error

converged_boundary is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.

Fixed-effect inference

--fixed-effect-test selects satterthwaite (the default), kenward-roger, or none.

The four fixed_* arrays have the same offsets and coefficient order as beta. They contain standard errors, statistics, denominator degrees of freedom, and p-values. With ordinary least-squares or Satterthwaite inference, a coefficient statistic is a signed t statistic. With Kenward-Roger inference, it is an F statistic with one numerator degree of freedom. For every task with extra covariates, the summary also contains a test of the joint null that all task-specific fixed effects are zero.

The six contrast_* arrays share contrast_test_offset and follow the contrast metadata order. A one-row ordinary least-squares or Satterthwaite contrast reports a signed t statistic; Kenward-Roger reports an F statistic with one numerator degree of freedom. All methods retain the signed L beta estimate and their standard error. Negative contrast_test_offset means that no valid contrast tests were emitted for that task.

fixed_test_status is one of:

not_requested
ok
boundary_ols
line_search_conditional
fit_not_converged
invalid_contrast
information_singular
numerical_error

At converged_boundary, the random term is removed and the fixed model is refitted as ordinary least squares. fixed_test_method is ordinary-least-squares, fixed_test_status is boundary_ols, and the denominator degrees of freedom are n-rank(X). At line_search_failed, the last retained REML iterate is tested with the requested Satterthwaite or Kenward-Roger method and fixed_test_status is line_search_conditional; the optimizer flag remains visible in status.

Phenotypes are scaled internally by their task-specific OLS residual RMS before optimization. Reported fixed effects, fixed-effect covariance, variance components, and restricted likelihood are transformed back to the original input phenotype units. Input binary files are never modified.

The strong-Wolfe zoom uses safeguarded quadratic interpolation with a 2% endpoint margin and falls back to bisection. At an evaluation limit or a collapsed bracket, the last valid likelihood-improving point encountered is accepted. line_search_failed therefore means that the search found no valid point that improved the starting likelihood.

Generic finalized output

The Python CLI exports one TSV row per task. It includes the full summary plus:

beta_json
covariance_packed_lower_json
fixed_effect_se_json
fixed_effect_statistic_json
fixed_effect_denominator_df_json
fixed_effect_p_value_json
contrast_estimate_json
contrast_standard_error_json
contrast_statistic_json
contrast_numerator_df_json
contrast_denominator_df_json
contrast_p_value_json

Contrast JSON arrays follow contrast_index; IDs and right-hand sides are stored in the manifest-referenced contrast metadata TSV.