SpectraREML/docs/FORMAT.md

# SpectraREML file contract

All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.

## Shared sample order

The following matrices must use exactly the same sample order:

1. GRM;
2. common design matrix;
3. phenotype matrix;
4. extra-covariate matrix.

When supplied, `--grm-id` is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.

## GRM

`--grm-bin` uses the GCTA lower-triangle packed `float32` layout:

```text
G[0,0],
G[1,0], G[1,1],
G[2,0], G[2,1], G[2,2], ...
```

For `n` samples, the exact file size is `4 * n * (n + 1) / 2` bytes.

## Common design

`--base-x` is a row-major `float64` matrix with shape

```text
n_samples × n_base_covariates
```

It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.

## Phenotypes

`--phenotypes` is a row-major `float64` matrix with shape

```text
n_phenotype_rows × n_samples
```

Each task selects one row through `phenotype_row`.

## Extra fixed-effect covariates

`--extra-covariates` is an optional row-major `float32` matrix with shape

```text
n_extra_covariate_rows × n_samples
```

Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.

## Task table

`--tasks` is a UTF-8 tab-separated file with exactly four columns:

```text
task_index  task_id  phenotype_row  n_extra_covariates
0           trait_a  0              0
1           trait_b  1              2
```

Requirements:

- `task_index` is consecutive and zero based;
- `task_id` is nonempty and unique;
- `phenotype_row` is within the phenotype matrix;
- `n_extra_covariates` agrees with the CSR offsets.

## CSR task-to-covariate mapping

`--extra-offsets` is an `int64` array of length `n_tasks + 1`. It begins with zero and is nondecreasing.

`--extra-indices` is an `int32` array of length `offsets[-1]`. For task `i`, its extra-covariate row indices are

```text
indices[offsets[i]:offsets[i+1]]
```

An index must be in `[0, n_extra_covariate_rows)`, and a task cannot reference the same row twice.

## Block output

For block number `KKKKKK`:

```text
block_KKKKKK.summary.tsv
block_KKKKKK.beta.f64.bin
block_KKKKKK.cov.f64.bin
block_KKKKKK.fixed_se.f64.bin
block_KKKKKK.fixed_stat.f64.bin
block_KKKKKK.fixed_ddf.f64.bin
block_KKKKKK.fixed_p.f64.bin
block_KKKKKK.complete
```

The summary header is:

```text
task_index
task_id
status
n_fixed
n_extra_covariates
beta_offset
cov_offset
sigma_g2
sigma_e2
h2
logL
iterations
line_search_steps
grad_inf
fixed_test_method
fixed_test_status
fixed_test_offset
extra_joint_num_df
extra_joint_den_df
extra_joint_f
extra_joint_p
fixed_test_error
error
```

`beta_offset`, `cov_offset`, and `fixed_test_offset` count `float64`
elements, not bytes. A negative beta/covariance offset means that no estimates
were emitted. A negative fixed-test offset means that no valid coefficient-wise
fixed-effect tests were emitted; successful REML estimates are retained even
when inference fails.

The covariance array uses the row-wise packed lower triangle:

```text
(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...
```

The `.complete` marker is written last and contains tab-separated key/value rows:

```text
format  spectra-reml-block-v2
block   0
tasks   256
beta_elements  4096
cov_elements   34816
fixed_test_elements  4096
```

Consumers must ignore blocks without `.complete`.

## Status values

```text
converged
converged_boundary
max_iterations
line_search_failed
rank_deficient
invalid_input
non_positive_covariance
numerical_error
```

`converged_boundary` is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.

## Fixed-effect inference

`--fixed-effect-test` selects `satterthwaite` (the default),
`kenward-roger`, or `none`.

The four `fixed_*` arrays have the same offsets and coefficient order as
`beta`. They contain standard errors, statistics, denominator degrees of
freedom, and p-values. With Satterthwaite inference, a coefficient statistic is
a signed t statistic. With Kenward-Roger inference, it is an F statistic with
one numerator degree of freedom. For every task with extra covariates, the
summary also contains an F test of the joint null that all task-specific fixed
effects are zero.

`fixed_test_status` is one of:

```text
not_requested
ok
boundary_conditional
fit_not_converged
invalid_contrast
information_singular
numerical_error
```

At `converged_boundary`, inference conditions on the accepted active set
`sigma_g2=0`; `fixed_test_status` is `boundary_conditional` and only residual
variance uncertainty contributes to the small-sample adjustment.

Phenotypes are scaled internally by their task-specific OLS residual RMS before
optimization. Reported fixed effects, fixed-effect covariance, variance
components, and restricted likelihood are transformed back to the original
input phenotype units. Input binary files are never modified.

The strong-Wolfe zoom uses safeguarded quadratic interpolation with a 2%
endpoint margin and falls back to bisection. At an evaluation limit or a
collapsed bracket, the last valid likelihood-improving point encountered is
accepted. `line_search_failed` therefore means that the search found no valid
point that improved the starting likelihood.

## Generic finalized output

The Python CLI exports one TSV row per task. It includes the full summary plus:

```text
beta_json
covariance_packed_lower_json
fixed_effect_se_json
fixed_effect_statistic_json
fixed_effect_denominator_df_json
fixed_effect_p_value_json
```

Project-specific software can attach coefficient names. The public C++
inference API also accepts general linear hypotheses `L beta = rhs`; the batch
format currently emits coefficient-wise tests and the joint extra-covariate
test.