267 lines
7.5 KiB
Markdown
267 lines
7.5 KiB
Markdown
# SpectraREML file contract
|
||
|
||
All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.
|
||
|
||
## Shared sample order
|
||
|
||
The following matrices must use exactly the same sample order:
|
||
|
||
1. GRM;
|
||
2. common design matrix;
|
||
3. phenotype matrix;
|
||
4. extra-covariate matrix.
|
||
|
||
When supplied, `--grm-id` is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.
|
||
|
||
## GRM
|
||
|
||
`--grm-bin` uses the GCTA lower-triangle packed `float32` layout:
|
||
|
||
```text
|
||
G[0,0],
|
||
G[1,0], G[1,1],
|
||
G[2,0], G[2,1], G[2,2], ...
|
||
```
|
||
|
||
For `n` samples, the exact file size is `4 * n * (n + 1) / 2` bytes.
|
||
|
||
## Common design
|
||
|
||
`--base-x` is a row-major `float64` matrix with shape
|
||
|
||
```text
|
||
n_samples × n_base_covariates
|
||
```
|
||
|
||
It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.
|
||
|
||
## Phenotypes
|
||
|
||
`--phenotypes` is a row-major `float64` matrix with shape
|
||
|
||
```text
|
||
n_phenotype_rows × n_samples
|
||
```
|
||
|
||
Each task selects one row through `phenotype_row`.
|
||
|
||
## Extra fixed-effect covariates
|
||
|
||
`--extra-covariates` is an optional row-major `float32` matrix with shape
|
||
|
||
```text
|
||
n_extra_covariate_rows × n_samples
|
||
```
|
||
|
||
Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.
|
||
|
||
## Task table
|
||
|
||
`--tasks` is a UTF-8 tab-separated file with exactly four columns:
|
||
|
||
```text
|
||
task_index task_id phenotype_row n_extra_covariates
|
||
0 trait_a 0 0
|
||
1 trait_b 1 2
|
||
```
|
||
|
||
Requirements:
|
||
|
||
- `task_index` is consecutive and zero based;
|
||
- `task_id` is nonempty and unique;
|
||
- `phenotype_row` is within the phenotype matrix;
|
||
- `n_extra_covariates` agrees with the CSR offsets.
|
||
|
||
## CSR task-to-covariate mapping
|
||
|
||
`--extra-offsets` is an `int64` array of length `n_tasks + 1`. It begins with zero and is nondecreasing.
|
||
|
||
`--extra-indices` is an `int32` array of length `offsets[-1]`. For task `i`, its extra-covariate row indices are
|
||
|
||
```text
|
||
indices[offsets[i]:offsets[i+1]]
|
||
```
|
||
|
||
An index must be in `[0, n_extra_covariate_rows)`, and a task cannot reference the same row twice.
|
||
|
||
## Named shared fixed-effect contrasts
|
||
|
||
Optional arbitrary hypotheses use two files. `--contrast-matrix` is a
|
||
row-major `float64` matrix with shape `n_contrasts x n_base_covariates`; its
|
||
columns follow `base_x` exactly. Task-specific extra fixed-effect columns are
|
||
assigned zero weights automatically. `--contrast-metadata` is a UTF-8 TSV:
|
||
|
||
```text
|
||
contrast_index contrast_id rhs
|
||
0 group_a_b 0
|
||
```
|
||
|
||
Indices are consecutive and zero based, IDs are unique, and every matrix row
|
||
must contain a nonzero weight. Each row tests `L beta = rhs` with the method
|
||
selected by `--fixed-effect-test` (Satterthwaite or Kenward-Roger).
|
||
|
||
## Block output
|
||
|
||
For block number `KKKKKK`:
|
||
|
||
```text
|
||
block_KKKKKK.summary.tsv
|
||
block_KKKKKK.beta.f64.bin
|
||
block_KKKKKK.cov.f64.bin
|
||
block_KKKKKK.fixed_se.f64.bin
|
||
block_KKKKKK.fixed_stat.f64.bin
|
||
block_KKKKKK.fixed_ddf.f64.bin
|
||
block_KKKKKK.fixed_p.f64.bin
|
||
block_KKKKKK.contrast_estimate.f64.bin
|
||
block_KKKKKK.contrast_se.f64.bin
|
||
block_KKKKKK.contrast_stat.f64.bin
|
||
block_KKKKKK.contrast_numdf.f64.bin
|
||
block_KKKKKK.contrast_ddf.f64.bin
|
||
block_KKKKKK.contrast_p.f64.bin
|
||
block_KKKKKK.complete
|
||
```
|
||
|
||
The summary header is:
|
||
|
||
```text
|
||
task_index
|
||
task_id
|
||
status
|
||
n_fixed
|
||
n_extra_covariates
|
||
beta_offset
|
||
cov_offset
|
||
sigma_g2
|
||
sigma_e2
|
||
h2
|
||
logL
|
||
iterations
|
||
line_search_steps
|
||
grad_inf
|
||
fixed_test_method
|
||
fixed_test_status
|
||
fixed_test_offset
|
||
contrast_test_offset
|
||
extra_joint_num_df
|
||
extra_joint_den_df
|
||
extra_joint_f
|
||
extra_joint_p
|
||
fixed_test_error
|
||
error
|
||
```
|
||
|
||
`beta_offset`, `cov_offset`, and `fixed_test_offset` count `float64`
|
||
elements, not bytes. A negative beta/covariance offset means that no estimates
|
||
were emitted. A negative fixed-test offset means that no valid coefficient-wise
|
||
fixed-effect tests were emitted; successful REML estimates are retained even
|
||
when inference fails.
|
||
|
||
The covariance array uses the row-wise packed lower triangle:
|
||
|
||
```text
|
||
(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...
|
||
```
|
||
|
||
The `.complete` marker is written last and contains tab-separated key/value rows:
|
||
|
||
```text
|
||
format spectra-reml-block-v4
|
||
block 0
|
||
tasks 256
|
||
beta_elements 4096
|
||
cov_elements 34816
|
||
fixed_test_elements 4096
|
||
contrast_test_elements 1280
|
||
```
|
||
|
||
Consumers must ignore blocks without `.complete`.
|
||
|
||
## Status values
|
||
|
||
```text
|
||
converged
|
||
converged_boundary
|
||
max_iterations
|
||
line_search_failed
|
||
rank_deficient
|
||
invalid_input
|
||
non_positive_covariance
|
||
numerical_error
|
||
```
|
||
|
||
`converged_boundary` is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.
|
||
|
||
## Fixed-effect inference
|
||
|
||
`--fixed-effect-test` selects `satterthwaite` (the default),
|
||
`kenward-roger`, or `none`.
|
||
|
||
The four `fixed_*` arrays have the same offsets and coefficient order as
|
||
`beta`. They contain standard errors, statistics, denominator degrees of
|
||
freedom, and p-values. With ordinary least-squares or Satterthwaite inference,
|
||
a coefficient statistic is a signed t statistic. With Kenward-Roger inference,
|
||
it is an F statistic with one numerator degree of freedom. For every task with
|
||
extra covariates, the summary also contains a test of the joint null that all
|
||
task-specific fixed effects are zero.
|
||
|
||
The six `contrast_*` arrays share `contrast_test_offset` and follow the
|
||
contrast metadata order. A one-row ordinary least-squares or Satterthwaite
|
||
contrast reports a signed t statistic; Kenward-Roger reports an F statistic
|
||
with one numerator degree of freedom. All methods retain the signed `L beta`
|
||
estimate and their standard error. Negative `contrast_test_offset` means that
|
||
no valid contrast tests were emitted for that task.
|
||
|
||
`fixed_test_status` is one of:
|
||
|
||
```text
|
||
not_requested
|
||
ok
|
||
boundary_ols
|
||
line_search_conditional
|
||
fit_not_converged
|
||
invalid_contrast
|
||
information_singular
|
||
numerical_error
|
||
```
|
||
|
||
At `converged_boundary`, the random term is removed and the fixed model is
|
||
refitted as ordinary least squares. `fixed_test_method` is
|
||
`ordinary-least-squares`, `fixed_test_status` is `boundary_ols`, and the
|
||
denominator degrees of freedom are `n-rank(X)`. At `line_search_failed`, the
|
||
last retained REML iterate is tested with the requested Satterthwaite or
|
||
Kenward-Roger method and `fixed_test_status` is
|
||
`line_search_conditional`; the optimizer flag remains visible in `status`.
|
||
|
||
Phenotypes are scaled internally by their task-specific OLS residual RMS before
|
||
optimization. Reported fixed effects, fixed-effect covariance, variance
|
||
components, and restricted likelihood are transformed back to the original
|
||
input phenotype units. Input binary files are never modified.
|
||
|
||
The strong-Wolfe zoom uses safeguarded quadratic interpolation with a 2%
|
||
endpoint margin and falls back to bisection. At an evaluation limit or a
|
||
collapsed bracket, the last valid likelihood-improving point encountered is
|
||
accepted. `line_search_failed` therefore means that the search found no valid
|
||
point that improved the starting likelihood.
|
||
|
||
## Generic finalized output
|
||
|
||
The Python CLI exports one TSV row per task. It includes the full summary plus:
|
||
|
||
```text
|
||
beta_json
|
||
covariance_packed_lower_json
|
||
fixed_effect_se_json
|
||
fixed_effect_statistic_json
|
||
fixed_effect_denominator_df_json
|
||
fixed_effect_p_value_json
|
||
contrast_estimate_json
|
||
contrast_standard_error_json
|
||
contrast_statistic_json
|
||
contrast_numerator_df_json
|
||
contrast_denominator_df_json
|
||
contrast_p_value_json
|
||
```
|
||
|
||
Contrast JSON arrays follow `contrast_index`; IDs and right-hand sides are
|
||
stored in the manifest-referenced contrast metadata TSV.
|