163 lines
3.6 KiB
Markdown
163 lines
3.6 KiB
Markdown
# SpectraREML file contract
|
||
|
||
All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.
|
||
|
||
## Shared sample order
|
||
|
||
The following matrices must use exactly the same sample order:
|
||
|
||
1. GRM;
|
||
2. common design matrix;
|
||
3. phenotype matrix;
|
||
4. extra-covariate matrix.
|
||
|
||
When supplied, `--grm-id` is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.
|
||
|
||
## GRM
|
||
|
||
`--grm-bin` uses the GCTA lower-triangle packed `float32` layout:
|
||
|
||
```text
|
||
G[0,0],
|
||
G[1,0], G[1,1],
|
||
G[2,0], G[2,1], G[2,2], ...
|
||
```
|
||
|
||
For `n` samples, the exact file size is `4 * n * (n + 1) / 2` bytes.
|
||
|
||
## Common design
|
||
|
||
`--base-x` is a row-major `float64` matrix with shape
|
||
|
||
```text
|
||
n_samples × n_base_covariates
|
||
```
|
||
|
||
It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.
|
||
|
||
## Phenotypes
|
||
|
||
`--phenotypes` is a row-major `float64` matrix with shape
|
||
|
||
```text
|
||
n_phenotype_rows × n_samples
|
||
```
|
||
|
||
Each task selects one row through `phenotype_row`.
|
||
|
||
## Extra fixed-effect covariates
|
||
|
||
`--extra-covariates` is an optional row-major `float32` matrix with shape
|
||
|
||
```text
|
||
n_extra_covariate_rows × n_samples
|
||
```
|
||
|
||
Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.
|
||
|
||
## Task table
|
||
|
||
`--tasks` is a UTF-8 tab-separated file with exactly four columns:
|
||
|
||
```text
|
||
task_index task_id phenotype_row n_extra_covariates
|
||
0 trait_a 0 0
|
||
1 trait_b 1 2
|
||
```
|
||
|
||
Requirements:
|
||
|
||
- `task_index` is consecutive and zero based;
|
||
- `task_id` is nonempty and unique;
|
||
- `phenotype_row` is within the phenotype matrix;
|
||
- `n_extra_covariates` agrees with the CSR offsets.
|
||
|
||
## CSR task-to-covariate mapping
|
||
|
||
`--extra-offsets` is an `int64` array of length `n_tasks + 1`. It begins with zero and is nondecreasing.
|
||
|
||
`--extra-indices` is an `int32` array of length `offsets[-1]`. For task `i`, its extra-covariate row indices are
|
||
|
||
```text
|
||
indices[offsets[i]:offsets[i+1]]
|
||
```
|
||
|
||
An index must be in `[0, n_extra_covariate_rows)`, and a task cannot reference the same row twice.
|
||
|
||
## Block output
|
||
|
||
For block number `KKKKKK`:
|
||
|
||
```text
|
||
block_KKKKKK.summary.tsv
|
||
block_KKKKKK.beta.f64.bin
|
||
block_KKKKKK.cov.f64.bin
|
||
block_KKKKKK.complete
|
||
```
|
||
|
||
The summary header is:
|
||
|
||
```text
|
||
task_index
|
||
task_id
|
||
status
|
||
n_fixed
|
||
n_extra_covariates
|
||
beta_offset
|
||
cov_offset
|
||
sigma_g2
|
||
sigma_e2
|
||
h2
|
||
logL
|
||
iterations
|
||
line_search_steps
|
||
grad_inf
|
||
error
|
||
```
|
||
|
||
`beta_offset` and `cov_offset` count `float64` elements, not bytes. A negative offset indicates that no estimates were emitted for that task.
|
||
|
||
The covariance array uses the row-wise packed lower triangle:
|
||
|
||
```text
|
||
(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...
|
||
```
|
||
|
||
The `.complete` marker is written last and contains tab-separated key/value rows:
|
||
|
||
```text
|
||
format spectra-reml-block-v1
|
||
block 0
|
||
tasks 256
|
||
beta_elements 4096
|
||
cov_elements 34816
|
||
```
|
||
|
||
Consumers must ignore blocks without `.complete`.
|
||
|
||
## Status values
|
||
|
||
```text
|
||
converged
|
||
converged_boundary
|
||
max_iterations
|
||
line_search_failed
|
||
rank_deficient
|
||
invalid_input
|
||
non_positive_covariance
|
||
numerical_error
|
||
```
|
||
|
||
`converged_boundary` is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.
|
||
|
||
## Generic finalized output
|
||
|
||
The Python CLI exports one TSV row per task. It includes the full summary plus:
|
||
|
||
```text
|
||
beta_json
|
||
covariance_packed_lower_json
|
||
```
|
||
|
||
Project-specific software can attach coefficient names and derive contrasts without changing the numerical engine.
|