Add reusable SpectraREML batch AI-REML engine

docs/FORMAT.md

@@ -0,0 +1,162 @@
+# SpectraREML file contract
+
+All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based.
+
+## Shared sample order
+
+The following matrices must use exactly the same sample order:
+
+1. GRM;
+2. common design matrix;
+3. phenotype matrix;
+4. extra-covariate matrix.
+
+When supplied, `--grm-id` is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order.
+
+## GRM
+
+`--grm-bin` uses the GCTA lower-triangle packed `float32` layout:
+
+```text
+G[0,0],
+G[1,0], G[1,1],
+G[2,0], G[2,1], G[2,2], ...
+```
+
+For `n` samples, the exact file size is `4 * n * (n + 1) / 2` bytes.
+
+## Common design
+
+`--base-x` is a row-major `float64` matrix with shape
+
+```text
+n_samples × n_base_covariates
+```
+
+It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns.
+
+## Phenotypes
+
+`--phenotypes` is a row-major `float64` matrix with shape
+
+```text
+n_phenotype_rows × n_samples
+```
+
+Each task selects one row through `phenotype_row`.
+
+## Extra fixed-effect covariates
+
+`--extra-covariates` is an optional row-major `float32` matrix with shape
+
+```text
+n_extra_covariate_rows × n_samples
+```
+
+Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero.
+
+## Task table
+
+`--tasks` is a UTF-8 tab-separated file with exactly four columns:
+
+```text
+task_index  task_id  phenotype_row  n_extra_covariates
+0           trait_a  0              0
+1           trait_b  1              2
+```
+
+Requirements:
+
+- `task_index` is consecutive and zero based;
+- `task_id` is nonempty and unique;
+- `phenotype_row` is within the phenotype matrix;
+- `n_extra_covariates` agrees with the CSR offsets.
+
+## CSR task-to-covariate mapping
+
+`--extra-offsets` is an `int64` array of length `n_tasks + 1`. It begins with zero and is nondecreasing.
+
+`--extra-indices` is an `int32` array of length `offsets[-1]`. For task `i`, its extra-covariate row indices are
+
+```text
+indices[offsets[i]:offsets[i+1]]
+```
+
+An index must be in `[0, n_extra_covariate_rows)`, and a task cannot reference the same row twice.
+
+## Block output
+
+For block number `KKKKKK`:
+
+```text
+block_KKKKKK.summary.tsv
+block_KKKKKK.beta.f64.bin
+block_KKKKKK.cov.f64.bin
+block_KKKKKK.complete
+```
+
+The summary header is:
+
+```text
+task_index
+task_id
+status
+n_fixed
+n_extra_covariates
+beta_offset
+cov_offset
+sigma_g2
+sigma_e2
+h2
+logL
+iterations
+line_search_steps
+grad_inf
+error
+```
+
+`beta_offset` and `cov_offset` count `float64` elements, not bytes. A negative offset indicates that no estimates were emitted for that task.
+
+The covariance array uses the row-wise packed lower triangle:
+
+```text
+(0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ...
+```
+
+The `.complete` marker is written last and contains tab-separated key/value rows:
+
+```text
+format  spectra-reml-block-v1
+block   0
+tasks   256
+beta_elements  4096
+cov_elements   34816
+```
+
+Consumers must ignore blocks without `.complete`.
+
+## Status values
+
+```text
+converged
+converged_boundary
+max_iterations
+line_search_failed
+rank_deficient
+invalid_input
+non_positive_covariance
+numerical_error
+```
+
+`converged_boundary` is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks.
+
+## Generic finalized output
+
+The Python CLI exports one TSV row per task. It includes the full summary plus:
+
+```text
+beta_json
+covariance_packed_lower_json
+```
+
+Project-specific software can attach coefficient names and derive contrasts without changing the numerical engine.