# SpectraREML file contract All raw binary files are little-endian, headerless, and contiguous. Integer indices and element offsets are zero based. ## Shared sample order The following matrices must use exactly the same sample order: 1. GRM; 2. common design matrix; 3. phenotype matrix; 4. extra-covariate matrix. When supplied, `--grm-id` is checked for the expected number of nonempty rows. Domain adapters remain responsible for verifying the actual identifiers and order. ## GRM `--grm-bin` uses the GCTA lower-triangle packed `float32` layout: ```text G[0,0], G[1,0], G[1,1], G[2,0], G[2,1], G[2,2], ... ``` For `n` samples, the exact file size is `4 * n * (n + 1) / 2` bytes. ## Common design `--base-x` is a row-major `float64` matrix with shape ```text n_samples × n_base_covariates ``` It must already contain every common fixed effect, including an intercept if required. SpectraREML does not add or standardize columns. ## Phenotypes `--phenotypes` is a row-major `float64` matrix with shape ```text n_phenotype_rows × n_samples ``` Each task selects one row through `phenotype_row`. ## Extra fixed-effect covariates `--extra-covariates` is an optional row-major `float32` matrix with shape ```text n_extra_covariate_rows × n_samples ``` Only rows referenced by at least one task are read and rotated. The file may be omitted when the row count and all task-specific counts are zero. ## Task table `--tasks` is a UTF-8 tab-separated file with exactly four columns: ```text task_index task_id phenotype_row n_extra_covariates 0 trait_a 0 0 1 trait_b 1 2 ``` Requirements: - `task_index` is consecutive and zero based; - `task_id` is nonempty and unique; - `phenotype_row` is within the phenotype matrix; - `n_extra_covariates` agrees with the CSR offsets. ## CSR task-to-covariate mapping `--extra-offsets` is an `int64` array of length `n_tasks + 1`. It begins with zero and is nondecreasing. `--extra-indices` is an `int32` array of length `offsets[-1]`. For task `i`, its extra-covariate row indices are ```text indices[offsets[i]:offsets[i+1]] ``` An index must be in `[0, n_extra_covariate_rows)`, and a task cannot reference the same row twice. ## Named shared fixed-effect contrasts Optional arbitrary hypotheses use two files. `--contrast-matrix` is a row-major `float64` matrix with shape `n_contrasts x n_base_covariates`; its columns follow `base_x` exactly. Task-specific extra fixed-effect columns are assigned zero weights automatically. `--contrast-metadata` is a UTF-8 TSV: ```text contrast_index contrast_id rhs 0 group_a_b 0 ``` Indices are consecutive and zero based, IDs are unique, and every matrix row must contain a nonzero weight. Each row tests `L beta = rhs` with the method selected by `--fixed-effect-test` (Satterthwaite or Kenward-Roger). ## Block output For block number `KKKKKK`: ```text block_KKKKKK.summary.tsv block_KKKKKK.beta.f64.bin block_KKKKKK.cov.f64.bin block_KKKKKK.fixed_se.f64.bin block_KKKKKK.fixed_stat.f64.bin block_KKKKKK.fixed_ddf.f64.bin block_KKKKKK.fixed_p.f64.bin block_KKKKKK.contrast_estimate.f64.bin block_KKKKKK.contrast_se.f64.bin block_KKKKKK.contrast_stat.f64.bin block_KKKKKK.contrast_numdf.f64.bin block_KKKKKK.contrast_ddf.f64.bin block_KKKKKK.contrast_p.f64.bin block_KKKKKK.complete ``` The summary header is: ```text task_index task_id status n_fixed n_extra_covariates beta_offset cov_offset sigma_g2 sigma_e2 h2 logL iterations line_search_steps grad_inf fixed_test_method fixed_test_status fixed_test_offset contrast_test_offset extra_joint_num_df extra_joint_den_df extra_joint_f extra_joint_p fixed_test_error error ``` `beta_offset`, `cov_offset`, and `fixed_test_offset` count `float64` elements, not bytes. A negative beta/covariance offset means that no estimates were emitted. A negative fixed-test offset means that no valid coefficient-wise fixed-effect tests were emitted; successful REML estimates are retained even when inference fails. The covariance array uses the row-wise packed lower triangle: ```text (0,0), (1,0), (1,1), (2,0), (2,1), (2,2), ... ``` The `.complete` marker is written last and contains tab-separated key/value rows: ```text format spectra-reml-block-v4 block 0 tasks 256 beta_elements 4096 cov_elements 34816 fixed_test_elements 4096 contrast_test_elements 1280 ``` Consumers must ignore blocks without `.complete`. ## Status values ```text converged converged_boundary max_iterations line_search_failed rank_deficient invalid_input non_positive_covariance numerical_error ``` `converged_boundary` is a successful residual-only solution accepted after the one-sided variance-component score and likelihood checks. ## Fixed-effect inference `--fixed-effect-test` selects `satterthwaite` (the default), `kenward-roger`, or `none`. The four `fixed_*` arrays have the same offsets and coefficient order as `beta`. They contain standard errors, statistics, denominator degrees of freedom, and p-values. With ordinary least-squares or Satterthwaite inference, a coefficient statistic is a signed t statistic. With Kenward-Roger inference, it is an F statistic with one numerator degree of freedom. For every task with extra covariates, the summary also contains a test of the joint null that all task-specific fixed effects are zero. The six `contrast_*` arrays share `contrast_test_offset` and follow the contrast metadata order. A one-row ordinary least-squares or Satterthwaite contrast reports a signed t statistic; Kenward-Roger reports an F statistic with one numerator degree of freedom. All methods retain the signed `L beta` estimate and their standard error. Negative `contrast_test_offset` means that no valid contrast tests were emitted for that task. `fixed_test_status` is one of: ```text not_requested ok boundary_ols line_search_conditional fit_not_converged invalid_contrast information_singular numerical_error ``` At `converged_boundary`, the random term is removed and the fixed model is refitted as ordinary least squares. `fixed_test_method` is `ordinary-least-squares`, `fixed_test_status` is `boundary_ols`, and the denominator degrees of freedom are `n-rank(X)`. At `line_search_failed`, the last retained REML iterate is tested with the requested Satterthwaite or Kenward-Roger method and `fixed_test_status` is `line_search_conditional`; the optimizer flag remains visible in `status`. Phenotypes are scaled internally by their task-specific OLS residual RMS before optimization. Reported fixed effects, fixed-effect covariance, variance components, and restricted likelihood are transformed back to the original input phenotype units. Input binary files are never modified. The strong-Wolfe zoom uses safeguarded quadratic interpolation with a 2% endpoint margin and falls back to bisection. At an evaluation limit or a collapsed bracket, the last valid likelihood-improving point encountered is accepted. `line_search_failed` therefore means that the search found no valid point that improved the starting likelihood. ## Generic finalized output The Python CLI exports one TSV row per task. It includes the full summary plus: ```text beta_json covariance_packed_lower_json fixed_effect_se_json fixed_effect_statistic_json fixed_effect_denominator_df_json fixed_effect_p_value_json contrast_estimate_json contrast_standard_error_json contrast_statistic_json contrast_numerator_df_json contrast_denominator_df_json contrast_p_value_json ``` Contrast JSON arrays follow `contrast_index`; IDs and right-hand sides are stored in the manifest-referenced contrast metadata TSV.