Unify FFN and TrajMixer model architectures
This commit is contained in:
64
MODEL_ARCHITECTURES.md
Normal file
64
MODEL_ARCHITECTURES.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Model architectures
|
||||
|
||||
DeepHealth uses one codebase for both supported history-block architectures.
|
||||
Select the architecture explicitly when starting a training run:
|
||||
|
||||
| `model_architecture` | History block | Checkpoint fingerprint |
|
||||
| --- | --- | --- |
|
||||
| `transformer_ffn_v1` | Temporal attention + SwiGLU FFN | `blocks.*.mlp.w1/w2/w3` and `blocks.*.ln2` |
|
||||
| `traj_mixer_v5` | Temporal attention + TrajMixer | `blocks.*.mlp.intra_*`, `gate_proj`, and `output_proj` |
|
||||
|
||||
`transformer_ffn_v1` is the CLI default; pass `traj_mixer_v5` explicitly for
|
||||
TrajMixer runs.
|
||||
|
||||
## Training
|
||||
|
||||
Next-step example:
|
||||
|
||||
```powershell
|
||||
python train_next_step.py --model_architecture traj_mixer_v5 --n_layer 12
|
||||
```
|
||||
|
||||
All-future example:
|
||||
|
||||
```powershell
|
||||
python train_all_future.py --model_architecture transformer_ffn_v1 --n_layer 12
|
||||
```
|
||||
|
||||
New runs are separated by architecture:
|
||||
|
||||
```text
|
||||
runs/
|
||||
transformer_ffn_v1/
|
||||
<run_name>/
|
||||
traj_mixer_v5/
|
||||
<run_name>/
|
||||
```
|
||||
|
||||
Use `--runs_root` to place this structure under a different root. Existing run
|
||||
directories are not moved or renamed.
|
||||
|
||||
Each generated `train_config.json` records `model_architecture`, total parameter
|
||||
count, and trainable parameter count.
|
||||
|
||||
Both training entry points use the single `--n_layer` option to set the number
|
||||
of history backbone blocks. The same value is passed to `DeepHealth.n_layer`
|
||||
and saved as `n_layer` in `train_config.json`; it must be at least 1.
|
||||
|
||||
## Architecture validation
|
||||
|
||||
Evaluation resolves the architecture before constructing the model and always
|
||||
loads weights with `strict=True`.
|
||||
|
||||
- Every config must include an explicit `model_architecture` marker.
|
||||
- Checkpoint fingerprints are used to validate that the selected architecture
|
||||
matches the stored weights.
|
||||
- A config marker that conflicts with the checkpoint fingerprint raises an
|
||||
error instead of silently choosing one architecture.
|
||||
- Unsupported historical TrajMixer markers such as `traj_mixer_v2`,
|
||||
`traj_mixer_v3`, and `traj_mixer_v4` are rejected.
|
||||
- Checkpoints and configs created before architecture markers were introduced
|
||||
are intentionally unsupported.
|
||||
|
||||
Project code should use the architecture factory rather than instantiate a
|
||||
history block directly.
|
||||
Reference in New Issue
Block a user