SEP 52 — One backend shape and a small native seam#
SEP |
52 |
Title |
One backend shape and a small native seam |
Author |
John Travers |
Drafting agent |
Opus 5.5 (2026-11-15; revised after review, 2026-11-15) |
Status |
In progress |
Review status |
Reviewed |
Reviewers |
John Travers (human); GPT-6 (AI) |
Type |
Change |
Created |
2026-11-15 |
Resolution |
Accepted 2026-11-15 by John Travers in PR #850, after the GPT-6 review, decisions D1–D7 and the revision that answered R1–R9. |
Requires |
SEPs 25, 39, 45 and 47 |
Supersedes |
— |
Superseded-by |
— |
Implementation |
Opus 5.5 (AI) for John Travers, through four stacked pull requests (D7): (1) phases 0–1, branch |
Manual pages |
— |
Part I — Proposal#
1. Abstract#
Give numpy_ref and native the same internal structure, and make the C++/Python seam of
native as small as that structure allows. SEP 47 gave both backends one external
contract: prepare returns a prepared execution, and each start opens a RunSession
that owns one trajectory. Behind that contract the two backends are built differently.
Native composes a workspace, engines, marches and a session in C++. The reference is an
inheritance stack: NumpyRefBackend (3,000 lines) on HostDriver (2,600 lines), with
HostSession steering it through halt hooks.
This proposal splits both backends into the same five parts:
the backend’s own lowering;
a workspace that holds the state and does every state-sized operation;
engines, the operations a march asks of the workspace;
marches, which own their controller state;
a session that owns the schedule, the boundary monitor and the events.
Both backends read the same run records, built once by shared code, and HostDriver is
deleted. Two things that no longer earn their cost are removed: the Batch scan executor
with every group path behind it, and the leading axis of one that the trajectory’s arrays
still carry for it, in translation, in the backends and on disk.
The native binding shrinks:
Today |
Proposed |
|
|---|---|---|
Bound names |
about 220 |
a |
Configuration |
about 400 names in 15 to 20 calls |
one |
Record converters |
about 21, hand-written |
one generic converter |
Test-only entry points move to a private _testing module, which exposes native’s engines
through the same Python protocols as the reference’s.
Success is judged by four things:
Numbers: the session suites pass with their cases and bounds unchanged.
Measured changes: each change to the reference’s numbers is measured and accepted case by case.
Coverage: no test or line of coverage is lost except with deleted code.
Size: the measured line counts fall by roughly the estimate of §6.7.
2. Motivation and scope#
Why now. The internal seams were shaped by a history that no longer applies:
several Python backends that shared
HostDriver;a native backend that subclassed it;
a Python loop that drove native one step at a time;
a lockstep batch meant to amortise launch costs on a device.
SEP 43 retired numba_cpu. ADR-077 fixed the backends at two. SEP 47 moved every native run
into its session. What remains is a reference built for a family of subclasses that has
one member, a native binding that still carries the per-step interface the session
replaced, and a batch executor that no longer batches. solprop’s backends are unstable in
0.x (ADR-058), and solprop-native is pinned to solprop’s own version. A restructure now
costs no user anything. Once others contribute backend features, the same restructure
costs every open branch.
The native seam today. The nanobind module solprop_native._core is defined in
native/src/bindings.cpp (4,559 lines), bindings_transverse.cpp and helpers.cpp.
It binds about 220 names:
State, 96 names.31 are on the production path: construction, configuration and a few readbacks.
12 are run metadata and 7 serve the timing probe.
39 are reached only from tests.
7 are reached from nowhere, for example
set_flow_actions,table_evaluate,adams_fused_accept,adams_observeandadams_exponential_observe.
Module functions, 31. 8 are on the production path, 3 serve diagnostics or calibration, 19 are for tests and 1 is unused.
Test-only classes:
Monitor,TransverseEdge,ScriptedMarchandScriptedRkMarch.
Before a session marches, Python makes 15 to 20 configuration calls:
the
Problem, 161 leaf fields in 14 groups;z-table and flux-table uploads;
stage tables, the march policy, controller weights, the flow guard and the low-storage settings;
four monitor and edge bindings;
the session plan.
About 400 distinct names cross. About 21 of the records are dictionaries that the binding converts one string key at a time, and Python builds most of them by hand beside a dataclass for the same thing. Some values cross twice:
the tableau nodes;
low_storage_flows;compensatedandinterpolate.
Counts cross beside the arrays they describe, and C++ checks them again. About 60 % of
bindings.cpp converts or forwards. Real logic sits inside the binding unit itself: the
coupled-flow guard, the march entry points and the lease between state and session. ctest
cannot reach it there. Of the stub _core.pyi, 247 lines are generated; the remaining
~700 are hand-written and already miss ScriptedRkMarch.
The adapter solprop_native.backend still holds the Python engine for the native Adams
history (_ResidentAdams). Its per-step methods have no production caller: the reference
Adams march runs only on numpy_ref, and the native march is compiled (SEP 39). The
runtime decisions, the dense observation at the planes and the Adams start-up are native.
The only C++-to-Python call is the TDSE host source (SEP 49).
The reference today. HostDriver holds:
the Runge–Kutta march, adaptive and replayed;
landing, halts and yields, step planes and natural stops;
the dense hold, the periodic statistics buffer and the stage scalars;
prepare, metadata and snapshots;every boundary-monitor check, the transverse edge and the noise models.
It also defines the HostStepping protocol. NumpyRefBackend overrides 18 of its methods
and supplies the physics. The Adams and low-storage marches (HostAdamsMarch,
HostLowStorageMarch) write the driver’s private fields directly: _z, _state,
_proposal, _schedule_index, _accepted_norms and _in_flight. HostSession owns the
events and receipts, but schedules nothing itself: it sets a halt check and calls
propagate_to. Monitor bookkeeping is interleaved with physics:
the split collar’s sums are kept in
_finish;the quadrature alarm’s latch is kept in
_begin_planeand_on_accepted_step;the TDSE reservation sits on the backend, not the engine.
A boundary warning is raised synchronously inside the driver on the reference, but
reported as a WARNING barrier event by native. The service loop special-cases
HostSession to read its driver (driver/session_loop.py).
Native’s C++ already has the shape this proposal asks of both:
session::Residentis the workspace;rk::Engineandmarch::Engineare the engines, and native’s low-storage step is also anrk::Engine;adams_march.cppandrk_march.cppare the marches;session.cppis the session.
The reference has half of it in Python: the AdamsEngine and LowStorageEngine protocols.
The batch executor today. Batch(size) advances compatible fixed-schedule scan points
“in lockstep inside one backend”. It no longer does what that promises:
On native, a group builds a complete backend, with its own workspace and plans, for every member, and runs one session per member, fusing nothing. It costs
sizetimes the memory ofLocal, which reuses one backend across points, for no gain in speed.On the reference, the lockstep group (
HostGroup,NumpyBatchBackend) fuses only the Runge–Kutta stage sums of its members, and the reference’s speed is not one of its goals.
Neither the GUI, the recipes nor the examples use it. It needs about 3,000 lines across the
driver and both backends: the group protocols, the group service loop and
compatible_groups, the group sessions, the lockstep runner and select_batch. It is also
the only reason for the leading axis of one, which the trajectory’s arrays still carry:
the operator’s value tables in the
PropagationProblem(translate/linop.py,translate/stage.py);the boundary functions that take a four-axis state (
translate/collar_check.py,boundaries.SplitCrossing,monitor.BoundaryMonitor, #618);the reference’s passes and history;
native’s load and readback;
the memory ledgers;
the stored state of a
MarchSnapshot.
n_batch appears on 165 lines in 20 files.
In scope:
Both backends’ internal structure: workspace, engines, marches and session, with the shared run records.
The deletion of
HostDriverand of theHostStepping,BatchBackend,GroupSessionandPreparedGroupprotocols, withBackend.max_batchandBackend.prepare_group.The removal of the
Batchexecutor and every group path, on both backends and in the driver.The removal of the leading axis of one, everywhere it is carried for the batch: translation, the boundary functions, both backends and the stored snapshot.
The native binding: what crosses, how records are converted, the stubs, the test-only module and where binding-side logic lives.
The timing probe’s native form.
Porting or deleting every test that depends on what is removed, under the rules of §7.
Out of scope:
The rest of the public interface:
Simulation, the other scan executors, recipes and the output’s other content.The kernel contract’s numerical content: the route union, the response descriptors and every operator’s values. Only the unit axis of its value tables changes.
RunRequestand theRunSessionand event contract of SEP 47.#110’s split of the integrator from its tolerance, and its per-route pipelines.
A fused batch through native’s stage passes (#356), which would now need its own group interface.
New features and performance work.
Any change to native’s numerics, beyond following the reference where §4 allows it.
3. Context#
The design pages:
backends.md §1, the seam and the table of marches;
architecture.md §9;
kernel-contract.md §3, the value tables; §8, the periodic-statistics hooks; and §§11–12, the compiled marches;
conventions.md §6, the axis names;
the native package’s
native/README.md.
Prior proposals and decisions:
SEP 47: backend-owned sessions; its §6.7 on groups and snapshots; the explicit test interfaces of its §6.8.
SEP 39 and SEP 45: the compiled Adams and Runge–Kutta marches, and their decision-trace tests.
SEP 25: the rounding-level comparison rules.
SEP 29: the helpers’ ownership,
PLAINandNATIVE.SEP 46: calibration’s use of the timing probe.
SEP 49: the TDSE host source. Its §10.1 records the deferral of slimming
numpy_refto a test library, which keeps the reference’s complete runs. This SEP keeps them too. It removes the reference’s group and batch paths, the narrower part §10.1 names, together with native’s, because the executor they serve goes.ADR-058: 0.x stability.
ADR-068: the seam has no version number.
ADR-077: two backends, binary64 only.
Issues:
#110, the step loops and the kernel seam. This SEP decides two of its four items. The step loops become one march per method on each backend, over engine protocols the two backends share. The batch axis is removed, with the lockstep batch. The integrator–tolerance split and the route union stay with #110, which is narrowed to them.
#618, the batch axis left after SEP 47. Fixed in full.
#171, one diverging point fails every member of its
Batchgroup. Closed as moot, with the executor.#828, multi-pass cells cannot use the
Batchexecutor. Closed as moot.#344, the low-storage hook names. Its naming item is fixed: the
_adams_*hooks become the workspace’s operations, with names that fit every march.#107, one host statistics module. Informed: native keeps its statistics in C++ (SEP 47 §6.4), and the reference’s move into its workspace. This SEP recommends closing #107 on that decision.
#356, a fused GPU batch. Changed: there is no group interface left to extend, so the issue records that a fused batch would bring its own, and its measurement still decides whether it is worth one.
The output-schema change that main already carries since 0.2.0
(changelog.d/output-schema-split.breaking.md) means files and scans written by 0.2.0
cannot be resumed by the next release. The snapshot change of §5.4 therefore removes no
resumability that release still has.
4. Theory#
No physics changes. This proposal changes how computations are arranged, not what they compute. Comparisons follow SEP 25: a matched discrete computation is held to a rounding budget fixed before the comparison, and an adaptive run to its accuracy envelope.
Rounding and simplification. The reference may change its numbers where a simplification needs it. The expected first case is the Runge–Kutta engine keeping its trial and committing it itself, where today the driver assigns the state the trial returns. Each such change is recorded in its phase report, and is accepted only on that evidence, never by regenerating a table without it. The report gives:
the computation it changes;
the largest difference found on fixed schedules, against that computation’s SEP 25 budget;
the frozen tables and decision traces it regenerates;
whether native follows it.
A difference above rounding needs a stated cause and the maintainer’s agreement in the report.
Measurements stay where they are. A move of ownership never moves a measurement to a different physical state. The split collar’s removal and its guard occupancies are the case in point. They are reductions of the field after the accepted step’s first band mask and before the temporal split:
the removed power is \(P\,(1 - e^{-\alpha h})\);
the occupancies are judged on \(P\,e^{-\alpha h/2}\);
where \(P\) is the power in each bin before the split, \(\alpha\) the collar’s rate and \(h\) the
step (BoundaryMonitor.split_sums; backends.md §1). Evaluating them on
the damped, re-masked field changes the removal. Reconstructing the earlier field from it
amplifies mask residue into fictitious loss, which
tests/test_monitor.py::test_a_deep_collar_reads_no_removal_from_residue guards against.
The simplification permission above does not extend to such a move.
Native’s session transcribes the reference’s scheduling decisions, so a change to the reference’s march or session bookkeeping is mirrored in C++ in the same phase. The decision-trace tests then hold the two together again.
5. Proposed interface#
Batch leaves solprop.driver.executors. Its users take Local for one process, or
Pool, Queue or Slurm for several, as the scans guide already describes. Nothing else
changes for a user. The other names below are private to the backends or to the binding.
5.2 The engine protocols#
The Python protocols mirror the C++ interfaces one-to-one, every argument included.
class RungeKuttaEngine(Protocol): # rk::Engine; the low-storage step too
def trial(self, z: float, dz: float) -> float: ... # the error norm
def accept(self, dz: float) -> AcceptRecord: ... # finish and commit
def extend(self, z: float, dz: float) -> None: ... # dense stages at a plane
def reading(self) -> BoundaryReading: ... # the committed state's
@property
def in_flight(self) -> bool: ...
class AdamsEngine(Protocol): # march::Engine
def start(self, z: float) -> None: ...
def trial(self, z: float, dz: float, stencil: Stencil) -> NDArray: ...
def accept(self, dz: float, free: int, stencil: Stencil,
kept: Sequence[HistoryEntry]) -> AcceptRecord: ...
def chord_phase(self, z1: float, z2: float) -> float: ...
def step_carrier(self, order: int) -> NDArray: ... # the last trial's rotations
def begin_attempt(self, cost: int) -> None: ... # the TDSE budget
def end_attempt(self, outcome: AttemptOutcome) -> None: ...
def reading(self) -> BoundaryReading: ...
The commit contract:
The Runge–Kutta trial keeps its stages and end state in the workspace. It commits nothing until
accept, and a rejected trial leaves the committed state as it was.The low-storage step is the same engine, with one documented difference. Its trial overwrites the state, the register
A. From the start oftrialuntilacceptreturns,in_flightis true. An exception or a non-finite result in that interval leaves no committed state at that position, so the session ends the run failed with no checkpoint there, as both backends do today. The engine owns the in-flight flag and the finish; the march owns the committed position. The stiffness diagnostic is the engine’s, a field of itsAcceptRecord(NaN on native, as today). The march reads it from there, and the session reads the march’s state, never a private field.The Adams
acceptreceives the retained history entries, oldest first, which the interaction formulation’s transport uses.step_carrierreturns the last trial’s carrier rotations. The march stores them in the accepted step’s record, and the session’s dense observation applies them to the interpolation weights, asadams_march.cppanddense.cppdo today.The dense hold, the dense observation and the snapshot rows are the session’s, as they already are on native. The engines’
hold,restore,observeand snapshot methods leave the Python protocol.The TDSE reservation and the trial’s outcome move from the backend onto the engine, as
begin_attemptandend_attempt.The split collar’s sums. An accepted step’s finish computes them in the workspace, from the masked field before the split (§4). It returns them in its
AcceptRecord, and the session accumulates and drains them:the removal adds over every accepted step since the previous check, and the next boundary reading drains it;
the found occupancies are those of the latest accepted step;
a rejected trial, a dense observation and a probe trial contribute nothing, because none of them finishes a step;
loading a state empties the accumulator, and attaching a monitor, at a launch or a resume, starts it from zero, as each backend does today.
These rules transcribe today’s behaviour, and phase 3 pins them with tests before any ownership moves.
A march holds its own state and touches no other object’s fields.
5.3 The native binding#
_core
build_record() -> dict
configure_runtime(record) # threads, spin, BLAS, as one record
wisdom_export() -> str | None; wisdom_import(text) -> bool
available_device_memory() -> int | None
BackendRefusal, EnvelopeConvergenceError, SessionLifecycleError
schemas() -> dict # every record's fields and array specs
Problem # one record; includes every set-once
# table, guard and geometry
Workspace(problem)
.start(run, host_source=None) -> Session # run: the shared run records
.probe(start) -> Probe # §5.5
.diagnostics() -> dict # metadata, memory, sizing, planning,
# crossed bytes
.close(); .closed
Session
.advance(work_limit, deadline) -> Batch
.acknowledge(sequence)
.export(ticket, recipe, out)
.noise_phase(ticket, z) -> ndarray
.request_stop(); .request_snapshot()
.status() -> Status
.snapshot(ticket) -> Snapshot | None # march state, field and history rows
.records() -> dict # monitor, edge, cap, quadrature peak
.continue_monitor(continuation)
.fail_outside() -> Batch | None
.close()
_core._testing # private; no contract (§6.5)
One Problem, one run record. Everything that is fixed for a workspace crosses in
Problem:
the z, flux and stage tables;
the flow guard’s certificate;
the controller weights;
the low-storage settings;
the monitor’s weights and occupancy;
the edge geometry and its lines.
Everything that belongs to one run crosses in the run record:
the initial field and noise;
the method policy and schedule;
the plane plan and statistics plans;
the monitor configuration;
a resume’s march state and history rows, and its monitor continuation.
Python derives nothing that C++ then derives again. There are three real constraints:
a workspace is reused across the points of a scan;
the TDSE host source is a live Python object;
the CPU execution record’s second calibration stage reads a built workspace’s sizing.
The first two keep start separate from the workspace. The third keeps the rebuild that
exists today. A workspace holds at most one live session or probe; start and probe
refuse while another holds it, through the lease that exists today.
Records by field table. Each C++ record declares its fields once, next to its definition: one line per field. A field gives its name, its member and, for an array:
its dtype;
its rank;
its extents, written in terms of the record’s own counts or another field’s extents (for example the Raman table’s
(n_terms, 2, n_to)).
A single template converts a Python dataclass, or a mapping, into the record and back. It handles scalars, enumerations by name, optionals, nested records, lists of records and arrays. It refuses, naming the field’s path:
a missing or unknown field;
a wrong dtype;
a non-contiguous or overlapping array;
a wrong rank;
extents that disagree with the relation;
a size whose product overflows.
All of these are checked before anything is allocated. A count that an array’s extents give
is read from the array, not sent beside it. The relations it took part in are kept as
extent relations, so a transposed table with the right element count is still refused, as
native/tests/test_backend.py requires today.
Array ownership:
Input arrays are borrowed for the conversion only, with the interpreter lock held, and copied into the record’s own storage. Read-only arrays are admitted, since nothing writes them.
Arrays returned in
BatchandSnapshotown their memory. Each holds a capsule over storage moved out of the session, so it stays valid after acknowledgment, workspace reuse and close, as today’s batch arrays do.Fewer copies is an aim, not a criterion. Removing the full
Problemcopy at construction is expected; any further reduction is measured (§7) before it is claimed.
_core.schemas() exports every table: names, dtypes, ranks and extent relations. One
test holds each Python record to its C++ table, so a renamed field, a changed dtype or a
lost extent relation fails. The Fields<> helper that binds Problem today is that
converter’s first case.
Generated stubs. The whole _core.pyi is generated by nanobind’s stubgen, and the
native gate checks it, as it checks Problem’s part today. The binding gives each record
argument and result a signature naming its Python dataclass, so the stub stays typed.
Bindings marshal; logic lives in C++. The binding unit converts, releases the interpreter lock and forwards. The lease between workspace and session, the coupled-flow guard and the march entry points move into the C++ library, where ctest reaches them. The device refusals stay as they are, the CPU-only TDSE host source among them.
5.4 Arrays without a batch axis#
A trajectory’s arrays have no leading axis of one anywhere:
the state, the reference’s passes and history, and native’s load and readback:
(n_modes, n_pol, n_omega);the operator’s value tables in
PropagationProblemand the stage terms (kernel contract §3);the boundary functions of #618;
the memory ledgers;
the stored state of a
MarchSnapshot.
Changing the snapshot’s layout on disk is an output-schema change. It is made with the next
release’s one schema change, and a snapshot written before it is refused by name, not
squeezed. As §3 notes, that release cannot resume 0.2.0’s files in any case.
conventions.md §6 drops n_batch.
5.5 The timing probe#
Simulation.estimate times one step for front ends; the service’s estimate handler calls
it. SEP 46 calibration, which runs by default on a machine’s first native run, times trials
at several positions across a run from the launch state. It repeats each one identically,
and compares the trials’ fields between candidate thread settings. A session cannot take a
trial at an arbitrary position without propagating there, so the probe stays, on both
backends, with today’s contract:
timing_probe(start)on the backend, andWorkspace.probe(start)on native, loads the start’s field, noise, position and resume flag. Its march history, ledgers and monitor continuation are not read.It is refused while a session holds the workspace.
A Runge–Kutta trial at
(z, dz)commits nothing, andfield()returns an owning copy of its end state.The low-storage exception stays. Its trial advances the discarded registers in place,
field()refuses, and the workspace is closed afterwards.
6. Implementation strategies#
6.1 Layers and the seam record#
The work stays within the backends layer (architecture.md §3), the driver’s scan executors, the native package, and translation’s array layout. The seam has no version number (ADR-068). Every change to a guarded surface updates its guard, with a release-note fragment, in the phase that makes it:
The
PropagationProblemvalue tables lose their unit axis (§5.4). Their frozen descriptor tables intests/backends/test_driver.pyare rewritten in the commit that changes them. No field is added or removed, and no value changes.The
RunRequestand the stage bundles’ scalars are unchanged.The
Backend, engine and session protocols change (§5.1, §5.2). The tests that pin their member lists change with them.Native’s record tables are guarded by the schema test of §5.3 from the phase that adds them.
6.2 Removing Batch and the groups#
This is pure deletion, and comes first among the reference’s changes because it removes a
user of HostDriver’s template methods:
the executor and
compatible_groups;the group service loop in
driver/_scan_batch.py;GroupSession,PreparedGroupandprepare_group;HostGroup,host/group_session.pyandNumpyBatchBackend;select_batchandBatchBackend;native’s
NativeGroupSession;the
group_batchbenchmark.
The scans guide and the API, model-selection, interactive, progress and waveguide pages
that mention Batch are updated. A Removed changelog fragment names the replacement
executors. The tests that exercise groups are deleted with them. A test that used a group
only to reach a member’s behaviour is kept, run singly.
6.3 The reference#
HostDriver is dissolved, not refactored in place:
The Runge–Kutta march becomes a march over
RungeKuttaEngine, with its state in a march object, beside the Adams and low-storage marches. The low-storage march takes the same engine protocol, as on native (§5.2).The scheduling, halts, step planes, dense holds and observation, periodic rows, the monitor and edge checks and the noise reset move into
HostSession. That makes the session the Python counterpart ofsession.cpp, which native’s header already calls a transcription of these functions.NumpyRefBackendkeeps lowering,prepare,metadata,memory_ledger, the probe andclose.Its physics moves into
NumpyWorkspace, together with the operationsHostHistoryandHostLowStoragecall today, as named methods. That is roughly 2,000 lines: stage and source, finish, mask, statistics and the module tables.
The interleaved bookkeeping is pulled out of the physics, without moving any measurement:
the split collar’s sums are computed in the workspace’s finish and accumulated by the session (§5.2);
the quadrature alarm becomes a workspace reading that the session reports, as native’s
quadrature_peakalready is;the TDSE reservation moves onto the engines.
The scripted test drivers become scripted engines under the real marches: today’s
HostScriptedDriver and the trace harnesses’ _HostHarness. That is how native’s scripted
engines already test its marches. The unit axis leaves the passes, the history and the
translation functions in this phase (§5.4).
6.4 The native seam#
The adapter’s configuration code collapses into building one Problem, then one run record
per run. These all go, with the flags that sequence them (_configured, _rk_staged,
_rk_weighted):
the lazy
_configurecalls;the post-construction uploads (
table_upload,adams_stage_tables);the monitor binds (
bind_monitor,bind_occupancy,bind_edge,bind_edge_lines).
The probe becomes native (§5.5). It replaces:
the adapter’s per-step trial path (
_trial_step,_prepare_trial,_score_trial);_ResidentLowStorage.step;_ZTables’ per-position evaluation, and the bindings behind them.
They are removed only once the probe tests below pass on the replacement:
repeated trials;
the owning readback;
the four-evaluation low-storage timing;
the low-storage refusal of
field().
The Adams engine’s adapter shrinks to its construction and resume. Its per-step methods are dead and go, with their bindings.
One diagnostics() record replaces the 12 metadata properties, the memory ledger, the
execution sizing, the planning record and the crossed bytes. It is what
NativeBackend.metadata already assembles.
6.5 Tests#
The tests that call native directly are kept as tests. What changes is the surface they call. The binding’s test-only names fall into seven groups, each with a destination:
Group |
Today |
Destination |
|---|---|---|
Decision traces |
|
|
Fault injection |
|
|
Owners |
|
|
Engine-level parity |
|
|
State-level statistics |
|
One |
Helpers |
|
|
Runtime and accounting |
|
|
_testing is built into the same extension and shipped in the wheel. It promises nothing,
and its users are solprop’s own tests. The installed-wheel smoke test
(native/ci/smoke.py) imports it and runs one of its checks.
On the reference:
about 39 test files call
propagate_todirectly, and 46 call private physics methods;the scripted and trace harnesses subclass
HostDriver;about 31 files exercise groups.
These move to sessions, to workspaces and engines, or to scripted engines, or are deleted with the groups. A test whose only purpose was the old structure is deleted, and its phase report says why. The trace corpora keep their decisions; a corpus is regenerated only under §4’s rule.
6.6 Choices of tool#
nanobind stays.
It already builds the extension against Python’s stable ABI as one
cp314-abi3wheel per platform, tested on 3.14 and 3.15.It is the smallest and fastest-compiling C++ binding library that also handles NumPy arrays with explicit ownership.
It releases and reacquires the interpreter lock for the host source.
Its
stubgencan generate the whole stub.
The binding’s size comes from the interface’s shape, not the library.
pybind11 would be larger and slower, for nothing here.
Cython fits Kokkos’s C++20 templates poorly and adds a language.
A plain C interface called through ctypes or cffi would make the extension independent of Python’s ABI, and usable from another language. It would cost the typed exceptions, array ownership and lock handling that nanobind gives, and move marshalling into Python. The interface proposed here, one record in and a session out, would let a C interface be added later if another front end ever needs one.
The release build should pin nanobind’s version. Today it takes whatever newest release
satisfies >=2.9 at build time.
Field tables, not code generation or reflection. Generating both languages’ records from a schema file would add a build step and a third place for every field. C++26 reflection would remove even the tables, but nvcc, HIP Clang and Apple Clang will not offer it for years. When they do, the tables can be replaced without touching the Python side.
6.7 Cost#
Rough estimates, to be compared with the measured counts at Final:
Area |
Today |
Estimate after |
|---|---|---|
Native binding units (C++) |
about 5,000 lines |
1,500 to 2,000, plus about 400 for the converter with its array specifications |
|
14,154 lines |
10,000 to 11,000, |
|
about 25,600 lines |
19,000 to 20,500 |
Hand-written |
about 700 lines |
none |
Tests |
— |
churn in about 150 files; the net count is not estimated |
backends and the driver lose:
most of
HostDriver;the protocol surface;
the duplicated bookkeeping;
the groups and the batch executor, about 3,000 lines.
They gain the Runge–Kutta march object and a larger session. The axis removal changes many lines and removes few. No dependency is added.
Native’s per-step run time is not expected to change, because the marches are already native. Preparation may get slightly cheaper with fewer calls and copies, which matters only for scans of small cases.
7. Verification and validation#
A pinned corpus first. Before phase 3 changes anything, a corpus is recorded from the current tree on fixed schedules. It covers every march (adaptive and replayed Runge–Kutta, fused and exponential Adams with and without retained-history transport and carrier dense output, low-storage) and every route family, with:
the saved fields;
the boundary-monitor and edge records, the split collar’s removal and occupancies included;
the resume outcomes: a snapshot, a resume and its continuation;
the probe’s trials, readbacks and low-storage refusal.
The restructured reference is compared with it:
bitwise where the order of operations is unchanged, as a diagnostic on one build and thread setting only, never as a cross-build or cross-thread gate (SEP 25);
otherwise within each computation’s SEP 25 budget;
anything beyond that is handled by §4’s rule.
Numbers:
Session suites. Both backends’ session suites pass with their cases and bounds unchanged: lifecycle, resume, snapshot, the boundary monitor and the edge, and every route’s propagation tests.
Matched-discrete parity. Native against the reference keeps its bounds.
Decision traces hold both marches to the same decisions. A regenerated corpus is listed with its cause.
Adaptive runs keep their accuracy envelopes in the existing qualification cases.
The engine contracts are qualified on the cases where they are consequential:
interaction Adams with retained history;
exponential Adams with a carrier frame and interpolated planes;
a low-storage failure during the trial, and during the finish, each ending the run failed with no checkpoint at that position;
the split collar’s removal and occupancies over several accepted steps between checks, with rejected trials, dense observations and a resume between them, with the deep-collar residue test unchanged.
Tests are not lost. Each phase report carries an inventory of every removed binding,
hook, protocol member or group path. It lists the tests that used it, and each test’s
destination: ported, or deleted with a reason. The coverage measured by the existing lanes
does not fall except for deleted code: native’s C++ line coverage and solprop’s Python
coverage of backends/ and driver/. The native C++ tests gain the logic moved out of the
binding unit.
Records and arrays:
The schema test of §5.3 fails when a Python record and its C++ table disagree.
Each record round-trips through the converter.
The malformed-layout refusals of today keep passing: transposed tables, a wrong leading axis and non-contiguous arrays.
So do the lifetime tests of returned arrays after acknowledgment, reuse and close.
The generated stub is checked by the native gate.
Builds and lanes:
CI’s lanes run as they do today: the CPU builds and tests on Linux, macOS and Windows; the coverage lanes; the native quality stages; and the compilation, with their tests, of the CUDA and HIP targets.
The installed-wheel smoke test imports
_testing.No device run is part of this SEP’s qualification. The device builds are qualified by compilation only, and the phase report records that limit. A later device campaign runs the GPU lane on the new seam.
Performance:
Native step time is measured with
python -m benchmarks abon about ten steps or a short--lengthwindow, on a few cases covering each march, before and after phase 4. The hardware, thread settings and repetitions are recorded. A median slowdown above 3 % on a case, beyond its repeat scatter, is investigated and fixed before the phase merges.Native preparation time is measured on a small-case scan.
The probe’s time and peak memory are measured before and after.
The reference’s test-suite wall time should not rise noticeably.
Size: poe size and the binding units’ line counts are measured before and after each
phase. The copy counts and peak host storage of a workspace’s construction are measured
before any reduction is claimed.
8. Documentation#
User manual:
The scans page loses its fixed-schedule batch section.
The API, model-selection, interactive, progress and waveguide pages that mention
Batchare updated.A
Removedchangelog entry names the replacement executors.
Developer pages:
backends.md §1, §3 and the table of marches: one shape, two implementations, and no groups.
architecture.md §9.
kernel-contract.md §3, the value tables’ axes; §8; and §§11–12, for the hooks and names.
conventions.md §6, which drops
n_batch.scans.md and benchmarking.md.
native/README.md: the binding’s rules, namely records by field table, bindings that only marshal,_testingthat promises nothing, and generated stubs.testing.md: the engine-level parity pattern and the scripted engines.
An ADR records the shared shape, the removal of groups and the native seam’s rules.
9. Compatibility and impact#
Batchis removed. A script that imports it fails at import, and the changelog namesLocalandPoolas replacements. On native,Localholds one backend whereBatchheld one per member.Poolwith as many workers holds about whatBatchdid, and runs its points concurrently.The output schema changes once, for the snapshot’s layout (§5.4), in the next release’s one schema change. A snapshot written before is refused by name. Saved fields and records are otherwise unchanged.
solprop.backendschanges its private and protocol surface, which ADR-058 leaves unstable.The
PropagationProblemvalue tables lose their unit axis. Their values and every other field are unchanged. The frozen tables are rewritten under ADR-068’s process.Reference numbers may move under §4’s rule, and each move is recorded.
No deliberate difference from Luna.jl is involved.
10. Alternatives considered#
Shrink the native seam alone. This is the smaller change, and phase 0 does part of it anyway. It leaves two structures for one design. It also leaves the reference unable to share the engine-level parity tests, or to serve as a line-by-line source for native’s session.
Keep
Batchand retire only the reference’s fused group. This was the draft’s recommendation. It keeps an executor that, on native, costssizetimesLocal’s memory for no speed. It also keeps the group protocols, the group service loop and native’s group session, and the unit axis they justify.Keep the private unit axis, or keep it on disk only. It no longer serves anything once groups go. Keeping it on disk would preserve no resumability, because the next release cannot resume 0.2.0’s files in any case (§3).
Slim
numpy_refto a test library, with no session or step loop (SEP 49 §10.1). This was deferred on 2026-11-12 to keep independent complete runs for development, debugging and verification. This SEP keeps them, and gives them the same structure as native’s.Let the reference run Python engines under native’s session. This would make the structures identical by construction. It breaks SEP 47 §6.6: the reference must not call native’s controller to establish the agreement it is compared on.
Drive native’s engines from Python marches. This is the per-step seam SEP 47 removed. It costs a crossing per trial and a second implementation of every march’s bookkeeping.
Time estimates and calibration through a short session, with no probe. A session cannot take trials at arbitrary positions without propagating there, nor repeat one trial identically, both of which calibration needs (§5.5).
Keep dictionaries, with a schema test only. This removes drift but not the ~21 converters or the hand-built dictionaries.
Typed nanobind classes for every record, built by Python. This is what the 2026-11-11 cleanup plan proposed. Stubs then type every record for free. But the reference needs the same records without native installed, so they would exist twice. The Python dataclass stays the one definition, and the field table only describes it on the C++ side.
Problem, which only native reads, stays a bound C++ class.A C interface or another binding library: §6.6.
11. Phases#
The phases land through four stacked pull requests (decision D7, a delivery exception
recorded in SEP 0): phases 0 and 1; phase 2 with phase 3’s axis removal; the rest of
phase 3; phases 4 and 5. Each is based on the one before, and each leaves main usable,
with green required CI, when it merges. Each phase updates the guards and release notes
of what it changes (§6.1).
Phase 0 — native deletions and the test module.
Delete the bindings that nothing reaches, and the dead per-step adapter code (the
_ResidentAdamstrials, accepts and observations). The probe’s paths stay until phase 4.Move the test-only bindings into
_testingand repoint their tests.Add
_testingto the wheel smoke test.Fix the stale docstrings that cite
_ResidentAdamsand_ResidentLowStorage.Pin nanobind in the release build.
Verify: the coverage rule, and the inventory of §7.
Phase 1 — remove Batch and the groups (§6.2), with its documentation and changelog.
Verify: the coverage rule, the inventory, and the single-run suites unchanged.
Phase 2 — shared records, engine protocols and the corpus.
The run records in
solprop.backends.The
RungeKuttaEngineprotocol.AdamsEnginealigned with C++.Native’s adapter builds its dictionaries from the records.
The corpus of §7 recorded.
Verify: unchanged results on both backends, bitwise on one build as a diagnostic.
Phase 3 — the reference restructured, and the axis removed.
NumpyWorkspace, the engines, the marches and the session’s scheduling.HostDriverdeleted.Boundary warnings as events.
The split collar’s rules pinned, then moved.
The unit axis removed from translation, the boundary functions, the reference and native’s adapter, with the frozen tables rewritten and the snapshot layout changed.
The tests ported.
This phase may be split: the axis first, then Runge–Kutta, then Adams and low-storage.
Verify: the corpus comparison, the session suites, the traces, the engine-contract cases, and each numerical change under §4.
Phase 4 — the native seam.
The field-table converter and
schemas().Problemabsorbs the set-once data.Workspace,start(run),probe(start)andSessionas in §5.3.The old probe paths removed once the probe tests pass.
diagnostics().The logic moved out of the binding unit.
The generated stub.
The engine-level parity tests run against both backends.
Verify: the session suites, parity, the record tests, the coverage rule, the compilation of the device targets, the step-time A/B, the probe’s cost and the size counts.
Phase 5 — documentation and the ADR.
The developer pages of §8 not already updated with their phases.
The ADR.
#110, #618, #171, #828, #344, #107 and #356 updated or closed.
12. Open questions#
Q1. Settled by the maintainer on 2026-11-15:
Batchand every group path are removed (§6.2).Q2. Settled on 2026-11-15:
_testingis built into the extension and shipped in the wheel (§6.5).Q3. Which of today’s binding-level oracle tests to replace with C++ tests or session-level tests, not port? This is decided per test in phases 0 and 4. The default is to port.
Q4. Settled on 2026-11-15: the probe stays, as
probe(start)with today’s low-storage exception (§5.5).Q5. Which reference simplifications to take that move numbers? Each is decided in phase 3’s report under §4. The engine holding its trial is the expected first.
13. References#
W. Jakob, “nanobind: tiny and efficient C++/Python bindings,” documentation, https://nanobind.readthedocs.io (accessed 2026-11-15).
M. von Löwis, “PEP 384 — Defining a Stable ABI,” Python Enhancement Proposals (2009). https://peps.python.org/pep-0384/
W. Childers, P. Dimov, D. Katz, B. Revzin, A. Sutton, F. Vali and D. Vandevoorde, “Reflection for C++26,” ISO/IEC JTC1/SC22/WG21 paper P2996 (2025).
Part II — Implementation record#
Milestone 1 (phases 0–1) — 2026-11-16#
Branch feature/sep-0052-backend-shape-p1
(#853), the first of the four stacked pull
requests of D7, started 2026-11-15 by Opus 5.5 (AI) for John Travers. The two phases were
developed in parallel by AI agents on separate branches and merged into the milestone
branch, where the tests both touched were rerun.
Scope. Phase 0, native deletions and the test module (e9631521); phase 1, removal
of Batch and the groups (e52efb20); their merge (3a47dfc2).
Delivered, phase 0.
Deleted bindings nothing reached:
table_evaluate,table_rate,set_flow_actions(with its wrapper) andshutdown_runtime.shutdown_kokkosstays, because the at-exit hook uses it.Deleted the dead per-step Adams adapter:
_ResidentAdams’shold,restore,start,snapshot_rows,fused_trial,fused_accept,exponential_trial,exponential_accept,observeandmarch_stepped, and the four bindings only they reached (adams_fused_trial_at,adams_fused_accept,adams_observe,adams_exponential_observe). No test used them. The C++ behind them stays, because the compiled engines, the dense output and ctest use it. Construction,reset,value,snapshot_nbytesandload_rowsstay, as do the timing probe’s paths (_trial_step,_prepare_trial,_score_trial,_ResidentLowStorage.step,_ZTables), which phase 4 replaces._core._testing: a submodule of the same extension (bind_testinginnative/src/bindings.cpp, withbind_helpersandbind_transversebinding into it). It holds 54 names in the seven groups of §6.5: the decision-trace marches and their runners,adams_march_fault,Monitor,TransverseEdgeand the watch functions, the engine-level parity functions, the state-level statistics, the vector and width helpers, and the runtime probes. State methods became functions that take the state first. Eight more names moved than §6.5 lists, because tests reach them (adams_exponential_trial_at,adams_exponential_accept,adams_start,adams_compensation_start,adams_hold,adams_restore,adams_rows_to_host,table_rate_at).reference_stream_copiesdid not move: calibration uses it.Statewent from 96 bound names to 55.Stub:
_core.pyi(953 → 636 lines) links a hand-written_core_testing.pyi(310 lines). nanobind’sstubgentypes every record argument as a baredict, which the native tests’ strict typing refuses, so the generated stub waits for phase 4’s typed record signatures.native/scripts/problem_stub.py --checknow also compares the_testingstub’s names and members with the built module.Smoke test:
native/ci/smoke.pyimports_testingand checksvector_expagainst NumPy.nanobind pinned to 3.1.0 for the wheel build and the development group.
Stale citations of the deleted adapter fixed in
dense.hpp,dense_output.hpp,engines.hpp,engines.cppanddocs/dev/stepper.md;native/AGENTS.mdstates the_testingrule.
Delivered, phase 1.
Deleted modules:
driver/executors/batch.py,driver/batch.py,driver/_scan_batch.py,backends/host/group.py,backends/host/group_session.py,backends/numpy_ref/batch.pyandsolprop_native/group.py.Removed members:
Backend.max_batch,Backend.prepare_group,BatchBackend,GroupSession,PreparedGroup,check_group,check_member_ticket,start_group,select_batch, the scan’s group hook and its exports, and thegroup_batchbenchmark. Also group-only:events.GroupBatch,ExportTicket.member, the member argument ofsession_core.bind, the native counters’ group keys andStatsSpec.same_tables.Backendnow has exactly the members of §5.1, and its frozen surface intests/backends/test_driver.pyis updated.Documentation: the scans guide loses its fixed-schedule batch section, and
Batchleaves the API page and about 25 guide pages; the developer pages (scans.md,architecture.md,backends.md,kernel-contract.md,stepper.md,benchmarking.md,accuracy.md,conventions.md,testing.md,structure.md) and the regeneratedbackends.svgandlayers.svgno longer describe groups. TheRemovedfragment namesLocal,Pool,QueueandSlurm; three unreleased fragments that only announcedBatchfeatures are deleted.The unit axis is untouched; it leaves in milestone 2.
Tests (§7 inventory).
Phase 0: no test was deleted. Every caller of a moved name is repointed to
_testing, about 30 test files and helpers intests/andnative/tests/. Tests that used the deleted adapter methods now use small helpers over the moved bindings (native_adams_start,native_adams_rows,native_exponential_trial,native_stage_rows,native_rate_weightsintests/sessions/runs.py).Phase 1, deleted whole, group-only:
tests/backends/test_batch_lockstep.py,test_batch_validation.py,test_batch_responses.py;tests/driver/test_batch.py,test_scan_batch.py;tests/sessions/test_group.py,test_group_seam.py,test_native_group.py;native/tests/test_batch.py(95 tests).Phase 1, ported: the scan builder became
tests/builders.py::fixed_fibre_point; the failed-launch probe test moved totest_scan_worker_validation.py; the group’s final warning became an eligibility assertion intest_native_session_monitor.py; the z-table batch test became one single run per fibre against the reference; the monitor’s batch-versus-Localscans run onLocal;test_scans.pyandtest_exponential_march.pylost their batch parametrisation.Phase 1, deleted inside mixed files, each because the group was its subject: the lockstep and refusal tests in
test_exponential_march.py,test_low_storage.py,test_field_free_planes.py,test_moving_guide.py,test_split_block.py,test_general_modal.py(7),test_native_coupled_flow.py,test_scan_stop_progress.py,test_prepare.pyand the native group tests; the batch-equals-members cases intest_optmat_cases.py,test_vector_raman_routes.py,test_coherent_capillary.py,test_envelope_plasma.py,test_vector_raman_output.pyandtest_scan_derived_noise.py; the GPU group test intest_native_session_device.py. The single-run cases of each file stay.
Verification. On an Apple M5 Max, CPU editable build: the directly affected test
modules at four workers (about 3,000 tests across tests/backends, tests/driver,
tests/sessions, the monitor, benchmark and capillary modules, native/tests, the
structure figures, the guide snippets and the changed modules’ doctests), all passing
except native/tests/test_band_plasma.py::test_response_column_boundary_preserves_full_and_tail_windows[19],
a thread-sizing assertion that also fails without this change on that machine; the
tests both phases touched, rerun after their merge; collection of every test; ruff,
basedpyright, the docstring check, codespell, the changelog check, poe docs and
poe size-check. Not run locally: ctest (only comments changed in the C++ library),
clang-tidy, the full and coverage lanes, the wheel build and smoke test, and the two
GPU-marked tests, which run in CI or need a device.
Cost (poe size --all, Python only). Total 271,036 → 266,928 lines: backends
24,834 → 22,753, solprop_native 14,154 → 12,877, driver 27,203 → 26,460 (that
figure also includes an unrelated change merged from main). Binding units:
bindings.cpp 4,559 → 4,459, of which 683 now bind _testing. Tests: about 5,100
lines removed with the group tests.
Deviations. None from Part I. The _testing stub is hand-written until phase 4,
as stated above.
Open questions. None new.
Handoff. Milestone 2 adds the run records and engine protocols, records the
comparison corpus of §7 from this tree, pins the split collar’s rules, and removes the
unit axis. ApertureSet.same_as is kept because tests use it, though production no
longer does.
Milestone 2 (phase 2 and phase 3’s axis removal) — 2026-11-16#
Branch feature/sep-0052-backend-shape-p2
(#854), stacked on milestone 1, by Opus 5.5
(AI) for John Travers. Three parts were developed by AI agents on separate branches and merged in
order: the corpus (39fab778), the run records and protocols (32fe1130) and the axis
removal (e75271e8).
Delivered.
The pinned corpus (§7), recorded from the milestone 1 tree before anything moved:
tests/backend_corpus_cases.py,tests/data/generate_backend_corpus.pyandtests/data/backend_corpus.npz(1.2 MB), compared bytests/backends/test_backend_corpus.py. Thirteen cases, each on both backends: replayed, adaptive and low-storage Runge–Kutta on a fibre; fused Adams with transported history, exponential Adams, and exponential Adams in the carrier frame with interpolated planes on a capillary; envelope plasma; general modal; radial; Cartesian; the line route with a real-space collar under exponential Adams and under low-storage; and the planar hybrid. Each records the last field, the planes and their statistics, the step history, the monitor and edge records (split removal and occupancies included), a stop at the middle plane resumed from its file, and two probe trials or the low-storage refusal. Fixed schedules are held tomarch_bounds.march_bound(SEP 25), statistics to four times that, and adaptive runs to their accuracy envelope; a bitwise comparison is reported as a diagnostic only. Moving a field or the removal by ten budgets is detected in every case. The generator refuses to overwrite the corpus without--force, which its docstring permits only under §4.The split collar’s rules pinned (§5.2), in
tests/sessions/test_split_collar_rules.py, on both backends through sessions, each with a guard that what it rules out exceeds 100 tolerances.The run records,
solprop.backends.records:run_records(problem, request, start)builds aRunRecordsof the method’s policy (RungeKuttaPolicy, lifted fromrk_coreand now carrying the landings;adams_policybuilding theMarchPolicy, formerly derived twice),LowStorageSettings, the schedule, thePlanePlanand theMonitorRecords(MonitorConfig,EdgeConfig,MonitorContinuation). The monitor, edge and notice records moved here fromsolprop_native._watchand_transverse, and the quiet-collar rulecollar_rests, formerly duplicated, lives here once. Native’s session builds every configuration dictionary from the records (solprop_native/_records.py); the reference reads its controller policy, landings, Adams policy, quiet-collar rule and plane bookkeeping from them. What crosses into_coreis unchanged.The engine protocols,
solprop.backends.engines:RungeKuttaEngine,AdamsEngine,AcceptRecord,BoundaryReading,HistoryEntryandAttemptOutcome, as §5.2 states them, with the commit contract and the low-storage difference in the docstrings. The oldAdamsEngineandLowStorageEngineprotocols remain in use until milestone 3.tests/backends/test_engines.pypins the new members and signatures.The unit axis removed everywhere (§5.4, #618): translation’s value tables (
phi_coefficientsis(n_intervals, n_deg, n_modes, n_pol, n_omega)), the separable operators, the stage terms,SplitCrossingandBoundaryMonitor, the reference’s state, passes, history and ledgers, native’s load, readback, chord tables and history rows (in the Python adapter; no C++ changed), the carrier chord, resume, and the storedMarchSnapshot.HOST_SURFACEintests/backends/test_driver.pylosesn_batch; no field was added or removed.conventions.md§6 dropsn_batch, and ADR-003’s status records the change.Output schema 3.
SCHEMA_VERSIONgoes from 2, which 0.2.0 wrote, to 3, the one bump for the next release. A snapshot with a leading axis of one is refused by name (”…the layout (1, …) of a file written before output schema 3…”), and a snapshot of any other wrong shape is refused too, where it used to be skipped with a warning.Changelog fragments:
sep-0052-run-records.changed.md,sep-0052-unit-axis.changed.md(the seam fragment of ADR-068) andsep-0052-snapshot-layout.breaking.md.
Verification. On an Apple M5 Max, CPU editable build: the corpus passes unchanged
after the axis removal, without regeneration, so the layout change moved no number past
its budget; the corpus also passes on the milestone 1 tree, so phase 0 moved none. The
directly affected modules pass: tests/translate (1,420), tests/backends, tests/driver,
tests/processing and tests/recipe (about 4,200), tests/sessions, tests/output,
tests/nonlinear, the monitor, split-crossing and spatial-boundary modules, the decision
traces, and the affected native/tests files, apart from the thread-sizing failure noted
in milestone 1. After the three merges, the corpus, the collar rules, the records,
protocols, frozen tables, traces, snapshot layout and native session-history modules were
rerun together (391 passed). One run under load once failed
test_native_session_history.py::test_a_noisy_run_on_a_z_table_meets_the_reference[input-runge-kutta]
(accepted step ends 2.3e-6 relative against a 1e-6 tolerance); it passed on every rerun
and is likely the multithreaded reductions’ order. Not run locally: the full and coverage
lanes, the docs build, the whole native suite.
Decisions.
The dense-observation rule of §5.2 cannot be exercised: translation refuses dense observation beside the split collar, so the rule holds because the combination cannot run. The refusal is pinned instead.
BoundaryReadingfollows §5.2 and leaves out the split sums, which travel inAcceptRecord; native’s C++ reading still carries them until phase 4.Schema 3, not a number above the pre-0.1.0 development schemas (which reached 28): a development file that says 3 is still refused by its configuration hash and missing fields, though not by name.
The calibration store key keeps its
(1, …)shape, because it is written on users’ disks and changing it would orphan their stored refinements.
Findings.
A native run’s bits depend on what ran before it in the process: at one build and thread setting, a native run after a
numpy_refrun differs in the last bits from one in a fresh process. SEP 25 allows it, but it is worth an issue.
Cost (poe size --all, Python). Total 266,928 → 267,882; backends 22,753 → 24,208
(the records and protocols, mostly docstrings); translate 35,021 → 35,005. Tests: about
2,700 lines added, most of them the corpus and the record and collar tests.
Handoff. Milestone 3 restructures the reference over the protocols and records, and
deletes HostDriver; the corpus is its comparison. HostLowStorage still derives its flow
treatment, HostDriver still owns its row period and step planes, and the reference’s
monitor resumes through its own BoundaryMonitor path: all three go with the driver. The
private-import count is at its cap (87), because records.py reads two private monitor
names.
Part III — History#
2026-11-15: Drafted, from a review of the native binding’s use in production and of the reference’s structure at
main07218ed3. Tracking issue: #849.2026-11-15: AI review by GPT-6 at
9a148c340843769474fff390943cdcbaca3e6f2b, recorded below. Only the review metadata and Part III changed; Part I is unchanged.2026-11-15: The maintainer’s decisions on the review’s questions, and Part I revised in response to every point, by Opus 5.5 (drafting agent); both recorded below. The revision removes the
Batchexecutor and the unit batch axis, keeps the timing probe, and completes the protocol, engine, converter and qualification contracts.2026-11-15: John Travers reviewed the revised proposal and decided D7 (stacked phase pull requests). Accepted by John Travers in PR #850. Part I is frozen.
2026-11-15: Implementation started by Opus 5.5 (AI) for John Travers, on
feature/sep-0052-backend-shape-p1.2026-11-16: Milestone 1 (phases 0–1) reported in Part II.
2026-11-16: Milestone 2 (phase 2 and the axis removal) reported in Part II.
AI review — 2026-11-15#
Reviewer: GPT-6 (AI). Reviewed all of Part I on
PR #850, at
9a148c34,
against the current backend, session, monitor, native binding and test code, SEP 0,
SEP 47 and ADR-068. Read the tracking issue #849 and the six issues listed in §3.
The two existing automated PR findings are acknowledged in R2 and R3 below.
Recommendation: revise before acceptance. The comments below distinguish concrete conflicts with existing behavior from interface questions that the proposal needs to settle. No proposal text, acceptance decision or response to this review is supplied here.
Evidence: source inspection, plus the existing NumPy-reference tests
test_a_deep_collar_reads_no_removal_from_residue,
test_a_timing_probe_commits_nothing_and_reads_back_its_trial and
test_a_low_storage_probe_has_no_trial_field: three passed, their three native cases
skipped because native was not installed. These checks establish the current reference
behavior only. No native build, device run, performance measurement or proposed
implementation was tested.
R1 — High: preserve the split collar’s measurement point (§§4, 5.1–5.2, 6.2). Moving the split sums out of
_finishinto a session reading needs a more precise contract than a reading of the committed state. Insrc/solprop/backends/numpy_ref/backend.py::NumpyRefBackend._finish, the sums are taken after the first hard mask but before the temporal split and its final mask.src/solprop/monitor.py::BoundaryMonitor.split_sumscomputes the removed power asP * (1 - exp(-alpha * h))and the guard occupancies fromP * exp(-alpha * h / 2), withPthe power before the split. Applying that calculation to the damped field changes the removal; reconstructing the earlier field from the final masked field can amplify mask residue into fictitious loss. The existing deep-collar regression intests/test_monitor.pyspecifically guards the latter failure. Native also gathers the sums during its split operation innative/src/general/trial.cpp.Requested change: specify that the workspace captures the required small pre-split reductions during each accepted finish, and that the session owns their accumulation and reporting. State when these values are drained, including several accepted steps between checks, rejected trials, dense observations and resume. Preserve the removal and guard-occupancy tests. Moving ownership must not move the measurement to a different physical state under the general permission to simplify reference arithmetic in §4.
R2 — Medium: independent sessions do not by themselves fix #171 (§§2–3, 6.2, 7–9; Q1). Native already has independent member sessions, but
native/src/solprop_native/group.py::NativeGroupSession._turnmakes a non-boundary terminal failure a group-wide error. The scan consumersrc/solprop/driver/_scan_batch.py::_consumeraisesbatch.error. This preserves SEP 47 §6.7, which explicitly distinguishes a member’s boundary stop from a non-finite norm, step floor, trial exception or in-flight failure that stops the group. ReplacingHostGroupwith the native arrangement therefore leaves #171’s failure policy intact. This also conflicts with leaving the event contract out of scope and requiring all existing session cases to remain unchanged. The automated review identified the same conflict.Requested decision: either leave #171 open and remove the claim that Q1 fixes it, or explicitly include a new failure policy for both backends and the scan consumer. In the latter case, distinguish recoverable member failures from shared executor failures, retain each member’s actual commit and checkpoint eligibility, add the
['done', 'failed', 'done']regression requested by #171, and update the user guide’s batch section. Then §8 cannot promise no manual changes.R3 — Medium: use the current seam-change process (§§6.1, 9, 11).
KERNEL_SCHEMA_VERSIONno longer exists. ADR-068 and kernel-contract.md explicitly removed the counter and require the affected frozen surface tests and a release-note fragment to change with the seam. The proposed bump cannot be performed as written. The automated review also reports this.Requested change: replace all three bump instructions with the current process. Distinguish unchanged scientific descriptor fields from changed protocol and engine surfaces; preserve the former’s frozen cases and update the latter’s guards in the phase that changes them, rather than postponing the seam record to phase 4.
R4 — Medium: retain or replace the live backend facts omitted from the new surface (§5.1). The stated list of what remains on
Backendomitsrequest,nameandreads_transverse_edge. These have production consumers outside the host driver:driver/run.py::run_problemanddriver/_scan_run.py::launch_statisticsreadbackend.request, anddriver/session_loop.py::_refuse_unread_collarreads the capability and name. RemovingHostSteppingdoes not remove these callers.Requested change: make the proposed protocol inventory complete, including its properties and attributes. Retain these facts or name their replacement and the caller migration. In particular, keep the requested transverse-collar refusal before the first step, and explain how a reused backend exposes the request its buffers and plans were prepared for.
R5 — Medium: complete the engine and commit contracts (§§5.1–5.2, 6.2, 6.4). The sketches are described as one-to-one C++ interfaces, but omit consequential parts of the current ones.
march::Engine::acceptinnative/include/solprop_native/adams_march.hppalso receives the retained history entries.ResidentEngine::acceptuses their rows, oldest first, to transport the interaction-formulation history.step_carrier(order)supplies the last trial’s rotations for dense output. Neither transfer appears in the proposed protocol or in the description of what moves to the session.Low-storage native execution is already an
rk::Engine, withtrial,accept,readingandin_flight, rather than juststep. Its trial overwrites the state; an exception or non-finite result before finish leaves a poisoned terminal with no current checkpoint. The reference currently records that state inHostLowStorageMarch.replay_step, on the driver being deleted. The general claim that an engine keeps an uncommitted trial is not true for this method.Requested change: define the retained-history and carrier transfers, whether through explicit arguments or a documented derivation from the trial record. Decide whether low-storage also uses the Runge–Kutta engine protocol on Python, or state its separate commit and failure contract. Name the owner of the in-flight flag, the finish, the committed position and the stiffness diagnostic, and the session’s means of reading their result without private-field access. Include interaction Adams with retained history, carrier dense output, and low-storage failure during trial and finish in the engine-contract qualification.
R6 — Medium: specify how the new probe loads a run, and preserve the low-storage exception (§§5.3, 6.3, 11; Q4). The proposed
Workspace.probe()has no run or start argument, while the initial field and noise are assigned tostart(run). Where does a probe on a freshly built workspace obtain its launch, resume position, schedule cursor and noise policy without opening a session? TodayNativeBackend.timing_probe(start)explicitly loads these through_load.Also, the blanket promise of an uncommitted trial and field readback does not describe low-storage probing.
TimingProbeinbackends/protocol.pyexplicitly permits its discarded registers to advance in place, requires that backend to be closed afterwards, and refusesfield(). The corresponding test intests/driver/test_estimate.pypassed on the reference during this review.Requested change: give the probe an explicit load path and an ownership rule excluding a live session. Preserve the low-storage exception, or explicitly propose and account for the additional storage or reload needed to change it. Keep repeated-trial, owning-readback and four-evaluation low-storage timing tests, and make removal of the active
_ResidentLowStorage.stepand_trial_stepprobe paths depend on their replacement; phase 0’s deletions must not remove them early.R7 — Medium: the generic converter needs an array contract beyond names and C++ member types (§§5.3, 6.3–6.6, 7). A name and member pointer identify a
std::vector<double>, but not the rank and related extents of the array it represents. The presentFields<>already carries extra rank metadata and keeps shapes for validation. For example,native/tests/test_backend.pyrefuses transposed Raman tables with the correct element count and fixed-term tables whose leading axis is wrong. Removing redundant counts is compatible with those checks only if the proposal names the authoritative shapes and the relationships still checked before allocation. A schema test comparing names and types alone will not detect a loss of those checks.The promised reduction from three copies to one also needs an ownership decision. The present
exact_vectorcopies array data into a record; changing how records are described does not by itself remove that copy or the later device upload. ReturnedBatchandSnapshotarrays must remain valid after acknowledgment, workspace reuse and close. Nanobind’s array ownership rules require an explicit owner for a view; they do not make session scratch into an owning snapshot.Requested change: specify array dtype, rank, contiguity, read-only admission, related-shape validation and size-overflow checks, and say which inputs are borrowed during conversion versus copied into persistent storage. Specify owning returned arrays and the GIL/lifetime boundary for any borrowed Python memory. Keep the malformed-layout and lifetime regressions, add record round-trip checks, and measure copy counts and peak host/device storage before making the one-copy claim an acceptance criterion. Include how the generated stubs retain useful record and array types when the binding accepts generic dataclasses or mappings.
R8 — Medium: clarify whether private array layouts change (§§2–3, 6.1–6.2, 6.6, 9). §3 says the single-trajectory batch axis leaves the backends, but the concrete scope and phases mainly remove
max_batchand the fused reference group. Those are different changes.NumpyRefBackend, its pass functions and native’s load/readback adapters currently use a private leading axis of one; conventions.md §6 permits that. SEP 47 §6.7 separately requires the leading axis in existingMarchSnapshotpayloads to remain readable.Requested decision: retain the private layouts and narrow the claim, or explicitly include their conversion, saying which Python arrays and C++ layouts change. In the latter case, account for the pass, transform, test and memory-ledger work in the phases and cost estimate, and preserve the snapshot serialization boundary with refusal of a non-unit stored axis. Removing a protocol attribute alone does not settle this scope.
R9 — Medium: make qualification match the reach of the binding change (§§6.4–6.6, 7, 11). The new records, moved bindings and generated stubs affect all native builds, yet §7 does not distinguish CPU, CUDA and HIP qualification or say which wheel tests exercise the shipped
_testingmodule. Passing a CPU session suite does not check device uploads, resident histories or event/export lifetimes after their binding changes. The existing device refusals, including the CPU-only TDSE host source, must also survive.Requested change: name the required build and execution lanes, the device witnesses or explicitly recorded limits, and the installed-wheel import and
_testingchecks. Pin the pre-change comparison corpus before restructuring it, including representative monitor records, resume outcomes and probe behavior as well as fields. Clarify phase 1’s bitwise requirement: it can be a diagnostic for an unchanged computation in a controlled configuration, but should not become a cross-build or cross-thread numerical gate contrary to SEP 25. Specify that the performance A/B uses about ten steps or a short length window, with hardware, thread settings, repetitions and a regression decision rule recorded; “a few release cases” must not select full-length release tracking runs. Include the native probe’s time and memory cost, since it is being replaced even though ordinary native steps are expected to be unchanged.
Maintainer decisions — 2026-11-15#
Decided by John Travers on questions the review raised, each put to him with its context and the drafting agent’s recommendation.
D1 (R2, Q1): remove
Batchand every group path. Native’s group builds a backend for every member and fuses nothing, so the executor costssizetimesLocal’s memory for no speed. The reference’s fused stage sums serve only the reference’s speed, which is not a goal. Nothing in the GUI, the recipes or the examples uses it. #171 and #828 close as moot.D2 (R8): remove the unit batch axis everywhere, on disk too. This covers translation’s value tables, the boundary functions of #618, both backends, the memory ledgers and the stored snapshot. The snapshot’s layout changes with the next release’s one output-schema change. That release cannot resume 0.2.0’s files in any case, because of the schema change already on
main.D3 (R6, Q4): keep the timing probe as
probe(start), with today’s low-storage exception. SEP 46 calibration, which runs by default on a machine’s first native run, needs trials at arbitrary positions from the launch state, repeated identically, with their fields compared. A session cannot provide that.D4 (R7, §10): run records stay Python dataclasses, described in C++ by field tables. This departs from the 2026-11-11 cleanup plan, which bound the C++ structs as nanobind classes, because the reference needs the same records without native installed.
Problemstays a bound C++ class.D5 (R9): device builds are qualified by compilation only. This SEP makes no device run. Its phase report records the limit, and a later device campaign runs the GPU lane.
D6 (Q2):
_testingships in the wheel, built into the same extension, so the installed-wheel jobs run the native oracle tests against what users install.D7 (§11): the implementation merges through four stacked pull requests, one per group of phases, each based on the one before, rather than one pull request. The restructure touches about 150 test files and runs in parallel with SEP 51; stacking keeps each review coherent and lets work continue up the stack while CI runs once per milestone. SEP 0 records the exception.
Response to review — 2026-11-15 — Opus 5.5#
Every point was checked against the code at main 281a9543 before answering. The review is
right on each; R2, R6 and R8 also led to the maintainer’s decisions above, which go further
than the review asked.
R1 (split collar’s measurement point): accepted.
NumpyRefBackend._finishtakessplit_sumson the masked field before the temporal split. Native gathers the same sums during its split, and both zero the accumulator when a monitor is bound and drain it at a reading. The changes:§4 now states that a move of ownership never moves a measurement, with the formulae and the deep-collar test.
§5.2 has the workspace’s finish compute the pre-split sums and return them in its
AcceptRecord. It states the session’s accumulation and drain rules for several accepted steps between checks, rejected trials, dense observations, probe trials, loads and resumes, as transcriptions of today’s behaviour.§6.3 and §7 pin those rules with tests before ownership moves, and keep the removal, occupancy and residue tests.
R2 (independent sessions do not fix #171): accepted, and superseded by D1. The draft’s claim was wrong: native’s group ends the group on a non-boundary member failure, as SEP 47 §6.7 requires. Rather than fix the policy, the maintainer removed the executor. The changes:
§§2, 3, 5, 6.2, 8, 9 and 11 now remove
Batchand every group path.#171 and #828 close as moot, and #356 is told it would need its own group interface.
§8 now lists the manual pages that change.
R3 (the seam has no version number): accepted.
KERNEL_SCHEMA_VERSIONwas removed by ADR-068, and the draft’s three bumps are gone. §6.1 now follows ADR-068’s process:the value tables’ axis change rewrites
test_driver.py’s frozen tables in the same commit;RunRequestand the stage bundles’ scalars are unchanged;the protocol, engine and record guards change in the phase that changes them, each with a release-note fragment, not in a final documentation phase.
R4 (live backend facts): accepted. §5.1 now lists the whole
Backendprotocol with its attributes and consumers. It keeps:name;reads_transverse_edge, for the refusal before the first step;request, which a reused backend’s runs are checked against;the
helpersclass attribute.
R5 (engine and commit contracts): accepted.
§5.2’s Adams
accepttakes the retained history entries, oldest first.step_carrier(order)is in the protocol, and the carrier rotations’ path to the session’s dense observation is stated.The low-storage step uses the Runge–Kutta engine protocol on both backends, as it already does on native. Its destructive trial, the in-flight interval and the failure with no checkpoint are documented as one stated difference.
The in-flight flag and the finish are the engine’s; the committed position is the march’s; the stiffness diagnostic travels in the
AcceptRecord.§7 qualifies interaction Adams with retained history, carrier dense output, and low-storage failure during the trial and the finish.
R6 (the probe’s load path): accepted, with D3. §5.5 gives the probe an explicit
probe(start)load, the same astiming_probe(start)today. It is refused while a session holds the workspace, and keeps the low-storage exception. §6.4 removes the old probe paths only after the repeated-trial, owning-readback, four-evaluation low-storage and refusal tests pass on the replacement. Phase 0 no longer deletes them.R7 (the converter’s array contract): accepted. In §5.3 a field table’s array entry carries its dtype, rank and extent relations. The converter refuses, before any allocation and naming the field:
a wrong dtype or rank;
a non-contiguous or overlapping array;
extents that disagree with the relation;
a size whose product overflows.
So the transposed-table refusals survive the removal of redundant counts. Inputs are borrowed under the interpreter lock and copied into the record; read-only arrays are admitted; returned arrays own their storage through capsules. The one-copy claim is withdrawn as a criterion and replaced by a measurement (§7). The schema test now covers dtypes, ranks and extent relations, records round-trip, and the stubs name the dataclass types through signatures.
R8 (private array layouts): accepted, and widened by D2. The draft’s claim and its phases disagreed. The axis is now removed everywhere (§5.4), with the frozen tables, the passes, the adapters, the ledgers and the snapshot layout changed in phase 3. The snapshot’s on-disk change is an output-schema change, and an older snapshot is refused by name, never squeezed. §6.7 notes that the axis removal touches many lines and removes few.
R9 (qualification): accepted, with D5 for devices. §7 now:
names CI’s lanes;
adds
_testingto the installed-wheel smoke test;records that the device builds are qualified by compilation only, as a stated limit;
pins a corpus (fields, monitor and edge records, resume outcomes and probe behaviour) before phase 3;
keeps phase 2’s bitwise comparison as a single-build diagnostic, not a gate;
specifies the step-time A/B as about ten steps or a short window, with hardware, threads, repetitions and a 3 % median rule recorded;
adds the probe’s time and memory.
The automated findings on PR #850: the schema-version bump is answered by R3, the failure policy by R2, the pre-split measurement by R1, the probe’s load path by R6, and the carrier transfer by R5.