Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Design review: blueprint revision 5 contract amendment

1. Decision and scope

Decision: Accept for the bounded proposed-design scope below. The revised contracts address R4-01–R4-13 and assign a concrete consumer or deferred trigger to L1–L9. Acceptance means the reviewed mechanisms, boundaries and refusal paths are specified. It does not certify their implementation, scientific validity or performance, and does not change any ADR to accepted.

Proposal: blueprint revision 5, ADRs 0039–0048 and their capability-map/register reconciliation. Standard: Data Model–Based Design Charter, gates G1–G7. Author/reviewer: Codex, including review of its own amendment; this is not an independent second-person review. Date: 2026-09-13. Evidence: Proposed architecture, selected Interface-checked library mechanisms and the explicitly bounded historical Tested characterizations in the revision-4 review.

Observable outcome: implementers have one declared route for admitting physical meaning, resolving demand, binding complete stage inputs, handling guarded failures, and preserving content through alternate encodings. Existing Arrow/DataFusion operators perform their standard jobs; domain rules remain in the platform contracts that invoke them.

Method and coverage. Inspected the changed semantic contracts in blueprint §§4–9, 14, 18, 20–21 and 24–26, their declarations in §6, affected decisions and glossary, all new ADRs and their supersession boundaries, and the four capability maps’ current recommendations. Replayed the earlier review’s counterexamples against the proposed rules by inspection. Followed the concrete journeys in §5; checked section/port ownership, membership framing, numerical/failure policy and the implementation handoff. Static validation is recorded in plan 02.

This is a follow-up on the changed contracts. Unchanged thermodynamic correlations, every unit template, the full IDAES coverage matrix, all specialized diagnostics and later-phase solver/provider routes were not re-certified. No product compiler, solver, canonicalizer or Python adapter was implemented or executed by this amendment. The retained probe receipts were not regenerated. Context7 returned broad analyzer/runtime/reader material during reconciliation; the pinned source/probe evidence in the earlier review remains the basis for precise library claims. Unbuilt behavior is outside this acceptance claim, rather than silently being scored as tested.

The reviewed blueprint SHA-256 is 63629cd7221d7083f95b7e39f916cc6a6d3395599a2faadf5674e98c175c1a5b. Dependency pins/features, both lockfiles, the toolchain, charter and original probe source are unchanged; Cargo.toml has only a citation-comment amendment.

Supported limits: local single-writer publication; explicitly admitted Arrow layouts; the row/byte/offset envelope in blueprint §5.3; complete stage input keys; pure contracted kernels; and only numerical/backend routes whose actual implementation and conformance binding is available. Larger canonical streams, finer dependency tracking, Parquet optimization, coalescing and preimages retain register triggers. The design supplies rejection paths for unsupported cases.

2. Authority and lifecycle map

ConceptIdentity and authorityRevision boundary / permitted updateDerived representations
Relation and physical quantity meaningRegistry declarations, semantic IDs and complete quantity-operation rules; blueprint §4/§6.2/§8Versioned contract/package amendment admitted through P2Arrow fields/views, recursive validators, analyzer checks and generated adapters
Model / caseStable authored identity; separate snapshot classes and explicit model/overlay parentsChange set and atomic P2-valid commit; compiler cannot write authored/reference factsNormalized seeds, indexed/scalar math and case-bound views
Stage inputs and resultsNamed input/output ports; one producer per portComplete immutable output snapshot after postconditions; an earlier stage is never overwrittenCatalog handles, physical encodings, memo entries and attempt sidecars
Kernel and numerical policyOne KernelSpec, implementation/derivative bindings and selected versioned numerical policyNew contract or selected case policy changes dependency inputsUDF, native, exported and Pyomo bindings that pass the declared conformance scope
Logical / encoded identitypse-ids; canonical v2 relation/snapshot preimages versus finished-byte checksumsNew logical meaning changes content identity; alternate valid encodings can share that identityPhysical manifest, checksum-named objects and conditional ref
Evidence and failureRequired typed outcomes, terminal attempt/run state and noncanonical diagnostic objectsImmutable outcome publication; mutable workspaces remain run-ownedEXPLAIN/protobuf, structured diagnostics and provenance queries

Opaque behavior: specialized kernel bodies, numerical workspaces, solver callbacks and storage I/O remain ordinary code behind the declared contracts. Registration or a function signature cannot supply a missing derivative, conditional implementation or physical conversion.

Renames preserve semantic IDs while changing label-bearing relation content. Row/batch order, supported dictionary encodings and hidden null payload do not change declared logical identity. Changed semantic parents remain visible even when an unchanged relation shares storage; initial reuse does not discard lineage.

3. Semantic contracts and invariants

ContractRepresentation and boundaryDefined failureEvidence / decisive fixture
Active semantic admissionGenerated recursive validator at provider/bundle, pre-rewrite, analyzer, post-optimizer and output boundaries; blueprint §4.4schema.* or head mismatch before useProposed; E1 refutes passive registration; semantic_admission_paths
Complete quantity and head conversionQuantity-operation selection/derivation, exact types by default and checked explicit conversions; §8.3/§14.2Unsupported/ambiguous composition or lossy/incompatible head rejectedProposed; E3 plus head_conversion_exactness, quantity_composition
Decidable stage ownership and demandP3 seeds, finite bound scopes, P6 closure and port-bound P7–P10 bundles; §9.6/§14.1Unknown/ambiguous demand, undeclared port or late realization demand failsProposed; stage_bundle_graph, demand_seed_closure
Guarded operations and outcomesActual CASE/branch regions, complete kernel outcome relation and selected numerical policy; §7.3/§18Excluded branches do not execute; selected invalid operations return typed failure; unsupported routes refuseProposed; E2/E6 plus actual backend conformance
Stable content with checked physical envelopesRecursive canonical copy, explicit metadata/validity/framing/membership and separate checksums; §5.3/§20Unknown layout/version, size excess, incomplete member or corrupt encoding rejectedProposed; E4/E5 are counterexamples, not a v2 implementation proof
Complete reuse and accounted resourcesAll named inputs/absence/policies/binaries; shared runtime with fallible reservations; §14.3Changed key cannot hit; failed reservation cannot publish a partial successful stageProposed; incremental/clean and concurrent resource fixtures

Missing measurements remain nullable values. Invalid evaluation is a typed error, or a mandatory per-coordinate outcome in the separate tolerant operation; a null alone cannot stand for both. An absent optional input is part of the cache key. An empty required output is explicit, while a failed output is never an empty successful bundle. Unknown/conflicting rule facts remain outside decided heads.

Equivalence is role-specific: canonical content identity, physical checksum integrity, full values/multiplicities for pushed scans, guarded numerical outcomes under the selected absolute/relative/signed-zero policy, and attributable decoded meaning for noncanonical plan evidence. These are not interchangeable tests.

4. Derivation and execution design

OperationDeclared inputs / assumptionsOutput, effects and invalidation
Admit / normalizeP2-valid authored/reference snapshot; registry and quantity contractsTyped P3 expressions and finite demand seeds; no inspection-driven construction
Resolve / instantiateFeatures, scopes/domains, candidate universe including absence, method preferences and bindingsClosed or explicitly unresolved demand, then complete named math bundles; real requirements cannot appear as an undeclared P6 feedback loop
Match / group / expandBound rule ports, key discipline, null/empty/order policies and contribution provenanceShared DataFusion/Arrow predicate execution, aggregate/unnest descriptors and indexed equations until the expansion consumer
Bind / prepare / executeCase values separately from guesses, numerical policy, actual kernel/backend contracts and shared budgetOwned prepared/branch workspaces, mandatory outcomes and policy-qualified derivatives; refresh bound values independently of topology reuse
Encode / publish / reuseAdmitted complete relations, reserved buffers, membership and semantic parentsCanonical logical hashes; separate finished physical bytes; verified manifest then conditional ref. Complete-key memo hits only; attempts/evidence remain sidecars

The library opportunities are bounded: AnalyzerRule and recursive traversal for admission; built-in CASE for relational guards; physical predicates and Arrow filters for exact scans; standard aggregation/unnesting for P8 and memberships; shared RuntimeEnv/reservation APIs for accountable allocations; field-aware UDF arguments and scalar paths; safe ScalarBuffer owners; and explicit-schema readers for observations. No new query engine or general inversion/ingestion framework is justified. CSV/JSON feature wiring remains part of the future import consumer.

5. Representative journeys

Ordinary extension. A method requiring a second property declares the requirement and implementation/quantity contract. P3/P6 bind the finite scope and candidate set before P9 realization. Its adapters come from the admitted binding; an unavailable derivative or undeclared late dependency is rejected. Extension work is the new declaration, any new algorithm and its focused conformance fixture.

Meaningful change. Add a species, change a unit set, or remove a provider. The domain/package/candidate input changes, so the complete P7/P6-dependent keys change. Fresh output snapshots preserve the new lineage. A fixed-value change refreshes bound_values and any value-dependent analysis/preparation even if the instruction topology is the same. Clean compilation is the equivalence oracle.

Alternate representation. Construct equal nested nullable relations with different hidden payload/child validity and serialize each as IPC file or Parquet. Canonical v2 specifies normalization before logical hashing. Each finished encoding has its own checksum; restore decodes and validates meaning instead of substituting a byte checksum for logical identity. A Python consumer that loses an extension cannot re-enter a checked platform path by silently relabeling storage.

Failure. A mixed batch selects different branches and includes a log-domain error only in an excluded branch. CASE/native masks prevent its evaluation; a selected invalid branch instead returns the declared error/outcome. Cancellation or reservation failure releases owned workspaces and prevents successful stage publication. A truncated existing object fails verification before a new ref can expose it. These are proposed journeys with the named acceptance fixtures in plan 02.

6. Acceptance gates

Each Pass below is Proposed document scope, not runtime assurance.

GateVerdictEvidence or scope rationaleRequired action
G1 — AuthorityPassRegistry meaning, named stage producers and explicit snapshot classes separate model, derived output and sidecars; §4/§5.3/§14.1Generate the projections; preserve accepted ADR arguments
G2 — Semantic fidelityPassComplete quantity rules, exact head admission, explicit missing/failure states, recursive canonical representation and numerical policy; §7–§8/§14.2/§18Execute semantic/representation fixtures before claiming correctness
G3 — ValidityPassEvery supported admission route invokes the validator; unsupported layouts, conversions, ports and bindings have rejection paths; §4.4/§5.3/§18.5Test actual entry points, including nested and optimized-away fields
G4 — Hidden behaviorPassGuarded execution is a real control-flow route; kernels are declared pure; candidate and policy inputs are explicit; §9.6/§14.3/§18.2Branch/effect and late-demand negatives
G5 — Consistency and recoveryPassComplete stage publication, distinct logical/encoded identity, verified existing objects, complete manifest then ref; §14.3/§20Local interruption, corruption, CAS and cancellation fixtures; wider deployment stays deferred
G6 — Transformation and reusePassComplete keys, explicit ordered numerical contracts, lossless/converting head policy and full pushdown oracle; §7.3/§14.2–§14.4/§24Incremental/clean, branch/numerical and complete-result differential checks
G7 — Truthful capability claimsPassFalse registration/conditional/protobuf/resource guarantees withdrawn; explicit implementation bindings and unsupported refusal; §18.5/§18.9/§26Keep entries Proposed until actual routes pass; no performance claim from API availability

7. Principle findings

No residual MUST gap was identified in the revised, bounded contract decisions. The table records the decisive corrections, not new assertions that the runtime already enforces them. Verification names are Proposed fixtures in plan 02.

Finding / dispositionPrinciple IDsConcrete evidenceConsequence addressedCorrection in revision 5Verification
R4-01/05 — Addressed in designDM-06, DM-07, DM-41–DM-43, DM-59§4.4/§14.2; E1 ordinary query bypass and E3 lossy castsInvalid quantities can no longer be admitted by the specified route merely because storage is castable or registeredActive recursive admission and source-aware explicit conversionsemantic_admission_paths, head_conversion_exactness
R4-02/09 — Addressed in designDM-24, DM-28, DM-40§7.3–§7.4/§18.2/§21.2; E2/E6Excluded-branch errors, reordered cancellation and nonlinear Affine bodies had inconsistent behaviorCASE/regions, guarded ordered default, selected numerical policy and qualified LinearExpressionGuarded/numerical/nonlinear-affine fixtures
R4-03/04 — Addressed in designDM-12, DM-14, DM-15, DM-48§5.3/§20; E4Equal relations differed by hidden null bytes, while valid file bytes contradicted logical object namesRecursive v2, explicit membership/framing and independent encoded checksumsCanonical/encoding/membership/publication fixtures
R4-06/07 — Addressed in designDM-18, DM-21–DM-23, DM-31–DM-33§6.11/§9.6/§14.1–§14.4Ambiguous producer order and omitted domain/binding/value inputs could create stale modelsEarly seeds, bound ports, complete output snapshots and conservative stage keysDemand/stage graph and incremental/clean fixtures
R4-08/10 — Addressed in designDM-01, DM-06, DM-08, DM-25, DM-43, DM-44, DM-47§6.2/§6.11/§6.13/§8.3/§18.5Adapters invented quantity, missing/failure or derivative semanticsComplete composition and kernel contracts, explicit WeightedMean and mandatory outcomesComposed quantity and actual kernel/backend fixtures
R4-11/12 — Addressed in designDM-15, DM-48, DM-53, DM-54, DM-59, DM-60§5.4/§14.2/§24.1; E5An empty pushed result passed the old oracle; noncanonical protobuf was presented as stableFull values/multiplicities oracle; noncanonical diagnostic encodings outside reuseDeliberate over-pruning negative; decoded-plan attribution/round trip
R4-13 — Addressed in designDM-30, DM-35, DM-39, DM-59§5.3/§14.3/§18.2A query pool did not account for other live allocations or oversized relation constructionShared budget, fallible platform reservations and explicit supported envelopeSeparate query/canonicalization/result budgets, concurrent release and process peaks
Proportionality — Accepted simplificationDM-33, DM-56–DM-58§14.3/§24.3/§26; ADR-0041/0042Early fine invalidation and canonical-plan machinery added unproved semantic choicesComplete keys first; exact input equality for skipping, measured triggers for refinementR-22/R-01 before adoption; full dependency fixtures precede timing

Applicability. All twelve groups bear on these cross-boundary changes, but this review reassesses only the principles listed below and their revised mechanisms. Other principles and unchanged subsystems retain the earlier review’s coverage limits; they are not silently recertified or declared irrelevant. “Satisfied” is a verdict on specified Proposed design, with the boundary and rejection behavior identified above. No numerical maturity score is assigned.

Applicable principlesVerdictMechanism / limiting scope
DM-01–DM-04SatisfiedComplete domain meaning; registry authority; separate physical layouts; contracted opaque bodies
DM-06–DM-10SatisfiedTyped operations/outcomes, active admission, bound domains/ports and queryable declarations
DM-12–DM-15SatisfiedExplicit model/case/output/attempt roles, coherent publication and role-specific identity
DM-18–DM-20SatisfiedSeeds before realized IDs, candidate closure and declared bindings; inspection does not construct physics
DM-21–DM-25SatisfiedStage ownership, complete transformation contracts, traceable outputs and guarded lowering
DM-28–DM-30SatisfiedPure kernel contract, owned workspaces, typed partial/terminal outcomes and publication boundary
DM-31–DM-33, DM-35SatisfiedComplete inputs and absence, conservative initial granularity, shared resource/ordering contracts
DM-36–DM-40SatisfiedNative layouts and safe owners for concrete consumers, coarse streams, explicit numerical policy; benefits remain hypotheses
DM-41–DM-44SatisfiedMechanical adapters, explicit losses/conversions, actual binding requirements and unsupported refusal
DM-46–DM-48SatisfiedFresh semantic lineage, mandatory typed failure evidence, logical reproducibility distinct from diagnostic bytes
DM-51–DM-54SatisfiedExplicit v2 refusal/migration path, generated contracts and adversarial/differential acceptance specifications
DM-56–DM-60SatisfiedExisting mechanisms with consumers, smaller initial reuse, falsifiable evidence and named change-level fixtures

8. Alternatives and architectural leverage

AlternativeSemantic duplication / localityRisks and costPerformance evidenceDecision
Revision 4 as writtenAdapters/pass authors must invent missing validation, type, dependency and failure meaningUnsupported library guarantees and premature fine reuseNo end-to-end measurementRejected by the earlier review
Revision 5Shared contracts invoke existing Arrow/DataFusion mechanisms; kernel bodies remain ordinary codeMore explicit boundary work; bounded canonicalizer and coarse keys simplify correctnessBenefits and recomputation costs remain hypothesesSelected for the stated architecture
Smaller Slice A runtime with whole-stage recompilation and only currently consumed backend bindingsSame semantic registry/IR; omit memo hits and defer every optional adapter/optimization until a consumer needs itExtra recomputation, smaller implementation and failure surface; still requires active admission, quantity semantics and coherent publicationMeasure before enabling memo/transfer refinementsViable implementation sequence within revision 5, provided unsupported routes remain explicit

Registry and operation generation are justified by repeated Rust/Python/storage boundaries; typed aggregate/unnest operations have P8/membership consumers. A new distributed service, universal inversion engine, extra dataframe system or automatic dependency framework is not justified by these findings. Canonical normalization, sparse numerical algorithms, masks and buffer ownership remain ordinary typed code, not new DSLs.

9. Verification and measurement plan

The dependency-ordered packets and exact proposed fixtures are in plan 02.

Claim / riskEvidenceCheck and conditionsState
Documentation consistencyTested, static/governance scope onlyPlan 02’s exact recipe/results table; baseline zero; original headings, pass rows, governed markers, links and immutable ADR bodiesResults recorded in plan 02; no runtime inference
Library counterexamplesTested, historicalRetained revision-4 runner/output: six groups, zero failed assertions, Rust 1.98.1, Arrow 59.3.0/DataFusion 55.1.0, dev, force validation, locked offline resolutionNot rerun for prose; does not test the new integration
Active admission, quantity, kernel and numerical preservationProposedReal import/query/output/backend fixtures, including excluded branches, lossy casts and missing-versus-invalid outcomesRequired before behavioral acceptance
Canonicalization, identity and recoveryProposedNested/null/metadata metamorphisms; IPC/Parquet differential; corrupt existing objects and interruption at every publication stepRequired before storage acceptance
Complete reuse and exact scansProposedMutate each named semantic dependency versus clean compile; full pushed/unpruned values and multiplicitiesRequired before reuse/pushdown acceptance
Cost and resourcesProposedCold/warm construction through solve/restore; pool and process peaks, copies, spills, decoded bytes and transfer/solver time under recorded limitsNo speedup or process-wide OOM immunity claimed

No ordinary full Rust/Python/solver suite can establish unimplemented behavior. Current runs are scoped static/governance checks; future implementation changes must run their actual test/slice recipes. API-reference lint remains deferred under R-20 and is not credited as interface verification.

10. Exceptions and unresolved decisions

No SHOULD deviation is needed for these revised contracts: whole-stage reuse is the smallest currently trustworthy granularity, supported boundaries refuse unavailable behavior, and optional features have explicit triggers. This review does not close unimplemented behavioral acceptance by treating it as an exception.

Decision outside the initial supported scopeControl / ownerObservable revisit trigger
Finer reuse, then salsaR-22/R-01; paul-heyseComplete finer input fixtures plus measured whole-stage preparation cost
Larger canonical relations / fixed-boundary streamsR-23; paul-heyseActual workload exceeds declared bounds and justifies a versioned format decision
Parquet pruning / bounded coalescingR-24; paul-heyseEach consumer independently shows full equivalence and lower total cost
Predicate preimagesR-25; paul-heyseExact supported predicate equivalence, including null/endpoint/floating cases, then measured benefit
Cloud/multi-writer stores and later providersExisting register rows including R-10Concrete deployment/provider requirements and their capability/conformance checks

11. Decision and implementation changes

Accept the revised, bounded Proposed design contracts. All seven gates pass for that document scope. The review does not accept runtime behavior, every unchanged blueprint detail, an ADR status transition or a performance claim.

PriorityChange / next acceptance boundaryPrinciplesAcceptance evidenceRegression protection
1Generate/admit complete quantity, kernel, snapshot and port contractsDM-06–DM-08, DM-22, DM-43Plan 02 packets A–C; actual boundary negativesE1/E3 counterexamples become integration fixtures
1Implement guarded numerical outcomes and coherent identity/publicationDM-14–DM-15, DM-24, DM-30, DM-40Packets B/E/F; full lifecycle and backend routesE2/E4/E6, corruption and excluded-branch regressions
1Establish complete reuse and pushdown correctnessDM-31–DM-33, DM-53–DM-54Packets C/D; clean differential and full-result oracleDomain/candidate/absence/fixed-value mutations and over-pruning negative
2Adopt bounded standard operators and real observation importsDM-18, DM-25, DM-41, DM-56Packets C/G; independent expansion/import fixturesNull/empty/order/units/target cases
3Refine only after measuring the complete workloadDM-39, DM-58–DM-60R-22–R-25 experiments under identical meaning and resource limitsRecorded conditions, equivalence and end-to-end cost