status: evidence-map blueprint_revision: 5 pins: pyproject.toml and uv.lock regenerated: null reviewed: 2026-09-13

Python libraries — capability map

Current binding: blueprint revision 5. The original survey and version/probe receipts below retain their historical scope; they were not regenerated in this update. Earlier revision comparisons and parked recommendations are historical. The revision-4 review and ADR-0039–ADR-0048 determine the corrected design bindings below. Platform integration remains Proposed; interface and probe evidence are labeled at their original scope. No Python behavioral corpus was re-audited by this revision.

Fourth and final map in the series, completing blueprint §3.3. The Rust side is covered by arrow-rust.md, datafusion-rust.md and supporting-rust-libraries.md; this covers line 331 — the Python row — plus the adjacent libraries the Python boundary actually depends on.

Compiled 2026-09-13.

0. Purpose, sourcing constraint, and evidence rules

0.1 What this document is

Per cluster: what each library can actually do, which APIs the blueprint’s requirements land on, what we deliberately will not use, and where the blueprint assumes a capability that does not exist or is under-specified. A second pass asks what these libraries offer that the blueprint has not claimed, which would improve alignment with design_principles/DATA_MODEL_DESIGN_CHARTER.md (DM-01…DM-60, gates G1–G7).

This boundary deserves the scrutiny. Everything crossing it was already typed once, in Rust, against a registry. §4.4 asserts that Python consumers “register matching pyarrow.ExtensionType classes”; §21.5 asserts typed attrs classes with no Any fields. Those are DM-42 and G2 claims about a language with no compiler to enforce them — so they were measured rather than assumed.

0.2 Sourcing constraint

Nothing is taken from existing repository documentation. This is both an instruction and a fact: docs/library_ref/ has been deleted, taking the former pyomo.md, pyarrow.md, attrs.md, cattrs.md and msgspec.md references with it. Four fresh lanes:

Checked mechanically: zero [ref:] citations, and every marker resolves to a ledger row. A claim carrying no marker is a defect.

0.3 Status vocabulary

leverage · build on top · confirmed · refined / erratum · adopt / evaluate / reject · open item. Pass-2 recommendations are disciplined by charter §G and DM-58: a recommendation naming no principle and replacing no identified hand-written work does not appear.

0.4 A correction to this document’s own reconnaissance

An early probe of the neighbouring environment suggested that an unregistered pse.* extension type would surface in Python as pyarrow.UnknownExtensionType. Measurement showed otherwise (§3, PROBE 1): it deserializes to its plain storage type. UnknownExtensionType exists in the API but is not what this path produces. The corrected finding is materially different — and better — because the degradation is silent at the type level but detectable through retained field metadata. Recorded here because the wrong version briefly informed the plan.

1. Version anchors and environment receipts

1.1 Anchors — every library at its current PyPI release

Owner’s instruction: use the latest release of every library, not the versions §3.1 names. All eight verified against PyPI at compile time [pypi:*].

1.2 Two interpreters, and why

idaes-pse 2.12.0 classifies Python 3.10–3.13 only [pypi:idaes-pse]; its requires_python is unset, so pip would not block a 3.14 install, but it is untested there. Rather than guess, the extraction uses two environments:

py314/   CPython 3.14.7   pyomo 6.10.1, pyarrow 25.0.1, attrs 26.1.0, cattrs 26.2.0,
                          msgspec 0.21.1, numpy 2.5.3, scipy 1.18.1 (+ pint 0.26.1, see §5)
py313/   CPython 3.13.12  the same set plus idaes-pse 2.12.0
                          (pulls pint 0.26.1, pydantic 2.13.5, sympy 1.14.0, networkx 3.6.1)

Every library ==-pinned, mirroring the =-pin discipline §3.1 adopted for Rust. IDAES imports cleanly on 3.13.12 [probe].

This split is itself a §3.1 finding. §3.1 says “Python 3.11+”; IDAES currently caps the interpreter at 3.13 for anything that must run the parity harness (§24). The platform’s own Python package can target 3.14; the parity environment cannot.

1.3 Extraction receipt

Scripts, both requirement sets and all probe programs are committed under docs/capability-maps/evidence/python/ so every [api:…] and [probe] marker is re-derivable.

2. pyarrow — PyCapsule interop and the stream boundary (P1)

Role: §21.1 — “every table crosses as an Arrow C stream (__arrow_c_stream__), never as row objects”; D12 — “one adapter consumes a CanonicalMathProblem bundle over the Arrow C stream interface”.

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Gaps and risks

• datafusion (Python) lagged the Rust engine in the environment surveyed — 54.0.0 against the Rust side’s 55.1.0. It is only a consumer of the capsule protocol, so version skew is tolerable by design; worth stating explicitly, since §21.1 names it and a reader could infer the versions must match.
• pyarrow 25.0.1 against §3.1’s “pyarrow 15+” is a ten-major gap. Nothing observed here requires 15; but an unpinned floor means CI could resolve anything in that range, and extension-type behaviour (§3) is exactly the kind of thing that moved over those releases. Pin it.

3. pyarrow — extension types, metadata, and the loss profile (P2)

The cluster that decides whether the semantic typing built in Rust survives into Python. DM-06, DM-42, DM-44, G2.

Capability inventory

Measured: the loss profile (PROBE 1 and 2)

A pse.semantic_id-shaped extension over binary(16), with pse.* schema and field metadata, round-tripped through IPC and through __arrow_c_stream__:

PROBE 1  extension type through IPC
   registered   -> field type: SemanticId          | extension_name: pse.semantic_id
   UNregistered -> field type: FixedSizeBinaryType | extension_name: None | storage: fixed_size_binary[16]
   unregistered field metadata retains ARROW:extension:name: True

PROBE 2  metadata across IPC and PyCapsule
   IPC  schema md : b'rel-1'      IPC  field md : b'qt-demo'
   caps schema md : b'rel-1'      caps field md : b'qt-demo'
   caps id type   : SemanticId

Pass 2 — alignment opportunities

Gaps and risks

• Registration is global and process-wide. register_extension_type mutates a global registry; a second registration of the same name raises. A library that registers on import interacts badly with test isolation and with any host process that also registers pse.*. Recommend an idempotent register-once helper in python/pse/.
• Extension metadata is versioned by nothing. __arrow_ext_serialize__ returns opaque bytes; if the Rust-side Metadata shape changes, the Python __arrow_ext_deserialize__ must change in lockstep or silently misparse. Same gap the Arrow map recorded for the Rust side (DM-44, DM-51) — and it is worse here, because the two implementations are in different languages and different repositories’ release cycles.

4. Pyomo — model construction and expression lowering (P3)

Role: §21.2 steps 1–2 — the adapter algorithm; §18.5’s Pyomo binding row; D12.

Capability inventory

Measured (PROBE 3)

   LinearExpression      : LinearExpression | value: 2.0
     nargs 3 | polynomial_degree: 1
   Expr_if               : Expr_ifExpression | value: 1.0
   ExternalFunction decl : AMPLExternalFunction | callable in expr: ExternalFunctionExpression
   (warning) Defining AMPL external function, but cannot locate specified library "libpse_kernels.so"

Pass 2 — alignment opportunities

Gaps and risks

• Param(mutable=True) is required for §21.2’s parameters to be updatable between solves; an immutable Param is folded into expressions at construction. The blueprint says mutable=True — correct, and worth keeping as a generated-code invariant rather than a convention.
• Pyomo’s expression system is large and has several near-equivalent spellings for the same mathematics (LinearExpression vs. a sum of products). Since the adapter is generated from the IR, the choice should be made once in the generator, not per operator.

5. Pyomo — units, suffixes, solve, and results (P4)

This cluster contains the map’s sharpest finding: a direct contradiction between §21.2 and §3.3.

The pint contradiction

§21.2 step 1 requires “units attached from the unit set so that assert_units_consistent passes”. §3.3’s closing line states: “Explicitly not added: … a Python-side units library.”

Measured [probe], on a clean environment with the §3.3-compliant dependency set (no pint):

   m2.T = pyo.Var(units=u.K, initialize=320.0)
   pyomo.common.errors.DeferredImportError: The pint module (an optional Pyomo dependency)
   failed to import: ModuleNotFoundError: No module named 'pint'

Pyomo’s units system is pint. pyomo.core.base.units_container defers to pint_module and raises on attribute access when it is absent [probe]. So §21.2’s requirement cannot be satisfied under §3.3’s exclusion. This is not a transitive-dependency inconvenience — the two statements are incompatible as written.

Recommended resolution (and the basis for an amendment): §3.3’s exclusion is about authority, not about presence. Reword it to say that no Python-side units library is an authority for quantities — §8’s dynamic registry remains sole authority — while acknowledging that pint is required by Pyomo’s units machinery and is therefore present in the adapter and parity environments. The boundary rule then becomes enforceable: pint may validate what the adapter emits; it may never define a quantity, and no pse relation may be derived from a pint object.

Capability inventory and measurements (PROBE 4, 5)

PROBE 4   consistent model      : PASSES
          inconsistent model    : raises InconsistentUnitsError ("kelvin not compatible with …")
          identify_inconsistent_units -> ['bad']        <- names the culprit constraint

PROBE 5   Suffix EXPORT/IMPORT  : declared ok; scaling_factor[x] = 0.001
          SolverFactory(ipopt) available: False
          SolverFactory(cbc)   available: False
          SolverFactory(glpk)  available: False

Pass 2 — alignment opportunities

6. Pyomo — ecosystem tools and the ASL boundary (P5)

Role: §21.3 — “parity testing, and the Pyomo ecosystem: parmest, PyROS, GDPopt, pyomo.dae utilities, and IncidenceGraphInterface-based cross-checks”; §21.4 opaque kernels and ASL; §18.9’s capability matrix.

Availability — all confirmed present at 6.10.1 [probe]

IncidenceGraphInterface resolves at pyomo.contrib.incidence_analysis.interface [probe], confirming §21.3’s naming.

Pass 2 — alignment opportunities

Gaps and risks

• pyomo.contrib.* is a contrib namespace. Five of the six tools §21.3 relies on live there. That is a stability profile the blueprint should state, because §18.9’s matrix currently reads as though these are equivalent in standing to the native and NL backends. They are not: they are the least stable surface in the entire dependency set.
• §21.4’s rule that “Python callback functions are not used for solves because ASL-based solvers cannot execute them” is a correct and important constraint, and the ExternalFunction probe (§4) shows why it must be enforced early: the object constructs happily either way, and only the solver notices.

7. attrs + cattrs — the typed contract layer (P6)

Role: §21.5 — “typed attrs classes with explicit converters at the boundary, cattrs for structuring, no Any fields, and no Python-side model classes that duplicate relations.”

Note: §21.5 cites docs/library_ref/contract_substrate_discipline.md for this discipline. That file and its directory have been deleted — so the referenced definition is currently unreadable (erratum E1). What follows measures the discipline’s stated properties directly.

Capability inventory

Measured (PROBE 6)

   structure happy path  : VarRow(ordinal=1, semantic_id=b'…', lower=0.0)
   unknown key           : ACCEPTED (silently dropped)
   forbid_extra_keys     : rejected -> ClassValidationError
   wrong type            : rejected -> ClassValidationError
   detailed_validation   : ClassValidationError | sub-errors: 1
   Any field             : ACCEPTED  (nothing in cattrs forbids it)
   attrs introspection   : fields() -> ['ordinal', 'semantic_id', 'lower']

Pass 2 — alignment opportunities

8. msgspec — wire contracts, constraints, and schema generation (P7)

Role: §21.5 names msgspec among the Python contracts; §20.2’s manifest is JSON. The blueprint never says what msgspec is for — §9 adjudicates that.

Capability inventory

Measured (PROBE 7)

   encode                : b'{"name":"stoichiometry","rows":12,"ratio":1.0}'
   decode round-trip     : Rel(name='stoichiometry', rows=12, ratio=1.0)
   unknown field         : rejected -> ValidationError | Object contains unknown field `BOGUS`
   wrong type            : rejected -> ValidationError | Expected `int`, got `str` - at `$.rows`
   Meta(gt=0) violation  : rejected -> ValidationError | Expected `float` > 0.0 - at `$.factor`
   json.schema(Rel)      : {'$ref': '#/$defs/Rel', '$defs': {'Rel': {'title': 'Rel', 'type': 'object',
                            'properties': {...}, 'required': ['name','rows'],
                            'additionalProperties': False}}}

Three properties matter for this design, and all three were measured rather than assumed:

1. Errors carry a location. at $.rows, at $.factor — a JSON path to the offending value. §23.2 requires failures that carry the thing that failed; msgspec supplies it natively.
2. Constraints are declarative and enforced at decode. Meta(gt=0.0) on a scaling factor is checked at the boundary, not by hand-written code afterwards.
3. It generates JSON Schema, with additionalProperties: False derived from forbid_unknown_fields.

Pass 2 — alignment opportunities

Gaps and risks

• msgspec is 0.21.1 — pre-1.0, the only pre-1.0 library in the Python set. Its API has been stable in practice, but the version number is the honest signal and it should be ==-pinned, not floated.
• msgspec.Struct is not attrs — the two class systems do not interoperate, which is the substance of §9.

9. Adjudication: attrs/cattrs versus msgspec (P8)

§3.3 lists three libraries that overlap heavily. §21.5 names attrs and cattrs and never says what msgspec is for. Leaving this open means the same meaning gets expressed twice at the boundary — DM-56 (“optimize for fewer independent semantic decisions”), DM-57 (“a small coherent core”), DM-58.

What the measurements say

The adjudication

They are not redundant; they occupy different positions, and the evidence separates them cleanly:

• msgspec is a wire codec. Strict by default, located errors, schema generation, four formats. Its natural job is bytes ↔ typed object at a serialization boundary: §20.2’s manifest, and any JSON the platform reads or writes.
• attrs + cattrs is an in-memory contract layer. cattrs structures from already-decoded material — which is what the Arrow path produces. Rows arriving from a RecordBatchReader are Python objects, not bytes; there is no JSON to decode, so msgspec’s strength does not apply, while cattrs’ ability to structure into arbitrary attrs classes does.

Recommended division, to be written into §21.5:

The rule that keeps this from becoming two authorities: a given contract is defined in exactly one of the two systems, never both. The manifest is a msgspec Struct; a relation row is a generated attrs class. No type is expressed in both.

This also resolves why three libraries is not over-engineering under DM-58: attrs and cattrs are one thing (classes plus their structuring), msgspec is another (a codec), and each replaces hand-written work that would otherwise exist — provided the division above is stated rather than left to taste.

10. numpy — the array boundary (P9)

Role: §21.1/§21.2 move numeric columns across the boundary; D11 governs layout ownership; §24’s parity harness compares native and Pyomo results numerically.

Measured (PROBE 8) — the null/NaN collapse

   zero-copy with nulls  : REFUSED -> ArrowInvalid
                           "Needed to copy 1 chunks with 1 nulls, but zero_copy_only was True"
   zero_copy_only=False  : [ 1. nan -0.]  | dtype float64 | null became: nan
   no-null zero-copy     : [ 1.  2. -0.]  | shares buffer: True
   -0.0 sign preserved   : True

This is the cluster’s finding, and it is a G2 one.

Pass 2 — alignment opportunities

Gaps and risks

• numpy is unpinned in §3.1 and arrives transitively through both Pyomo and pyarrow. numpy 2.x changed several conversion and dtype behaviours relative to 1.x. For a parity harness that compares numbers, the numpy version is part of the reproducibility contract (DM-48) and should be pinned like everything else.

11. scipy — structural and numerical cross-checks (P10)

Role: scipy is not named in §3.3, but arrives with Pyomo and is the natural oracle for §15’s native structural and numerical analyses.

Capability inventory [probe], [api:scipy@1.18.1]

Pass 2 — the cross-check opportunity

Gaps and risks

• scipy, like numpy, is unpinned and transitive. If it is going to serve as a correctness oracle in CI, it is part of the test contract and must be pinned (DM-48).
• An oracle that disagrees is only useful if the disagreement is attributable. These cross-checks should compare invariants (matching cardinality, structural rank, singular-value count below tolerance), not object identity.

12. IDAES 2.12 — the parity surface and its dependency boundary (P11)

Role: §3.1 “IDAES 2.10 for parity tests only”; §24’s parity harness; the source of the behaviours §9–§17 reproduce.

Availability [probe], [api:idaes@2.12.0]

IDAES 2.12.0 imports cleanly on CPython 3.13.12. 191 modules, 2762 public items extracted. The modules the parity harness needs all resolve:

Version finding

§3.1 names 2.10; current is 2.12.0 [pypi:idaes-pse], and this map is built against 2.12.0 per the owner’s instruction to use latest throughout. Parity is only meaningful against a stated version — §24’s acceptance criteria compare platform behaviour to IDAES behaviour, and IDAES changed between 2.10 and 2.12. This must be pinned exactly, not floored.

The dependency boundary

IDAES 2.12.0 requires pyomo>=6.10.1 — exactly this map’s Pyomo pin, so the two are compatible [pypi:idaes-pse]. It also pulls a substantial tail: pint 0.26.1, pydantic 2.13.5, sympy 1.14.0, networkx 3.6.1, plus pandas, matplotlib and click [probe].

Gaps and risks

• IDAES caps the interpreter at 3.13 (classifiers 3.10–3.13; requires_python unset) [pypi:idaes-pse]. §3.1’s “Python 3.11+” is therefore only true for the platform package; the parity environment has an upper bound as well as a lower one. Two dependency sets, two interpreter ranges.
• The vendored idaes-pse/ tree in this repository is at the version the blueprint named (2.10 line), not 2.12.0. Reading the tree and testing against the pinned package are two different sources; the pinned package is the one parity claims are made against.

13. Survey — pandas, polars, datafusion-python (P12)

14. Principle-alignment register (Python boundary)

Capabilities the blueprint does not claim, which would improve charter alignment. Disciplined by §G and DM-58: every row names a principle and the hand-written work or risk it removes. Rejections are recorded too.

Adopt

Evaluate

Reject, with reason

15. Acceptance-gate review (Python boundary)

The survey’s revision-2 gate results are historical. Blueprint §21 now specifies the following Proposed mechanisms. The revision-5 review assesses the changed contracts; no current end-to-end Python/backend certification is supplied by this map.

16. Leverage matrix

17. Evidence ledger

18. Open items and recommended blueprint amendments

This is a Proposed design disposition, not a rerun of the Python corpus. The original API/probe sections retain their conditions. Current dependency and interpreter authority is pyproject.toml, uv.lock and blueprint §3.1.

The implementation sequence and boundary fixtures are recorded in plan 02. Performance and solver support remain unmeasured/unverified until those specific paths are exercised.