status: evidence-map blueprint_revision: 5 pins: Cargo.lock regenerated: null reviewed: 2026-09-13

DataFusion (Rust) — capability map

Current binding: blueprint revision 5 and ADR-0039–ADR-0048. The inventories and original extraction/probe receipts below retain their historical scope and dates; this update did not regenerate that corpus. Historical comparisons to revision 2 are not current acceptance claims. Corrected interpretations are recorded in place. The current design remains Proposed; pinned library mechanisms are Interface-checked, and only the named characterization experiments are Tested.

[review:E1]–[review:E6] refer to the retained six-group characterization and its conditions: Rust 1.98.1, Arrow 59.3.0, DataFusion 55.1.0, dev, Arrow force_validate, zero failure baseline. The revision-4 review explains the counterexamples; these markers do not certify a platform implementation.

Companion to arrow-rust.md; together they cover the engine stack that supporting-rust-libraries.md deliberately excluded. The register (§13) and gate review (§14) here are now DataFusion-only; the Arrow map carries its own. In the previous edition both were combined here, which meant neither document could be updated alone.

Compiled 2026-09-13 against datafusion 55.1.0 on arrow 59.3.0. Historical survey adjudicated blueprint revision 2 (2026-09-13). The previous edition adjudicated revision 1 and never said so; §0.1 lists what changed underneath it.

0. Purpose, sourcing constraint, and evidence rules

Per cluster: what DataFusion can actually do, which APIs the blueprint’s requirements land on, what we deliberately will not use, and where the blueprint assumes a capability that does not exist or is under-specified. A second pass asks what the engine offers that the blueprint has not claimed, which would improve alignment with design_principles/DATA_MODEL_DESIGN_CHARTER.md (DM-01…DM-60, gates G1–G7).

Sourcing constraint. Written without the repository’s existing Arrow/DataFusion material — no docs/library_ref/ prose, no datafusion-pyarrow-* skills, no pre-existing build/facts/df55-default surface (which is at 55.0.0 and would misrepresent the pin), no vendored datafusion/ checkout. Two of those are now moot rather than merely avoided: docs/library_ref/ has been deleted, and the vendored checkouts survive only as orphaned worktrees whose parent repositories are gone, so their tags can no longer be verified. Five fresh lanes:

Checked mechanically: no provenance marker is of the form [ref:…], and none cites /apache/arrow (the multi-language C++/Python line). A claim carrying no marker is a defect.

Status vocabulary and the Pass-2 discipline (charter §G, DM-58) are as defined in the Arrow map §0.3. Pass 3 (§12) is new and answers a third question: taking the complete inventory of the family — every crate, every public trait, every config key, every release delta — what is this design not leveraging at all?

0.1 What changed underneath this map

The previous edition was compiled at 14:35 on 2026-09-13; the blueprint was rewritten at 15:24.

1. Version anchors and the extraction receipt

1.1 Anchors

1.2 Why 55.1.0, and a pin-discipline erratum — resolved in revision 2

Retained as a record of why §3.1 reads as it now does, not as a live recommendation.

Two corrections to earlier reconnaissance, both worth recording because they are easy to repeat:

1. 55.1.0, not 55.0.0, is current. A web search reported 55.0.0 as latest; the crate pages show datafusion 55.1.0 released 2026-09-11 [docs.rs:datafusion@55.1.0], with datafusion-expr 55.1.0 the same day [docs.rs:datafusion-expr@55.1.0]. Search results were stale by two days.

2. Pinning the umbrella does not pin the family. Resolving datafusion = "=55.0.0" produced datafusion 55.0.0 with every sub-crate at 55.1.0, because the umbrella depends on datafusion-* ^55.0.0. That mixed state is worse than either coherent choice, and cargo tree -d does not detect it — it is not a duplicate. The identical problem exists in the Arrow family (arrow 59.2.0 resolving arrow-schema 59.3.0).

Recommended §3.1 amendment (half applied): pin every crate in both families with =, or commit a Cargo.lock and enforce it in CI. Revision 2 adopted the = pins; it did not adopt the lockfile. = pins bind direct dependencies only — nothing stops a transitive path from introducing a second version, and only a committed lockfile plus a CI assertion catches that. This extraction does both, and its lockfile is committed under evidence/rust/.

1.3 Features (datafusion 55.1.0)

Default: nested_expressions, compression (xz2/bzip2/flate2/zstd), crypto_expressions, datetime_expressions, encoding_expressions, parquet, sql, regex_expressions, unicode_expressions, unparser, recursive_protection. Optional: avro, backtrace, parquet_encryption, serde [docs.rs:datafusion@55.1.0]. Dispositions in §10.

1.4 Extraction receipt

toolchain       rustc 1.100.0-nightly (809936eac 2026-09-12)
cargo           1.100.0-nightly (7941be6fb 2026-09-11)
flags           -Z unstable-options --output-format json
format_version  61
run             2026-09-13T16:15:24-04:00 .. 16:16:22-04:00
targets         57 requested, 57 OK, 0 FAIL, 60 JSON documents
DataFusion      37 crates, every crate_version 55.1.0
                datafusion, _catalog, _catalog_listing, _common, _common_runtime,
                _datasource, _datasource_arrow, _datasource_csv, _datasource_json,
                _datasource_parquet, _doc, _execution, _expr, _expr_common, _ffi,
                _functions, _functions_aggregate, _functions_aggregate_common,
                _functions_nested, _functions_table, _functions_window,
                _functions_window_common, _macros, _optimizer, _physical_expr,
                _physical_expr_adapter, _physical_expr_common, _physical_optimizer,
                _physical_plan, _proto, _proto_common, _proto_models, _pruning,
                _session, _spark, _sql, _substrait
Arrow           20 crates, every crate_version 59.3.0   (companion map)
lockfile        361 packages; one version per family, asserted

Newly covered relative to the previous edition’s 20 crates, and each for a reason: datafusion-proto / -proto-common / -proto-models (revision 2’s F9 depends on them and they were absent), datafusion-ffi, datafusion-substrait, the whole -datasource-* family, -physical-expr-adapter, -functions-{nested,table,window}, -catalog-listing, -spark, -common-runtime, -macros, -doc.

The normalized form is written to build/facts/df551/ — 76,771 declarations, 169 public traits, 48,968 impl relations — alongside, and never overwriting, the older df55-default profile at 55.0.0. Trait coverage more than doubled: 169 against the previous 148, from nearly twice the crates.

Committed under docs/capability-maps/evidence/rust/: the extraction manifest, the resolved Cargo.lock, the generation script and log, the five probe programs, and their captured output (probe_output.txt).

1.5 §3.1’s “55 forms” — all confirmed at 55.1.0

§3.1 states that scan_with_args/ScanArgs, PhysicalPlanningContext, EnsureRequirements, is_strict and convert_to_state “are the 55 forms to code against”. Every one resolves:

[rustdoc:*@55.1.0]. §3.1’s further note that “planner contracts moved toward datafusion-session” is also confirmed: TableProvider, CatalogProvider, CatalogProviderList, SchemaProvider, Session and ScanArgs all live there.

1.6 Probe receipt — new in this edition

The previous edition made no measurements. These six programs test assertions the blueprint makes, rather than trusting the API shape.

2. Catalog, providers, and scan_with_args (D1)

Role: §5.4 — “the catalog is a DataFusion CatalogProviderList whose catalogs are snapshots …, whose schemas are the seven namespaces, and whose tables are relations”; the provider contract that follows it.

Where the contracts live. In 55.1.0 TableProvider, CatalogProvider, CatalogProviderList, SchemaProvider, Session and ScanArgs are all defined in datafusion-session [rustdoc:datafusion-session@55.1.0]. §3.1’s note — “planner contracts moved toward datafusion-session” — is confirmed, and it means pse-catalog depends on datafusion-session, not on the umbrella crate, which is a smaller and more stable dependency edge.

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Gaps and risks

• CatalogProviderList::register_catalog and SchemaProvider::register_table are mutating methods on the catalog itself. Session setup uses them; nothing after setup should. Since a snapshot session is supposed to be immutable for its lifetime (§5.4), the platform’s implementations should reject registration after a seal() point rather than relying on nobody calling them (DM-29, G5).
• MergeIntoClause and InsertOp in the trait signature are evidence that DataFusion is growing a full DML surface release over release. This is a standing upgrade hazard for this design: each upgrade may add more defaulted mutation methods. The governance test above should assert the set of implemented methods, not just the absence of today’s five.

3. Pushdown truthfulness, constraints, and statistics (D2)

The cluster where §5.4 makes its sharpest claim — “Truthfulness over ambition: a wrongly advertised exact filter is a correctness bug” — and where G7 (truthful capability claims) and G6 (transformation and reuse) are decided.

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Measured (PROBE A) — the §5.4 pushdown spike

The previous edition listed this as must-settle item 1 and blueprint §26 F17 still says “the pushdown spike precedes any Exact claim”. It has now been run: a recording TableProvider over a FixedSizeBinary(16) key column, capturing every expression offered to supports_filters_pushdown [probe].

PROBE A  filter shapes offered to a provider

q: WHERE id = arrow_cast(X'…03','FixedSizeBinary(16)')
   offered: ["rel.id = FixedSizeBinary(16, "0,0,…,3")"]        reached scan(): 1

q: WHERE id IN (arrow_cast(X'…01',…), arrow_cast(X'…02',…))
   offered: ["rel.id = FixedSizeBinary(16, "…,1") OR rel.id = FixedSizeBinary(16, "…,2")"]
                                                               reached scan(): 1
q: WHERE kind = 'a'
   offered: ["rel.kind = Utf8("a")"]                            reached scan(): 1

q: WHERE kind = 'a' AND v > 2.0
   offered: ["rel.kind = Utf8("a")", "rel.v > Float64(2)"]      reached scan(): 2

q: WHERE id IS NOT NULL
   offered: []                                                  reached scan(): 0

Five facts, and they change §5.4’s status from unverifiable to specifiable:

1. Equality on a FixedSizeBinary(16) key arrives as a plain binary expression with a literal on one side — rel.id = FixedSizeBinary(16, …). That is a shape a provider can match structurally. Exact is achievable on key equality, which is what §5.4 promises. The blocking item is resolved in the design’s favour.
2. The measured two-element IN became an OR. This is not a universal shape guarantee: support only qualified expression forms, handle supported InList forms explicitly, and return Unsupported for the rest. Blueprint §5.4 now specifies that conservative contract.
3. Conjunctions are split. A AND B arrives as two independent filters, so a provider can accept one Exact and decline the other. §5.4’s per-column policy is expressible exactly as written.
4. IS NOT NULL on a non-nullable column is eliminated before the provider sees it — zero filters offered. The optimizer used the schema’s nullability declaration. This is a small, pleasing confirmation that Field nullability is load-bearing, and a reminder that it is the one constraint DataFusion actually enforces (see Gaps).
5. supports_filters_pushdown is called more than once per plan — each filter appeared twice. The method must therefore be pure and idempotent; a provider that logged, counted, or mutated state there would double-count. Not documented in the signature.

Gaps and risks

• Exact requires completeness as well as soundness. Reapplying filters to returned rows cannot detect missing matches. Compare full values and multiplicities with a pushdown-disabled provider and include a deliberately empty/over-pruning negative control (review R4-11; blueprint §24.1).
• supports_filters_pushdown must be pure (PROBE A, fact 5). Worth a comment in the generated provider, since the obvious place to put instrumentation is exactly there.
• DataFusion enforces no relational constraint except Field nullability — and does not check that on ingest. Upstream is explicit: primary, unique and foreign keys and check constraints “are provided for informational purposes”, are “not validated or used during query planning”, and returning nulls for a non-nullable column “will result in runtime errors during execution” [c7:/apache/datafusion]. §5.4 declares “Constraints: primary key only”, which is sound as a declaration — but it must not be read as enforcement. The platform’s P2 invariant validation is the only thing that makes the claim true (G7, DM-43).
• Statistics::new_unknown(&Schema) is the honest default and should be the starting point for every provider, with statistics added only where the manifest supplies them — rather than constructing Statistics optimistically and hoping.
• ColumnStatistics has no notion of which rows a statistic covers. For a snapshot artifact this is fine (it covers the whole artifact); it becomes wrong the moment a provider serves a subset. Another reason §5.4’s “the provider never prunes rows of a coupled mathematical problem” matters beyond its stated rationale.

4. Logical plans, expressions, and the rule compiler (D3)

Role: §14.2 — “a typed RulePlanSpec (a small algebra: scan, filter, project, equi-join, anti-join for stratified negation, union, distinct) compiled to a DataFusion LogicalPlan through LogicalPlanBuilder, optimized, and executed against the snapshot session”; D10’s first role; D6’s insistence that the math IR is not Expr.

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Measured (PROBE B) — does an extension node survive the optimizer?

A UserDefinedLogicalNodeCore carrying a rule_id, wrapped around an ordinary filter-and-project plan, put through SessionState::optimize [probe]:

PROBE B  UserDefinedLogicalNode survival

  before optimize:            after optimize:
    PseRule: rule_id=rule-42    PseRule: rule_id=rule-42
      Projection: rel.v            Filter: rel.v > Float64(1)
        Filter: rel.v > Float64(1)   TableScan: rel projection=[v]
          TableScan: rel

  node survived        : true
  rule_id still visible: true
  physical planning    : FAILS without an ExtensionPlanner
                         -> "No installed planner was able to convert the
                             custom node to an execution plan: RuleNode"

Three facts. The node survives intact and rule_id is still readable in the rendered plan, so the DM-18 opportunity is real rather than hypothetical. The optimizer still optimises through it — the projection was pushed down into the TableScan beneath the extension node, so wrapping a rule body does not cost the rewrites. And physical execution requires registering a datafusion_session::ExtensionPlanner; the failure is a clean typed planning error, not a silent fallback.

Measured (PROBE D) — float and null semantics through the engine

§14.2 rule 3 and §5.3 both depend on how the engine treats null, NaN and -0.0. The Arrow map measured the kernel behaviour; this measures the engine behaviour over the same five values [0.0, -0.0, NaN, -1.0, null] [probe]:

PROBE D
  ORDER BY x            : ["-1", "-0", "0", "NaN", "null"]
  GROUP BY x (distinct) : ["-1", "0", "NaN", "null"]     <- 4 groups from 5 rows
  self equi-join rows   : 6

Sorting and grouping disagree about whether -0.0 and +0.0 are the same value. ORDER BY reproduces Arrow’s IEEE 754 totalOrder exactly — -0.0 strictly before +0.0, NaN after all finite values — which confirms the Arrow map’s PROBE 3 survives into the engine. But GROUP BY returns four groups from five rows: -0.0 and +0.0 are merged, and the surviving representative is 0, not -0. The self equi-join returns six rows, which decomposes as four from the {0.0, -0.0} pair matching each other, one from -1.0, and one from NaN matching itself — so the hash join also treats NaN as equal to NaN, which is not IEEE comparison semantics.

None of this is a DataFusion defect; grouping and joining use value-equality semantics by design. It matters because §5.3 states that “-0.0 is preserved (it is distinguishable and can matter to solvers)” and §26 repeats it. That is true of Arrow ordering and of the content hash — and false of any aggregation or join keyed on a Float64 column. A GROUP BY over a float key silently merges the two, and which sign survives is an engine choice. The design should either declare that float columns are never grouping or join keys, or state the collapse explicitly as a selected loss (DM-42, G2, DM-08).

Gaps and risks

• Plan fingerprinting is not a DataFusion capability, and revision 2’s choice of datafusion-proto does not change that — it relocates the problem rather than solving it. LogicalPlan offers no stable hash; proto offers an encoding whose stability depends on inputs the platform controls. See D9, which is where this risk now lives (DM-48, DM-51).
• A Float64 column used as a grouping or join key loses the -0.0 distinction (PROBE D). Not a defect, but it contradicts a claim §5.3 and §26 both make without qualification.
• LogicalPlan carries Dml, Ddl, Copy and Statement variants. Nothing in the blueprint should construct them, for the same authority reason as D1’s mutation methods — the rule compiler builds read-only plans only.

5. Optimizer, analyzer, and plan explainability (D4)

Role: §14.2 — plans are “optimized” before execution, and rule 5 requires determinism: “the DataFusion session is pinned (config snapshot, function registry hash, catalog snapshot) and the plan fingerprint is part of the pass record.”

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Measured (PROBE X2) — EXPLAIN is four surfaces, not one

PROBE X2  EXPLAIN formats accepted by 55.1.0
   indent   -> OK    "TableScan: rel projection=[v]"
   tree     -> OK    box-drawing render of the physical plan
   pgjson   -> OK    structured JSON
   graphviz -> OK    "// Begin DataFusion GraphViz Plan"
   json     -> rejected: "Expected 'indent', 'tree', 'pgjson' or 'graphviz'"

pgjson turns the plan from a string a human reads into a structure a pass can store, diff and query — which is what DM-27 actually asks for. §23.1’s observability and §19’s analytics both become considerably better if a pass record carries a parsed plan rather than a blob of text. And datafusion.explain.{analyze_level, analyze_categories, show_statistics, show_schema} select what it contains, so the verbosity is declared policy rather than a fixed cost (§12 E3).

Gaps and risks

• The optimizer is an ambient dependency of every derived relation. This is the cluster’s real finding. §2 D14 says “incrementality follows declared dependencies” and §14.4 keys memoization on input content hashes plus pass version. The DataFusion version and its rule set are inputs that affect output and are not in that key. A DataFusion upgrade should therefore invalidate compiled artifacts — which argues for including the engine version in the pass key (DM-31, DM-32).
• No physical-optimizer trait was exposed in the extraction’s datafusion-physical-optimizer index (the struct EnsureRequirements is present, but the rule trait did not surface), so the physical rule contract should be read directly before any custom physical rule is written. Open item.

6. Physical planning, streams, and partitioning (D5)

Role: §14.3 session and parallelism; §18.8 thread budget; D10’s boundary — DataFusion “does not run inside Newton iterations, and a solve is never a scalar function call”.

Capability inventory

§3.1’s “55 forms” claim is confirmed in full at 55.1.0: scan_with_args, ScanArgs, PhysicalPlanningContext, EnsureRequirements, is_strict and convert_to_state all exist, in the crates named above.

What the blueprint binds to

Pass 2 — alignment opportunities

Gaps and risks

• Thread budget crosses three owners. §18.8 gives DataFusion the tokio runtime and rayon the batch kernels. DataFusion’s partition parallelism is driven by target_partitions in SessionConfig (D7) — so the budget is set in config, not by the pool. Getting this wrong oversubscribes silently, which §18.8 says must be a configuration error rather than a runtime surprise. The configuration must therefore be validated at session construction.
• The ExecutionPlan trait’s full method set was not enumerated here (the physical-plan crate indexes 6008 items); a custom physical operator is out of scope for the blueprint (D11 keeps numerics native), so only the provider’s returned plan matters. If a custom operator is ever written, re-derive this cluster.

7. Scalar UDFs and the generated kernel adapters (D6)

Role: §18.5 — the ScalarUDFImpl row of the kernel-adapter table; §14.2 — generated wrappers registered in the session so rule bodies can evaluate kernels in batch; D10’s third DataFusion role.

Capability inventory — ScalarUDFImpl has 25 methods in 55.1.0

Argument carriers: ReturnFieldArgs { arg_fields, scalar_arguments } and ScalarFunctionArgs { args, arg_fields, number_rows, return_field, config_options } [rustdoc:datafusion-expr@55.1.0].

What the blueprint binds to

Pass 2 — alignment opportunities

Gaps and risks

• The trait is large and growing — 25 methods in 55.1.0, several (preimage, conditional_arguments, placement, struct_field_mapping) clearly newer than the blueprint’s description. Because KernelSpec generates the adapters (§18.5), the cost of adopting a new method is one generator change rather than N hand edits — which is DM-52 working as intended, and a good argument for keeping the generation path even though the trait is a moving target.
• invoke_with_args receives number_rows, which matters for the all-scalar case (no arrays to infer length from). A generated adapter that infers length from its first array argument breaks on all-scalar invocations.
• Interval’s public surface did not resolve to a field list in the extraction [rustdoc:datafusion-expr-common@55.1.0]; the exact bound representation for evaluate_bounds should be read from the crate before the range-analysis adapters are written. Open item.

8. Session, runtime, memory, and ambient inputs (D7)

Role: §14.3 — “one DataFusion SessionContext per snapshot with the catalog of §5.4, MemoryPool limits, and the generated UDF registry”; §18.8 thread budget; §20.4 reproduction.

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Measured (PROBE H) — what does a bounded pool actually produce?

A 64 KiB GreedyMemoryPool, two partitions, sorting 200,000 rows [probe]:

PROBE H
  error: "Not enough memory to continue external sort. Consider increasing the
          memory limit config: 'datafusion.runtime.memory_limit', or decreasing
          the config: 'datafusion.execution.sort_spill_reservation_bytes'"

The recommendation is confirmed, and more strongly than expected: the failure is not merely typed, it is actionable — the message names the two configuration keys that would resolve it. That is precisely the shape §23.2 wants from a runtime.resources diagnostic, and it arrives for free simply by choosing a bounded pool. The counterfactual is an OOM kill with no message at all.

Gaps and risks

• Spill introduces a filesystem dependency the blueprint does not discuss. DiskManager writes temp files when a pool spills. For a reproducible, content-addressed platform that is benign (spill is invisible in results) but it is a real resource and failure surface: a full temp filesystem becomes an execution failure mid-pass. §23.2’s runtime.infrastructure class covers it; the disk manager should be configured explicitly rather than defaulted. 55.1.0 makes this pluggable — datafusion_execution::{SpillFile, SpillWriter, TempFileFactory} let the platform own spill placement and lifetime, and datafusion.execution.{spill_compression, max_spill_file_size_bytes} govern its shape (§12 E2, E3).
• MemoryLimit as a return type means a pool can report “unbounded” — worth asserting at startup that the configured pool reports a finite limit.
• Arrow allocations outside a query are invisible to this pool. §18.6’s result ingestion and §4.2’s builders construct arrays directly; DataFusion’s MemoryPool governs query operators only. Arrow’s own MemoryPool (the pool feature) covers the remainder — Arrow map §11 register row 17.

9. Errors, diagnostics, and observability (D8)

Role: §23.1 observability and §23.2’s failure taxonomy — “failures carry the relation rows and source spans involved; they are never flattened into ‘run failed’”.

Capability inventory

What the blueprint binds to

Pass 2 — alignment opportunities

Measured (PROBE X3) — how is an invalid configuration reported?

§14.3 constructs a session from a configuration the platform assembles, and §23.2 requires failures to be typed. Measured [probe]:

PROBE X3  setting datafusion.explain.format = "json" (not a valid value)
   SessionConfig::set_str  -> PANICS
        thread panicked at datafusion-execution-55.1.0/src/config.rs:186
        called `Result::unwrap()` on an Err value: Configuration("Invalid explain
        format. Expected 'indent', 'tree', 'pgjson' or 'graphviz'. Got 'json'")
   ConfigOptions::set      -> typed Err: "Error setting config datafusion.explain.format"

The convenient API panics; the correct one returns a typed error. This is the exact shape of the Arrow map’s Field::extension_type() versus try_extension_type() finding, and it deserves the same treatment: a panic during session construction is not a config.invalid row in §23.2’s taxonomy, it is a process abort with a backtrace. Ban SessionConfig::set_str in platform code; assemble configuration through ConfigOptions::set and convert the Err into a typed failure. Governance grep, beside the Field::extension_type() one.

This also matters for §14.2 rule 5, which reads semantic settings back from information_schema.df_settings: the write path is where validation happens, and it is the path that panics.

Gaps and risks

• DataFusion emits no tracing spans of its own by default. §23.1 wants spans “per rule evaluation”; the engine does not provide them, so rule-level spans are platform code around plan execution, and operator-level visibility requires an external instrumentation layer. This is a real gap between §23.1’s ambition and what the engine supplies out of the box. Two 55.1.0 mechanisms narrow it (§12 E2): datafusion_common_runtime::JoinSetTracer injects instrumentation into every future and blocking closure the engine spawns, and CustomMetricValue lets platform-defined metrics travel with an operator and appear in EXPLAIN ANALYZE. Neither is a tracing span, but together they cover most of what §23.1 wants. Open item, now with named candidates.
• SessionConfig::set_str panics on an invalid value (PROBE X3). Ban it.
• DataFusionError::Shared (an Arc-wrapped error) exists so one error can be reported to several consumers — a hint that concurrent execution can surface the same failure repeatedly. Pass-level failure recording should deduplicate.

10. Plan serialization, Substrait, SQL, and the unparser (D9)

Promoted from a survey to a full cluster, because blueprint revision 2 moved a load-bearing decision into it. F9 now specifies that the memo key and the pass record carry “a datafusion-proto plan fingerprint”, computed as “blake3 of the plan’s datafusion-proto encoding under the platform’s LogicalExtensionCodec” (§14.2). The previous edition of this map never evaluated datafusion-proto — its open item 6 offered only “a formatting surface” or “a platform structural walk”, and the crate was absent from the extraction corpus. It is now extracted and measured, and the result is the most consequential finding in this edition.

Role: §14.2 rule 5 and the plan fingerprint; §14.3 memoization; §20.2 manifest; §20.4 reproduction; §5.4 the catalog the plan refers to.

Capability inventory

Measured (PROBE C) — is a datafusion-proto fingerprint sound?

Three questions, in the order they matter: can it encode our plans at all; is the encoding deterministic; is it stable across sessions [probe].

PROBE C  datafusion-proto encoding

   without a codec:   FAILS -> NotImplemented("LogicalExtensionCodec is not provided")
                               ("Error serializing custom table")
   with a codec:      len=220   same plan encoded twice: identical=true
   equivalent plan, fresh session:                       identical=false
   same query, different whitespace:                     identical=false
   different predicate:                                  differs=true
   round-trip (encode -> decode -> re-encode):           identical=false

That last line is the alarming one: encoding, decoding and re-encoding in the same process with the same codec does not reproduce the bytes. So the instability is not a session artefact. A second probe isolated the cause [probe]:

PROBE C2/C3  plan-byte stability vs the number of field-metadata keys
             (6 independent encodings of the same query and schema)

   0 metadata key(s):  len=35   distinct encodings among 6 = 1   STABLE
   1 metadata key(s):  len=49   distinct encodings among 6 = 1   STABLE
   2 metadata key(s):  len=63   distinct encodings among 6 = 2   UNSTABLE
   3 metadata key(s):  len=77   distinct encodings among 6 = 3   UNSTABLE
   5 metadata key(s):  len=105  distinct encodings among 6 = 6   UNSTABLE

   3 keys, one shared Schema object reused:      distinct = 1   STABLE

The cause is Field::metadata’s HashMap iteration order. Rust seeds each HashMap independently, so two separately constructed but semantically identical schemas iterate their metadata in different orders, and datafusion-proto serialises them in iteration order. Reuse one Schema object and the instability vanishes; rebuild it and it returns. At five metadata keys, six encodings of the same plan produced six different byte strings.

What the blueprint binds to

Pass 2 and Pass 3 — alignment opportunities

Gaps and risks

• The fingerprint’s stability is a platform property, not an engine guarantee. Between the HashMap ordering above and the platform-written try_encode_table_provider, most of what determines the bytes is ours. DataFusion offers no stable-plan-hash API and does not claim to. §14.2 should state the fingerprint’s definition as a platform contract with its own version, exactly as §5.3 does for content hashes.
• Physical plans are a separate, larger surface. physical_plan_to_bytes exists and takes its own codec. F9 speaks only of a logical-plan fingerprint, which is the right scope — the physical plan depends on target_partitions and other execution-only settings that §14.2 deliberately excludes from the semantic-settings hash.
• datafusion-proto-models is separately published and separately versioned in principle; at 55.1.0 it moves with the family. Worth noting only because it is the crate whose wire format actually defines the bytes.

11. Logical types, extension types, and the type registry (D10) — new cluster

This is the largest single gap between the blueprint and DataFusion 55.1.0, and it did not exist in the previous edition because the surface did not exist in the extraction corpus.

DataFusion 55.1.0 provides a name-keyed extension registry and explicit validation helpers. X1 exercises a direct helper; it does not establish planning admission. E1 demonstrates the missing call on ordinary SELECT [review:E1]. The revised blueprint installs active semantic boundaries (§4.4).

Role: §4.4 (the ten pse.* extension types), §4.5 (the logical type catalog), §4.3 (metadata conventions), §5.4 (what the provider hands the engine), §18.5 (return_field_from_args and UDF typing).

This cluster is where DM-06 (“type semantic distinctions, not only machine representations”) stops being an Arrow-format property and becomes an engine property. The Arrow map’s A2 establishes that pse.semantic_id and pse.content_hash are two different semantic types over one storage type. What A2 could not offer was anywhere for the engine to enforce that. This is that place.

Capability inventory

Measured (PROBE X1)

Registering a pse.semantic_id type over FixedSizeBinary(16), exactly as §4.4 declares it [probe]:

PROBE X1  ExtensionTypeRegistry
   canonical types preloaded (7): arrow.timestamp_with_offset, arrow.opaque,
      arrow.fixed_shape_tensor, arrow.uuid, arrow.variable_shape_tensor,
      arrow.bool8, arrow.json
   register pse.semantic_id           -> 8 registrations
   resolve from a Field carrying
     ARROW:extension:name             -> storage_type = FixedSizeBinary(16)
   Field with the WRONG storage type  -> REJECTED
        "Error during planning: pse.semantic_id requires FixedSizeBinary(16), got Utf8"
   unregistered pse.* name            -> Err("Logical type not found")

And separately, that extension metadata survives the engine end to end [probe]:

PROBE E  ARROW:extension:name through planning and execution
   planned output field 'id'   : Some("pse.semantic_id")
   executed batch field 'id'   : Some("pse.semantic_id")
   after `SELECT id AS key_out`: Some("pse.semantic_id")
   as a GROUP BY key           : Some("pse.semantic_id")

Four consequences, in order of importance.

1. Direct validation is available, automatic plan validation is not established. Invoke the generated recursive validator at supported admission/plan/output boundaries; E1 proves registration alone is insufficient [review:E1].

2. An explicit registry lookup rejects an unknown name. An ordinary query may never perform that lookup. Python and Rust therefore both need an active platform boundary; the loss profile identifies the enforcing API rather than attributing enforcement to registration [review:E1].

3. ARROW:extension:name propagates through projection, aliasing and aggregation (PROBE E). §4.3’s claim that metadata is “a carrier” understates it: the carrier is preserved by the engine across operations that rewrite schemas. That is what makes an end-to-end semantic-type discipline feasible at all.

4. Seven canonical Arrow extension types are preloaded by default. MemoryExtensionTypeRegistry::new_with_canonical_extension_types() is what SessionStateDefaults uses, so arrow.json, arrow.uuid, arrow.bool8, arrow.opaque and the two tensor types are already recognised. The Arrow map’s §11 register row 2 (enable canonical_extension_types) has an engine-side counterpart that is on by default.

What the blueprint binds to

Pass 2 / Pass 3 — alignment opportunities

Gaps and risks

• Session scope is not validation coverage. Both engine and non-engine paths must invoke the declared checks. The registry supplies factories; Arrow typed helpers and platform cross-field/row validators supply distinct parts of the admission contract (§4.4; [review:E1]).
• create_extension_type_for_field is a provided method. A custom ExtensionTypeRegistry implementation that overrides it could weaken the check silently. Use MemoryExtensionTypeRegistry unless there is a reason not to.
• Registration order and duplicate names. add_extension_type_registration returns the previous registration for that name, so a later registration silently replaces an earlier one. Generated code should assert the return is None — a duplicate pse.* name is a build error, not a runtime surprise (G1).
• The registry validates storage type and metadata. It cannot validate cross-field constraints such as pse.ordinal_ref’s target relation, which the Arrow map records as an open item. That gap is unchanged.

12. Pass 3 — the under-leverage sweep

Passes 1 and 2 both start from the blueprint. Neither can find a capability the blueprint never gestured at, because neither enumerates the library independently. This pass does: four complete enumerations, each with a stated denominator, each row adjudicated bound, adopt, evaluate or reject, with reason. Charter §G and DM-58 still apply.

The denominator matters because of the rule this corpus inherits: absence from the extraction is not evidence that a feature does not exist. The previous edition covered 20 crates and 148 traits; this one covers 37 and 169.

12.1 E1 — the crate inventory

DataFusion 55.1.0 declares 44 workspace members [gh:apache/datafusion@55.1.0/Cargo.toml]; removing binaries, test harnesses, codegen helpers and example crates (datafusion-cli, datafusion-examples and its three FFI examples, datafusion-sqllogictest, datafusion-wasmtest, test-utils, benchmarks, the two */gen crates) leaves ~37 publishable library crates. All 37 are extracted.

12.2 E2 — the extension-point inventory

169 public traits across the 37 crates [rustdoc:df551@55.1.0], against 148 in the previous 20-crate corpus. Type-level machinery, sealed markers and internal accumulator plumbing are dismissed as a group. What follows is every trait that is a genuine place to insert platform behaviour and is not already covered by D1–D10.

12.3 E3 — the declared-policy surface

DataFusion’s policy surface is 155 configuration keys across eight namespaces [gh:apache/datafusion@55.1.0/docs/source/user-guide/configs.md]:

§14.2 rule 5 already gets the core of this right: it captures datafusion.optimizer.*, datafusion.sql_parser.* and datafusion.execution.time_zone into the semantic-settings hash and excludes execution-only keys such as batch_size and target_partitions. The enumeration confirms that partition is sound, and surfaces four keys that deserve individual attention:

The general finding: §14.2 rule 5’s semantic-settings hash is defined by namespace, but at least two semantically significant keys live under execution.*. The rule should be restated as an explicit allow-list of keys rather than a namespace prefix match, and the list versioned — otherwise a future DataFusion release adding a semantic key under execution.* silently escapes the hash (DM-31, G6).

Beyond config, the crate’s feature flags are the compile-time half (§1.3): defaults include unparser, recursive_protection, parquet, sql and the expression families; serde, avro, parquet_encryption and backtrace are off and stay off (D9).

12.4 E4 — the release delta, 54.0 → 55.1

Selected from the 55.0.0 changelog [gh:apache/datafusion@55.1.0/dev/changelog/55.0.0.md]. 55.1.0 itself is a bug-fix release with no new features.

12.5 Third-party extenders — bounded survey

13. Principle-alignment register (DataFusion)

Capabilities the blueprint does not claim, which would improve alignment with DATA_MODEL_DESIGN_CHARTER.md. Every row names a principle and the hand-written work or risk it removes. Rejections are included, because a recorded rejection is worth as much as an adoption.

In the previous edition this table combined Arrow and DataFusion rows. The Arrow rows now live in that map’s §11; these are DataFusion’s. Numbering is local to this map, and rows marked new come from §12’s Pass-3 enumeration or from this edition’s probes.

Adopt

Evaluate

Reject, with reason

14. Acceptance-gate review (DataFusion)

The previous gate interpretation overclaimed X1, UDF flags and sorted metadata insertion. E1/E2/E5 refute those mechanisms as universal enforcement. The revision-4 review replaces that interpretation; revision 5 specifies the active integrations.

Pinned interface availability is Interface-checked; integration stays Proposed until the named behavior is exercised. No numerical maturity total is inferred from this map.

15. Leverage matrix (DataFusion)

16. Evidence ledger

17. Open items and recommended blueprint amendments

Review E1–E6 and L1–L9 have retained source and receipts. Context7 supplied discovery leads; the pinned source and executed characterization bound the claims. No original extraction/probe corpus was regenerated here.