feat: wrap up data feature matrix & optimize example runs

Author: Spelkington

.claude/skills/flowthru-contributing/SKILL.md

@@ -16,3 +16,5 @@ When adding or modifying Flowthru internals:
 - Add pre-flight validation for environmental concerns (files, connections, external schemas).
 - Reserve runtime error handling for truly unpredictable failures.
 - Ask not just "Will this work?" but "When will it break?"
+
+Flowthru also depends on, and connects to, many other projects. Projects that may be helpful to directly introspect can be found in `docs/reference/misc/external/*/repo` subdirectories. If a source's `repo` subdirectory is missing, it can be pulled by running `nx run xdocs:pull <source>`.

docs/project.json

@@ -5,7 +5,7 @@
   "sourceRoot": "./docs",
   "targets": {
     "_build-reference": {
-      "//": "Generates markdown API reference from C# docstrings into docs/reference/src/. Inputs are scoped to the C# source + the docfx script — hand-written docs (tutorials/, guides/, explanation/) do NOT invalidate this cache, since docfx doesn't read them.",
+      "//": "Generates markdown API reference from C# docstrings into docs/reference/src/. Inputs are scoped to the C# source + the docfx script — hand-written docs (tutorials/, guides/, explanation/) do NOT invalidate this cache, since docfx doesn't read them. Private subtarget; composed into docs:build.",
       "executor": "nx:run-commands",
       "dependsOn": [ "flowthru:restore" ],
       "cache": true,
@@ -24,25 +24,40 @@
         "parallel": false
       }
     },
-    "build": {
-      "//": "Umbrella docs build target. Inputs mirror _build-reference so this stays in sync with the underlying docfx run; downstream consumers (site:_ingest-docs) read the produced reference/src/ tree directly.",
+    "_build-capability-matrix": {
+      "//": "Regenerates docs/reference/extensions/capability-matrix.md by running the file-based C# program at scripts/_test/capability-matrix.cs. The same script is exercised in freshness mode by tests:_test:capability-matrix-freshness; this target is the generation half. Private subtarget; composed into docs:build. Inputs scoped to the format-extension surface the generator reflects on plus the script itself, so unrelated source changes don't invalidate the cache.",
       "executor": "nx:run-commands",
-      "dependsOn": [ "_build-reference" ],
       "cache": true,
       "inputs": [
-        "{workspaceRoot}/src/**/*.cs",
-        "{workspaceRoot}/src/**/*.csproj",
-        "{workspaceRoot}/scripts/docfx-metadata.sh",
-        "{workspaceRoot}/Directory.Build.props",
-        "{workspaceRoot}/Directory.Build.targets"
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Capabilities/FormatRowFeatures.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Storage/IFormatBase.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Storage/IFormatRowReader.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Storage/IFormatRowWriter.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Storage/IFormatStreamReader.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Storage/IFormatSerializer.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Serialization/OptOutOfPropertyPlannerAttribute.cs",
+        "{workspaceRoot}/src/core/Flowthru.Core/Data/Storage/Format/JsonFormatSerializer.cs",
+        "{workspaceRoot}/src/extensions/Flowthru.Extensions.Csv/Data/Storage/Format/CsvFormatSerializer.cs",
+        "{workspaceRoot}/src/extensions/Flowthru.Extensions.Excel/Data/Storage/Format/ExcelFormatSerializer.cs",
+        "{workspaceRoot}/src/extensions/Flowthru.Extensions.Parquet/Data/Storage/Format/ParquetFormatSerializer.cs",
+        "{workspaceRoot}/scripts/_test/capability-matrix.cs"
       ],
-      "outputs": [ "{workspaceRoot}/docs/reference/src" ],
+      "outputs": [ "{workspaceRoot}/docs/reference/extensions/capability-matrix.md" ],
       "options": {
         "commands": [
-          "echo 'Docs Build Complete!'"
+          "dotnet run scripts/_test/capability-matrix.cs --verbosity quiet"
         ],
+        "cwd": "{workspaceRoot}",
         "parallel": false
       }
+    },
+    "build": {
+      "//": "Public docs build barrel. Composes private _build-* subtargets via dependsOn — each subtarget owns its own input/output footprint and caches independently. Adding a new MD generator (e.g., a future schema-gallery or diagnostic-id catalog) is a new _build-<name> private subtarget plus one line here. Mirrors the tests:test pattern.",
+      "executor": "nx:noop",
+      "dependsOn": [
+        "_build-reference",
+        "_build-capability-matrix"
+      ]
     }
   }
 }

docs/reference/extensions/capability-matrix.md

@@ -1,19 +1,38 @@
 # Flowthru Format Extension Capability Matrix
 
-This document is **auto-generated** from each format extension's `IFormatSerializer<TRow>.RowFeatures` declaration. Do not edit by hand — the `_test:capability-matrix-freshness` meta-test fails on drift.
+This document is **auto-generated** from each format extension's `IFormatBase<TRow>.RowFeatures` declaration and which capability segments it implements (`IFormatRowReader<TRow>`, `IFormatRowWriter<TRow>`, `IFormatStreamReader<TRow>`). Do not edit by hand — the `_test:capability-matrix-freshness` meta-test fails on drift.
 
-Regenerate locally via `nx run tests:_test:capability-matrix-freshness` or directly via the `Flowthru.Tools.CapabilityMatrix` tool.
+Regenerate locally via `nx run tests:_test:capability-matrix-freshness`, `nx run docs:build`, or directly via `dotnet run scripts/_test/capability-matrix.cs`.
+
+## Universal baseline
+
+All four formats round-trip the universal row-shape baseline:
+
+- CLR primitives (`int`, `string`, `bool`, `double`, `decimal`, …)
+- BCL scalar structs (`Guid`, `DateTime`, `TimeSpan`, `DateTimeOffset`, …)
+- `Nullable<T>` value types and nullable reference types
+- `[SerializedLabel("…")]` field-name mapping
+- `[SerializedEnum("…")]` enum value mapping
+- `required` members and positional-record activation
+
+These features are intrinsic to the planner's classification cascade and don't vary across formats. The matrix below tracks capabilities **on top of** that baseline — features where format-by-format support genuinely differs.
 
 ## Row-shape Features
 
-Each format declares which row-shape features it round-trips. A `✓` cell means the format claims support and the corresponding kit conformance fixture (under `tests/helpers/Flowthru.Tests.Kits/Fixtures/`) round-trips successfully. A `✗` cell means the format declares the feature unsupported — kit fixtures requiring that feature skip vacuously for this format.
+Each format declares which row-shape features it round-trips on top of the universal baseline. Cell semantics:
 
-| Format | IScalar wrappers | byte[] columns | Nested rows |
-|---|:---:|:---:|:---:|
-| **CSV** (Flowthru.Extensions.Csv) | ✓ | ✗ | ✗ |
-| **Excel** (Flowthru.Extensions.Excel) | ✓ | ✗ | ✗ |
-| **Parquet** (Flowthru.Extensions.Parquet) | ✗ | ✗ | ✗ |
-| **JSON** (Flowthru.Core (built-in)) | ✓ | ✓ | ✓ |
+- **`✓`** — format claims support; the matching kit conformance fixture round-trips successfully.
+- **`✗`** — format claims false; could be implemented but isn't. Tracked as a follow-up; kit fixtures requiring the feature skip vacuously.
+- **`—`** — structurally not applicable; the format's generic constraint (`where TRow : IFlatSchema`) prevents the schema shape from compiling. The matching fixture cannot be wired against this format.
+
+| Format | Schema shape | IScalar wrappers | Nested rows |
+|---|---|:---:|:---:|
+| **CSV** (Flowthru.Extensions.Csv) | Flat-only | ✓ | — |
+| **Excel** (Flowthru.Extensions.Excel) | Flat-only | ✓ | — |
+| **Parquet** (Flowthru.Extensions.Parquet) | Flat-only | ✗ | — |
+| **JSON** (Flowthru.Core (built-in)) | Flat or nested | ✓ | ✓ |
+
+Primitive-level format mechanics (`byte[]` blobs handled as base64/binary, timezone semantics on `DateTimeOffset`, etc.) are intrinsic to each format's underlying serialization library and aren't tracked here.
 
 ## Property Mapping
 
@@ -30,10 +49,16 @@ Format extensions are expected to consume Core's `PropertyMappingPlanner` for pe
 
 Medium-level capabilities of each format. See `Flowthru.Core.Data.Capabilities.StorageTraits` for the full surface.
 
+**Read / Write / Stream columns** carry two signals. Phase D (capability-segmented interfaces) split the format surface into `IFormatRowReader<TRow>`, `IFormatRowWriter<TRow>`, and `IFormatStreamReader<TRow>` (a sub-interface of the row reader, marking bounded-memory decoding). A format that does not implement a segment is *structurally* incapable of that operation — the absence is enforced by the type system, not a runtime trait flag. A format that implements the segment but reports `Traits.CanWrite = false` (etc.) is *runtime*-disabled.
+
+- **`✓`** — segment implemented and runtime trait permits.
+- **`—`** — segment not implemented (structural / compile-time signal). Calling code paths against the missing segment fail at compile time.
+- **`✗`** — segment implemented but runtime trait reports unavailable (e.g., medium pointed at a read-only file system).
+
 | Format | Read | Write | Stream | Append | Transactional |
 |---|:---:|:---:|:---:|:---:|:---:|
 | **CSV** | ✓ | ✓ | ✓ | ✗ | ✗ |
-| **Excel** | ✓ | ✗ | ✗ | ✗ | ✗ |
+| **Excel** | ✓ | — | — | ✗ | ✗ |
 | **Parquet** | ✓ | ✓ | ✓ | ✗ | ✗ |
-| **JSON** | ✓ | ✓ | ✗ | ✗ | ✗ |
+| **JSON** | ✓ | ✓ | — | ✗ | ✗ |
 

docs/reference/misc/external/akka-net/.source.env

@@ -0,0 +1,2 @@
+SOURCE_TYPE=repo
+SOURCE_ADDRESS=https://github.com/akkadotnet/akka.net.git

docs/reference/misc/external/ct-dotnet/.source.env

@@ -0,0 +1,2 @@
+SOURCE_TYPE=repo
+SOURCE_ADDRESS=https://github.com/CommunityToolkit/dotnet.git

docs/reference/misc/external/dapr-dotnet-sdk/.source.env

@@ -0,0 +1,2 @@
+SOURCE_TYPE=repo
+SOURCE_ADDRESS=https://github.com/dapr/dotnet-sdk.git

docs/reference/misc/external/masstransit/.source.env

@@ -0,0 +1,2 @@
+SOURCE_TYPE=repo
+SOURCE_ADDRESS=https://github.com/MassTransit/MassTransit.git

docs/reference/misc/external/ms-codecoverage/.source.env

@@ -0,0 +1,2 @@
+SOURCE_TYPE=repo
+SOURCE_ADDRESS=https://github.com/microsoft/codecoverage.git

docs/reference/misc/external/orleans/.source.env

@@ -0,0 +1,2 @@
+SOURCE_TYPE=repo
+SOURCE_ADDRESS=https://github.com/dotnet/orleans.git

docs/reference/src/core/Flowthru.Core/Flowthru.Core.Data.Storage.ComposedStorageAdapter-2.md

@@ -86,7 +86,9 @@ as explicit interface implementations to avoid polluting the base interface.
 
 ### <a id="Flowthru_Core_Data_Storage_ComposedStorageAdapter_2__ctor_Flowthru_Core_Data_Storage_IStorageMedium_Flowthru_Core_Data_Storage_IFormatSerializer__1__Flowthru_Core_Data_Storage_IContainerAdapter__0__1__"></a> ComposedStorageAdapter\(IStorageMedium, IFormatSerializer<TRow\>, IContainerAdapter<TContainer, TRow\>\)
 
-Creates a new composed storage adapter.
+Creates a new composed storage adapter from a write-capable format serializer.
+Backward-compatible entry point: any <xref href="Flowthru.Core.Data.Storage.IFormatSerializer%601" data-throw-if-not-resolved="false"></xref> is
+both a reader and a writer and is wired into both segments.
 
 ```csharp
 public ComposedStorageAdapter(IStorageMedium medium, IFormatSerializer<TRow> format, IContainerAdapter<TContainer, TRow> container)
@@ -96,15 +98,46 @@ public ComposedStorageAdapter(IStorageMedium medium, IFormatSerializer<TRow> for
 
 `medium` [IStorageMedium](Flowthru.Core.Data.Storage.IStorageMedium.md)
 
-The storage medium (file, memory, etc.)
+The storage medium (file, memory, etc.).
 
 `format` [IFormatSerializer](Flowthru.Core.Data.Storage.IFormatSerializer\-1.md)<TRow\>
 
-The format serializer (CSV, JSON, etc.)
+The full-duplex format serializer (CSV, JSON, Parquet).
 
 `container` [IContainerAdapter](Flowthru.Core.Data.Storage.IContainerAdapter\-2.md)<TContainer, TRow\>
 
-The container adapter (IEnumerable, IDataView, etc.)
+The container adapter (IEnumerable, IDataView, etc.).
+
+### <a id="Flowthru_Core_Data_Storage_ComposedStorageAdapter_2__ctor_Flowthru_Core_Data_Storage_IStorageMedium_Flowthru_Core_Data_Storage_IFormatRowReader__1__Flowthru_Core_Data_Storage_IFormatRowWriter__1__Flowthru_Core_Data_Storage_IContainerAdapter__0__1__"></a> ComposedStorageAdapter\(IStorageMedium, IFormatRowReader<TRow\>, IFormatRowWriter<TRow\>?, IContainerAdapter<TContainer, TRow\>\)
+
+Creates a new composed storage adapter with separate reader and writer segments
+(Phase D capability-segmented interfaces). Read-only formats — e.g.,
+<code>ExcelFormatSerializer</code> implementing only <xref href="Flowthru.Core.Data.Storage.IFormatRowReader%601" data-throw-if-not-resolved="false"></xref> —
+pass <a href="https://learn.microsoft.com/dotnet/csharp/language-reference/keywords/null">null</a> for the writer; the resulting adapter exposes
+<xref href="Flowthru.Core.Data.Storage.ComposedStorageAdapter%602.Traits" data-throw-if-not-resolved="false"></xref>.<xref href="Flowthru.Core.Data.Capabilities.StorageTraits.CanWrite" data-throw-if-not-resolved="false"></xref> as <a href="https://learn.microsoft.com/dotnet/csharp/language-reference/builtin-types/bool">false</a>
+and <xref href="Flowthru.Core.Data.Storage.ComposedStorageAdapter%602.Save(%600)" data-throw-if-not-resolved="false"></xref> fails fast.
+
+```csharp
+public ComposedStorageAdapter(IStorageMedium medium, IFormatRowReader<TRow> reader, IFormatRowWriter<TRow>? writer, IContainerAdapter<TContainer, TRow> container)
+```
+
+#### Parameters
+
+`medium` [IStorageMedium](Flowthru.Core.Data.Storage.IStorageMedium.md)
+
+The storage medium.
+
+`reader` [IFormatRowReader](Flowthru.Core.Data.Storage.IFormatRowReader\-1.md)<TRow\>
+
+Format reader segment. Required.
+
+`writer` [IFormatRowWriter](Flowthru.Core.Data.Storage.IFormatRowWriter\-1.md)<TRow\>?
+
+Format writer segment. <a href="https://learn.microsoft.com/dotnet/csharp/language-reference/keywords/null">null</a> for read-only formats.
+
+`container` [IContainerAdapter](Flowthru.Core.Data.Storage.IContainerAdapter\-2.md)<TContainer, TRow\>
+
+The container adapter.
 
 ## Properties