feat(runtime): add TensorRT-RTX runtime cache, dynamic shapes strategy, and native CUDA graph support to C++ runtime

[tp5uiuc]

Description

Extends three TensorRT-RTX runtime features that landed on the Python runtime (PythonTorchTensorRTModule) to the C++ runtime path (TorchTensorRTModule → core/runtime/TRTEngine). All three features center on nvinfer1::IRuntimeConfig, which the C++ runtime previously did not use — it called createExecutionContext(...) directly at four sites.

Features ported in this stack:

Feature	Python PR	This PR adds C++ runtime support
Runtime cache API	#4180 (merged)	yes
Dynamic shapes kernel specialization strategy	#4184 (merged)	yes
TensorRT-RTX native CUDA graph strategy	#4187 (draft)	yes

Without this PR, users on the C++ runtime path (TorchScript deployments, use_python_runtime=False) cannot access any of these TRT-RTX features, and runtime-cache warm-start savings (~8× measured in #4180) are unavailable on that path.

Commits

Original stack (reviewable independently):

1. feat(runtime): introduce IRuntimeConfig scaffolding and bump ABI to v9 — shared infra. Adds the IRuntimeConfig/IRuntimeCache members (RTX-only), a private recreate_execution_context() helper replacing 4 direct createExecutionContext call sites, three new SerializedInfoIndex entries (RUNTIME_CACHE_PATH_IDX, DYNAMIC_SHAPES_KERNEL_STRATEGY_IDX, CUDA_GRAPH_STRATEGY_IDX) with a single ABI bump 8 → 9. Python settings + compile() parameter threading. No behavior change — the per-feature apply_* appliers are empty stubs filled in by subsequent commits.

2. feat(runtime): add runtime cache to C++ runtime for TensorRT-RTX — mirror of feat: add runtime cache API for TensorRT-RTX #4180. Load on engine setup, atomic save on destructor (tmp + rename).

3. feat(runtime): add dynamic shapes kernel specialization strategy to C++ runtime — mirror of feat: add dynamic shapes kernel specialization strategy for TRT-RTX #4184. Wires IRuntimeConfig::setDynamicShapesKernelSpecializationStrategy.

4. feat(runtime): add TensorRT-RTX native CUDA graph strategy to C++ runtime — mirror of feat: add TRT-RTX native CUDA graph support #4187. Wires IRuntimeConfig::setCudaGraphStrategy and makes execute_engine.cpp TensorRT-RTX-aware: bypasses manual at::cuda::CUDAGraph capture on RTX (TRT-RTX handles it internally) and uses cudaStreamIsCapturing to detect outer whole-graph capture.

5. test: consolidate C++ runtime tests — folds C++ runtime cases into the existing test_000_runtime_cache.py and test_001_dynamic_shapes_kernel_strategy.py; adds model-level coverage in test_runtime_cache_models.py and test_dynamic_shapes_kernel_strategy_models.py.

Review-response commits (on top of the feature stack):

6. refactor(runtime): extract TRTRuntimeConfig — moves all TensorRT-RTX-specific IRuntimeConfig state into core/runtime/TRTRuntimeConfig.{h,cpp}, collapses three separate constructor args into a single TRTRuntimeConfig sink, uses enums (DynamicShapesKernelStrategy, CudaGraphStrategyOption), makes the destructor exception-safe, and contains the #ifdef TRT_MAJOR_RTX scatter to the new TU.

7. refactor(runtime): third-round review polish + cross-backend verification — inlines the FlattenedState macro, moves enum helpers + cache I/O into an anonymous namespace in the cpp, replaces (void)param; with TORCHTRT_UNUSED, uses std::underlying_type_t throughout, adds [[nodiscard]], adds ENABLE_FEATURE_DISABLE_RUNTIME_ALLOCATION as a local_define for the RTX Bazel configs so the runtime-allocation feature gate matches the RTX header's expectation.

Why bundle three features in one PR

All three features require an IRuntimeConfig on the engine, a single ABI bump, and extensions to the same serialization/deserialization code paths. Splitting into three independent PRs would trigger three consecutive ABI bumps and triple the surface area for backward-compat fallout. Keeping them in one stack keeps ABI changes atomic while still giving reviewers clean per-feature diffs.

Type of change

• New feature (non-breaking change which adds functionality — on the Python API surface)
• Breaking change (serialized-engine ABI bump from "8" to "9" — old .pt/.ep files targeting the C++ runtime will fail verify_serialization_fmt with a clear error, as with every prior ABI bump)
• This change requires a documentation update — default values ("lazy", "disabled") keep existing behavior; existing docs for the Python-runtime runtime cache already cover the concept

Checklist

• My code follows the style guidelines of this project (clang-format, black, isort, ruff, mypy, typos, buildifier — all pre-commit checks pass)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas (outer-capture detection, cache save-on-destructor contract, ABI bump rationale)
• I have made corresponding changes to the documentation (docstrings updated on CompilationSettings; full user-guide updates pending the feature PRs on the Python path)
• I have added tests to verify my fix or my feature (C++ runtime cases folded into test_000_runtime_cache.py, test_001_dynamic_shapes_kernel_strategy.py, test_001_cuda_graph_strategy.py; model-level coverage in test_runtime_cache_models.py, test_dynamic_shapes_kernel_strategy_models.py)
• New and existing unit tests pass locally on both backends (see test plan below)
• I have added the relevant labels to my PR

Test plan

Cross-backend verification on an A100 (ipp1 node) inside the main-native-x86_64-ubuntu24.04-cuda13.0 dev container, PyTorch nightly 2.13.0.dev20260420, CUDA 13.0.

TensorRT-RTX build

Wheel: torch_tensorrt_rtx-2.12.0.dev0+612556ba0 (built with python3 setup.py bdist_wheel --use-rtx, TensorRT-RTX 1.4.0.76).

Import smoke:

• ENABLED_FEATURES.tensorrt_rtx = True
• ABI_VERSION = 9
• SERIALIZATION_LEN = 14
• RUNTIME_CACHE_PATH_IDX = 11, DYNAMIC_SHAPES_KERNEL_STRATEGY_IDX = 12, CUDA_GRAPH_STRATEGY_IDX = 13

$ pytest runtime/test_000_runtime_cache.py \
         runtime/test_001_dynamic_shapes_kernel_strategy.py \
         runtime/test_001_cuda_graph_strategy.py
================= 35 passed, 3 skipped, 119 warnings, 3 subtests passed =================

Breakdown of the 35 RTX-side passes:

• Runtime cache — Python-runtime setup (TestRuntimeCacheSetup), persistence and warm-cache roundtrip on both Python and C++ runtimes (TestRuntimeCachePersistence, TestRuntimeCacheCppPersistence), concurrency/filelock (TestRuntimeCacheConcurrency), timing-cache skip (TestTimingCacheSkipped), and serialization-index registration (TestCppSerializationIndices).
• Dynamic shapes kernel strategy — setup / enum plumbing (TestDynamicShapesKernelStrategySetup), full {lazy, eager, none} end-to-end through both runtimes (TestDynamicShapesKernelStrategyCpp), dynamic-shape traversal, and invalid-value rejection (TestDynamicShapesKernelStrategyCppInvalidValue).
• CUDA graph strategy — settings defaults (TestCudaGraphStrategySettings), {disabled, whole_graph_capture} via C++ runtime (TestCudaGraphStrategyCpp), RTX-native override when set_cudagraphs_mode(True) is combined with a strategy, repeated inference, and invalid-value rejection (TestCudaGraphStrategyInvalidValue).

Regression:

$ pytest runtime/test_005_dynamic_allocation.py runtime/test_002_cudagraphs_cpp.py
================= 9 passed =================

Model-level: e2e ResNet18 compilation + inference via the C++ runtime path with each {lazy, eager, none} strategy and with runtime cache warm-roundtrip (added in tests/py/dynamo/models/).

Standard TensorRT build

Wheel: torch_tensorrt-2.12.0.dev0+612556ba0 (built with plain python3 setup.py bdist_wheel, TensorRT 10.16.0).

Import smoke:

• ENABLED_FEATURES.tensorrt_rtx = False
• ABI_VERSION = 9
• SERIALIZATION_LEN = 11 (no RTX-only slots in the FlattenedState)
• RTX-only index accessors are not registered by register_jit_hooks.cpp

$ pytest runtime/test_000_runtime_cache.py \
         runtime/test_001_dynamic_shapes_kernel_strategy.py \
         runtime/test_001_cuda_graph_strategy.py
================= 7 passed, 31 skipped =================

The 7 passes on standard TRT are:

• TestNonRTXUnchanged — confirms the existing Python-runtime paths are unaffected (runtime cache / timing cache behavior, no runtime_config member) — 2 tests.
• TestDynamicShapesKernelStrategyNonRTX::test_setting_ignored_on_non_rtx — confirms the new dynamic_shapes_kernel_specialization_strategy setting is silently ignored on non-RTX.
• TestCudaGraphStrategySettings::{test_default_value, test_settable_values} — CompilationSettings accepts the new fields on any backend.
• TestDynamicShapesKernelStrategyCppInvalidValue::test_invalid_strategy_raises + TestCudaGraphStrategyInvalidValue::test_invalid_strategy_raises — unknown strategy names are rejected at _pack_engine_info time even on non-RTX (validation is cross-platform; the engine-info serialization slots themselves are RTX-only).

The 31 skips all carry clean messages (sample):

SKIPPED [1] runtime/test_000_runtime_cache.py:68: Runtime cache is only available with TensorRT-RTX
SKIPPED [1] runtime/test_001_dynamic_shapes_kernel_strategy.py:58: Dynamic shapes kernel specialization strategy requires TensorRT-RTX
SKIPPED [1] runtime/test_001_cuda_graph_strategy.py:63:  CUDA graph strategy is a TensorRT-RTX feature

Regression on standard TRT:

$ pytest runtime/test_005_dynamic_allocation.py runtime/test_002_cudagraphs_cpp.py
================= 9 passed =================

Summary

Backend	Wheel	New / merged suites	Regression
TensorRT-RTX 1.4.0.76	torch_tensorrt_rtx-2.12.0.dev0+612556ba0	35 passed, 3 skipped	9 passed
Standard TensorRT 10.16.0	torch_tensorrt-2.12.0.dev0+612556ba0	7 passed, 31 skipped	9 passed

0 failures on either backend; all RTX-gated suites skip cleanly with descriptive messages on the standard TRT build.

[tp5uiuc]

Gated by #4164 which needs to merge first

[tp5uiuc]

Remove this comment.

[tp5uiuc]

You might want to set the allocation strategy here too.

  this->exec_ctx =
      make_trt(cuda_engine->createExecutionContext(

this->resource_allocation_strategy == TRTEngine::ResourceAllocationStrategy::kDynamic ? nvinfer1::ExecutionContextAllocationStrategy::kUSER_MANAGED : nvinfer1::ExecutionContextAllocationStrategy::kSTATIC));

See if

resource_allocation_strategy == ResourceAllocationStrategy::kDynamic
          ? nvinfer1::ExecutionContextAllocationStrategy::kUSER_MANAGED
          : nvinfer1::ExecutionContextAllocationStrategy::kSTATIC

can be extracted out as a common const variable and set both TRT_HAS_IRUNTIME_CONFIG and non TRT_HAS_IRUNTIME_CONFIG with it.