❓ [Question] How can you set the output names when converting to serialized engine?

[peterbjorgensen]

❓ Question

I am converting my model to a serialized engine using (model is a torch.nn.Module instance)

program = torch_tensorrt.dynamo.trace(model, arg_inputs=example_inputs)
engine_bytes: bytes = torch_tensorrt.dynamo.convert_exported_program_to_serialized_trt_engine(
        program,
        arg_inputs=example_inputs,
        require_full_compilation=True,
        immutable_weights=True,
        tiling_optimization_level="full",
        use_python_runtime=False,
        enabled_precisions={torch.float16, torch.bfloat16, torch.float32},
        use_explicit_typing=False,
        optimization_level=5,
        version_compatible=True,
        hardware_compatible=True,
)

The input names of the serialized engine are named after the inputs of the nn.Module, but the output names are just named output0, output1 ...

How can I set these names to something more useful? I need certain names for the model to work in our production environment.

[peterbjorgensen]

I see it is hardcoded here :(

TensorRT/py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py

Line 908 in 18ab5cf

name = f"output{i}"

[narendasan]

yeah we need a fixed naming convention that allows us to map ordered I/O from the pytorch side to names because technically TensorRT does not guarantee binding ordering. For convert_exported_program_to_serialized_trt_engine its conceivable to add an argument that allow users to provide a map for how they want their bindings to be named since it wont get run by torch-tensorrt's runtime.