✨[Feature] Multi-Framework Runner Support in tools

[peterkisfaludi]

Is your feature request related to a problem? Please describe.
The current tools/llm directory only supports text-generation models (LLMs and VLMs). The transformer architecture now underpins a much wider range of model families — diffusion models, speech models, encoder models, video generation — and users optimizing those models with Torch-TRT have no equivalent tooling. Each model type requires different I/O handling, evaluation metrics, and benchmarking methodology that can't be shoehorned into run_llm.py.

Describe the solution you'd like
Extend the tools/ directory with one entry point per task strategy, alongside the existing llm/ directory, each with its own run_.py entry point and benchmarking logic:

tools/
llm/ # existing: LLMs and VLMs
diffusion/ # new
audio/ # new
encoder/ # new
video/ # new
neural_operator/ #new

The model types to add:

Each runner should expose the same --benchmark flag and report latency/throughput in
model-appropriate units (tokens/s for text, images/s for diffusion, real-time factor for
audio).

Describe alternatives you've considered

• Extend run_llm.py with --model-type — a single script becomes unwieldy; diffusion models have fundamentally different I/O (image tensors, noise schedules, guidance scale) that doesn't map onto the LLM text-generation loop.
• Leave each model type to users — this is the status quo and means there is no canonical Torch-TRT path for non-LLM workloads, even though TRT has published optimizations for all of these families (diffusion, BERT, Whisper).
• Only add diffusion — diffusion models are the most visible gap, but ASR (Whisper) and encoder models (BERT) are equally well-established TensorRT use cases with existing NVIDIA blog coverage and deserve first-class support.

Additional context

[narendasan]

What is the intended goal for this tool? When we wrote run_llm we wanted to achieve two things, we wanted to test out experimental LLM passes like static kv cache insertion outside of the main torch-tensorrt tree and assess the performance of LLM models in torch-trt. But our expectation was that this is mostly a developer tool and that for the most part people would be writing python scripts for their particular models.

Static kv caching is now supported oob in transformers and we have done prior work on diffusion models like flux outside of run_llm: https://developer.nvidia.com/blog/double-pytorch-inference-speed-for-diffusion-models-using-torch-tensorrt/ and now we are working on VLAs also outside of run_llm, so would we want just a generic perf / acc tool for arbitrary huggingface models?

Generally you can compile most models from hf transformer / diffusers by compiling submodules: https://pytorch.org/blog/torch-compile-and-diffusers-a-hands-on-guide-to-peak-performance/ https://huggingface.co/docs/transformers/perf_torch_compile

Do we just want a CLI that produces the compiled exported program for a HF tag?

[peterkisfaludi]

Yes the main goal would be to be able to run any (or the topN important) HF models E2E, using torch-trt.

Do we just want a CLI that produces the compiled exported program for a HF tag?

Basically yes: here is a HF model, and I want to run it E2E with torch-trt (assuming the user does not want to deal with how to connect engines together etc)

[narendasan]

@peterkisfaludi can you take a look at #4223 and let me know if that was a long the lines of what you were thinking about?