Skip to content

add feature gate for tensorrt plugin#3518

Merged
lanluo-nvidia merged 9 commits intomainfrom
lluo/add_gate
May 24, 2025
Merged

add feature gate for tensorrt plugin#3518
lanluo-nvidia merged 9 commits intomainfrom
lluo/add_gate

Conversation

@lanluo-nvidia
Copy link
Copy Markdown
Collaborator

Description

Tensorrt 10.3.0 does not support tensorrt plugin which causing failures.

Fixes # (issue)

Type of change

Please delete options that are not relevant and/or add your own.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@github-actions github-actions Bot added component: conversion Issues re: Conversion stage component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels May 12, 2025
@github-actions github-actions Bot requested a review from gs-olive May 12, 2025 22:53
Comment thread py/torch_tensorrt/_features.py Outdated
Comment thread py/torch_tensorrt/_features.py Outdated
Comment thread py/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin.py Outdated
Comment thread py/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin_converter.py Outdated
@shchoi00
Copy link
Copy Markdown

shchoi00 commented May 15, 2025

+1

My environment

jetpack 6.2
torch 2.7
tensorrt 10.3
cuda 12.6

$ python3 -c "import torch_tensorrt"
Unable to import quantization op. Please install modelopt library (https://github.com/NVIDIA/TensorRT-Model-Optimizer?tab=readme-ov-file#installation) to add support for compiling quantized models
TensorRT-LLM is not installed. Please install TensorRT-LLM or set TRTLLM_PLUGINS_PATH to the directory containing libnvinfer_plugin_tensorrt_llm.so to use converters for torch.distributed ops
Traceback (most recent call last):
File "", line 1, in
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/init.py", line 125, in
from torch_tensorrt.runtime import * # noqa: F403
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/runtime/init.py", line 1, in
from torch_tensorrt.dynamo.runtime import ( # noqa: F401
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/init.py", line 10, in
from ._compiler import (
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/_compiler.py", line 15, in
from torch_tensorrt.dynamo import _defaults, partitioning
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/partitioning/init.py", line 1, in
from ._adjacency_partitioner import partition as fast_partition
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/partitioning/_adjacency_partitioner.py", line 20, in
from torch_tensorrt.dynamo.conversion._ConverterRegistry import (
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/conversion/init.py", line 1, in
from . import (
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/conversion/plugins/init.py", line 1, in
from torch_tensorrt.dynamo.conversion.plugins._custom_op import custom_op
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/conversion/plugins/_custom_op.py", line 6, in
from torch_tensorrt.dynamo.conversion.plugins._generate_plugin import generate_plugin
File "/home/ircv13/.local/lib/python3.10/site-packages/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin.py", line 5, in
import tensorrt.plugin as trtp
ModuleNotFoundError: No module named 'tensorrt.plugin'

@narendasan
Copy link
Copy Markdown
Collaborator

@shchoi00 yes this is support of Jetson as part of our build system overhaul

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin_converter.py	2025-05-20 15:32:36.323051+00:00
+++ /home/runner/work/TensorRT/TensorRT/py/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin_converter.py	2025-05-20 15:33:02.313006+00:00
@@ -36,10 +36,11 @@
    except ImportError as e:
        raise RuntimeError(
            "Unable to import TensorRT plugin. Please install TensorRT plugin library (https://github.com/NVIDIA/TensorRT-plugin-library?tab=readme-ov-file#installation) to add support for compiling quantized models"
        )
    from tensorrt.plugin._lib import QDP_REGISTRY
+
    torch_target = getattr(getattr(torch.ops, namespace), op_name)
    overload_str = overload if overload else ""
    overload_name = overload_str if overload else "default"
    torch_overload = getattr(torch_target, overload_name)
    assert (

Comment thread py/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin.py Outdated
Comment thread py/torch_tensorrt/dynamo/conversion/plugins/_generate_plugin_converter.py Outdated
Copy link
Copy Markdown
Collaborator

@narendasan narendasan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor change then looks good

Comment thread py/torch_tensorrt/_features.py Outdated
@github-actions github-actions Bot added the component: converters Issues re: Specific op converters label May 20, 2025
Comment thread py/torch_tensorrt/dynamo/conversion/impl/unsqueeze.py
@lanluo-nvidia lanluo-nvidia merged commit 5dbf2b9 into main May 24, 2025
81 checks passed
@lanluo-nvidia lanluo-nvidia deleted the lluo/add_gate branch January 10, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants