feat(rocksdb): support ToplingDB provider by WaterWhisperer · Pull Request #3024 · apache/hugegraph

WaterWhisperer · 2026-05-12T16:20:01Z

Purpose of the PR

Add ToplingDB support as an optional RocksDB-compatible storage path.

This PR rebases and prepares the existing ToplingDB adaptation work for review on the current master. ToplingDB is integrated as an optional provider for RocksDB-based storage, allowing users to enable RocksDB/ToplingDB configuration and the ToplingDB Web UI while preserving the standard RocksDB path.

Main Changes

Add hugegraph-rocksdb-provider module
Update HugeGraph Server RocksDB integration
Add ToplingDB preload support in server dist
Add PD / Store ToplingDB integration
Add Maven / CI / dependency metadata updates
Add ToplingDB docs and specs

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).
Need tests and can be verified as follows:
- Built packaged dist artifacts for Server / PD / Store
- Verified packaged rocksdbjni-8.10.2-SNAPSHOT.jar contains ToplingDB resources including org/rocksdb/SidePluginRepo.class
- Ran Server with ToplingDB enabled
- Ran Server without rocksdb.option_path
- Ran PD / Store with ToplingDB config

Does this PR potentially affect the following parts?

Dependencies (add/update license info & regenerate_known_dependencies.sh)
Modify configurations
The public API
Other affects (typed here)
- Adds native library preload logic for ToplingDB-enabled RocksDB JNI runtime.
- Adds optional ToplingDB Web UI support.
- Adds RocksDB provider SPI and routes RocksDB opening through the provider loader.
Nope

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

…b packages

…ckage

…th the HG default configuration

…TP Server

…positioning

…ple calls

…e provider

…and pd

# Conflicts: # install-dist/scripts/dependency/known-dependencies.txt

Copilot

Pull request overview

Adds an SPI-based RocksDB provider layer to support ToplingDB as an optional RocksDB-compatible backend, including config options for Topling YAML parameters and optional HTTP UI, plus distro/CI packaging changes to preload native libraries and web assets.

Changes:

Introduce hugegraph-rocksdb-provider module with provider discovery/selection and Standard/Topling providers.
Route Server/Store/PD RocksDB open/close through the provider loader and add rocksdb.option_path / rocksdb.open_http configs.
Add preload scripts/resources and CI workflow updates to support ToplingDB artifacts and runtime prerequisites.

Reviewed changes

Copilot reviewed 53 out of 61 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
pom.xml	Adds provider module, sets `rocksdbjni.version`, and excludes extracted Topling HTML/CSS from checks.
install-dist/scripts/dependency/known-dependencies.txt	Updates known dependency inventory for new/updated jars (incl. snapshot rocksdbjni).
install-dist/release-docs/licenses/LICENSE-rocksdbjni-8.10.2-SNAPSHOT.txt	Adds license text for the snapshot rocksdbjni distribution.
install-dist/release-docs/LICENSE	Updates LICENSE entries for new/updated dependencies and rocksdbjni source.
hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java	Switches RocksDB open/close to `RocksDBProviderLoader` and passes option/http params.
hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBOptions.java	Adds `rocksdb.option_path` and `rocksdb.open_http` options for store.
hugegraph-store/hg-store-rocksdb/pom.xml	Replaces direct rocksdbjni dependency with `hugegraph-rocksdb-provider`.
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/metrics/RocksDBMetricsConst.java	Removes metrics constants no longer available in RocksDB 8.x.
hugegraph-store/hg-store-dist/src/assembly/static/conf/rocksdb_store.yaml	Adds sample Topling/RocksDB YAML config for store.
hugegraph-store/hg-store-dist/src/assembly/static/conf/application-pd.yml	Documents new RocksDB option_path/open_http config fields.
hugegraph-store/hg-store-dist/src/assembly/static/bin/util.sh	Adds helper to locate server dist dir for preload integration.
hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh	Sources server preload script when available before starting store.
hugegraph-store/hg-store-core/pom.xml	Updates jraft-core version.
hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/rocksdb/RocksDBSessionsTest.java	Formatting-only update in test code.
hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStore.java	Formatting-only change in signature wrapping.
hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java	Opens RocksDB via provider loader and gates Topling HTTP to GRAPH_STORE.
hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBOptions.java	Adds `rocksdb.option_path` and `rocksdb.open_http` options for server.
hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/OpenedRocksDB.java	Closes RocksDB via provider loader to enable provider-specific cleanup.
hugegraph-server/hugegraph-rocksdb/pom.xml	Replaces direct rocksdbjni dependency with `hugegraph-rocksdb-provider`.
hugegraph-server/hugegraph-dist/src/assembly/travis/run-unit-test.sh	Installs RocksDB/Topling runtime prereqs before running unit tests.
hugegraph-server/hugegraph-dist/src/assembly/travis/run-core-test.sh	Installs RocksDB/Topling runtime prereqs before running core tests.
hugegraph-server/hugegraph-dist/src/assembly/travis/run-api-test.sh	Installs RocksDB/Topling runtime prereqs before API tests.
hugegraph-server/hugegraph-dist/src/assembly/travis/install-rocksdb.sh	New helper to preload Topling libs/resources in CI.
hugegraph-server/hugegraph-dist/src/assembly/travis/install-deps.sh	New helper to install native deps (liburing/libaio/jemalloc) in CI.
hugegraph-server/hugegraph-dist/src/assembly/static/conf/graphs/rocksdb_server.yaml	Adds sample Topling/RocksDB YAML config for server.
hugegraph-server/hugegraph-dist/src/assembly/static/conf/graphs/hugegraph.properties	Documents new `rocksdb.option_path` / `rocksdb.open_http` properties.
hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh	Sources preload script before starting server.
hugegraph-server/hugegraph-dist/src/assembly/static/bin/preload-topling.sh	New preload entrypoint to extract/preload native libs + resources.
hugegraph-server/hugegraph-dist/src/assembly/static/bin/init-store.sh	Sources preload script before store initialization.
hugegraph-server/hugegraph-dist/src/assembly/static/bin/common-topling.sh	New shared functions for extracting/preloading libs/resources and compat workarounds.
hugegraph-server/hugegraph-core/pom.xml	Updates jraft version property.
hugegraph-rocksdb-provider/src/main/resources/META-INF/services/org.apache.hugegraph.rocksdb.provider.RocksDBProvider	Registers Standard and Topling providers for SPI discovery.
hugegraph-rocksdb-provider/src/main/java/org/apache/hugegraph/rocksdb/provider/ToplingRocksDBProvider.java	Implements Topling SidePluginRepo reflection, YAML validation, and optional HTTP start.
hugegraph-rocksdb-provider/src/main/java/org/apache/hugegraph/rocksdb/provider/StandardRocksDBProvider.java	Implements standard RocksDB open/close behavior.
hugegraph-rocksdb-provider/src/main/java/org/apache/hugegraph/rocksdb/provider/RocksDBProviderLoader.java	Adds provider discovery, priority selection, and open/close delegates.
hugegraph-rocksdb-provider/src/main/java/org/apache/hugegraph/rocksdb/provider/RocksDBProvider.java	Defines SPI interface for provider open/close operations.
hugegraph-rocksdb-provider/src/main/java/org/apache/hugegraph/rocksdb/provider/AbstractRocksDBProvider.java	Adds base implementation for provider lifecycle and close handling.
hugegraph-rocksdb-provider/pom.xml	New module POM and dependency declarations for provider layer.
hugegraph-pd/hg-pd-dist/src/assembly/static/conf/rocksdb_pd.yaml	Adds sample Topling/RocksDB YAML config for PD.
hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml	Documents optional rocksdb option-path/open-http config in PD dist.
hugegraph-pd/hg-pd-dist/src/assembly/static/bin/util.sh	Adds helper to locate server dist dir for preload integration.
hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh	Sources server preload script when available before starting PD.
hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/store/HgKVStoreImpl.java	Opens/closes RocksDB via provider loader and threads option/http configs through.
hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java	Adds Spring-configurable `rocksdb.option-path` and `rocksdb.open-http` properties.
hugegraph-pd/hg-pd-core/pom.xml	Updates dependencies to use provider module and updates jraft version.
hugegraph-pd/hg-pd-cli/pom.xml	Updates jraft version.
docs/toplingdb/toplingdb.md	Adds ToplingDB feature/design/configuration documentation.
docs/toplingdb/toplingdb-troubleshooting.md	Adds troubleshooting guide for YAML/HTTP/lock issues.
docs/toplingdb/toplingdb-security.md	Adds security hardening recommendations for ToplingDB usage.
docs/toplingdb/toplingdb-operations.md	Adds operational guidance (monitoring/tuning/upgrade).
.specs/hugegraph-server/ToplingDB/task.md	Adds implementation task breakdown for ToplingDB work.
.specs/hugegraph-server/ToplingDB/requirements.md	Adds requirements spec for ToplingDB support.
.specs/hugegraph-server/ToplingDB/design.md	Adds design spec describing preload and open logic.
.github/workflows/server-ci.yml	Updates CI to package artifacts, install deps, and enable GitHub Packages access.
.github/workflows/pd-store-ci.yml	Updates CI packaging/deps/preload flow and enables GitHub Packages access.
.github/workflows/licence-checker.yml	Enables stage repo usage in workflow env.
.github/workflows/commons-ci.yml	Enables stage repo usage and adds GitHub Packages env vars.
.github/workflows/codeql-analysis.yml	Enables stage repo usage and adds GitHub Packages permissions/env vars.
.github/workflows/cluster-test-ci.yml	Enables stage repo usage and adds GitHub Packages env vars.
.github/workflows/check-dependencies.yml	Enables stage repo usage and adds GitHub Packages env vars.
.github/configs/settings.xml	Adds GitHub Packages repo for ToplingDB snapshot dependency resolution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        try {
+            // Delegate to provider-specific implementation
+            RocksDB rocksDB = doOpenRocksDB(options, dataPath);
+
+            return rocksDB;
+        } catch (Exception e) {
+            LOG.error("Failed to open RocksDB with provider: {} at path: {}",
+                      getProviderName(), dataPath, e);
+            throw e;
+        }


+        try {
+            // Delegate to provider-specific implementation
+            RocksDB rocksDB = doOpenRocksDB(dbOptions, dataPath, cfDescriptors, cfHandles);
+
+            return rocksDB;
+        } catch (Exception e) {
+            LOG.error("Failed to open RocksDB with column families using provider: {} at path: {}",
+                      getProviderName(), dataPath, e);
+            throw e;
+        }


+        try {
+            // Delegate to provider-specific implementation
+            RocksDB rocksDB =
+                    doOpenRocksDB(dbOptions, dataPath, cfDescriptors, cfHandles, optionPath,
+                                  openHttp);
+            return rocksDB;
+        } catch (Exception e) {
+            throw e;
+        }


+        } catch (Exception e) {
+            LOG.warn("Standard RocksDB is not available: {}", e.getMessage());


+    public void initialize() {
+        try {
+            // Load RocksDB native library
+            RocksDB.loadLibrary();
+            LOG.debug("Standard RocksDB library loaded successfully");
+        } catch (Exception e) {
+            LOG.error("Failed to load standard RocksDB library", e);
+            throw new RuntimeException("Failed to initialize standard RocksDB provider", e);
+        }


+            // Find provider with highest priority
+            RocksDBProvider selected = null;
+            int highestPriority = Integer.MIN_VALUE;
+
+            for (RocksDBProvider provider : providerCache.values()) {
+                if (provider.isAvailable() && provider.getPriority() > highestPriority) {
+                    selected = provider;
+                    highestPriority = provider.getPriority();
+                }
+            }


    @Value("${grpc.host}")
    private String host;

+    @Value("${rocksdb.option-path: ''}")


+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-api</artifactId>
+            <version>1.7.5</version>


+            // Prepare column family names for JSON
+            List<String> cfNames = new java.util.ArrayList<>();
+            for (ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
+                cfNames.add(new String(cfDescriptor.getName()));


+    public synchronized void loadProviders() {
+        if (loaded) {
+            return;
+        }
+
+        LOG.info("Loading RocksDB providers via SPI...");
+
+        ServiceLoader<RocksDBProvider> serviceLoader = ServiceLoader.load(RocksDBProvider.class);
+
+        for (RocksDBProvider provider : serviceLoader) {
+            try {
+                if (provider.isAvailable()) {
+                    providerCache.put(provider.getProviderName(), provider);
+                    LOG.info("Loaded RocksDB provider: {} (priority: {})",
+                             provider.getProviderName(), provider.getPriority());
+                } else {
+                    LOG.warn("RocksDB provider {} is not available in current environment",
+                             provider.getProviderName());
+                }
+            } catch (Exception e) {
+                LOG.error("Failed to load RocksDB provider: {}", provider.getClass().getName(), e);
+            }
+        }
+
+        if (providerCache.isEmpty()) {
+            LOG.warn(
+                    "No RocksDB providers found! Make sure providers are properly registered in " +
+                    "META-INF/services");
+        } else {
+            LOG.info("Successfully loaded {} RocksDB provider(s): {}",
+                     providerCache.size(), providerCache.keySet());
+        }
+
+        loaded = true;


syslucas added 30 commits May 11, 2026 00:05

feat(backend): Add RocksDB Plus (ToplingDB) support

f6318a3

fix(server): Use reflection to increase compatibility with old rocksd…

b8de659

…b packages

fix(server): fix bugs when using reflection openDB

21e1c12

feat: add some scripts to unzip and preload toplingdb from the jar pa…

7a86917

…ckage

chore: fix CI bugs

2c17aae

chore: change env.USE_STAGE to true

794b7fa

chore: try to modify the permission to solve the 401 problem

f412b17

fix(server): solve the bug of importing topling package

df3e2fb

fix(CI): eliminate pollution of environment variables

ae4b34b

install deps in hstore-ci

8ccac08

add compatibility for libaio.so.1 for ubuntu-24.04

af667df

chore(server): modify the configuration file to keep it consistent wi…

653e499

…th the HG default configuration

feat(server): add the configuration open_http to start the Topling HT…

a61247f

…TP Server

chore(server): fix formatting issues in scripts

325dc94

chore(server): adopt review suggestions from AI robots

6c7dd0a

chore(server): add third-party package verification

a45b7bb

chore(server): add unified error capture in the script to facilitate …

2430cd1

…positioning

chore(server): extract redundant code from scripts into common functions

c6fba41

chore(server): enhance script robustness

879f7cc

chore(server): corrected resource copy logic

dfe2124

chore(server): avoid repeated expansion of LD_PRELOAD caused by multi…

e486432

…ple calls

chore(docs): add rocksdb-plus documentation

64ff108

chore(docs): update rocksdb-plus documentation

fc63ea9

chore(docs): add specs

2fa2cf6

chore(docs): fix lint

c7631ec

chore(docs): update maven settings.xml conf

27429f0

chore(docs): add token permission hint

961247d

fix: add YAML file validation

fd2c77d

fix: update ci scripts

8d7c7c2

fix(server): fix environment issues in the script

96dadc5

syslucas and others added 10 commits May 11, 2026 00:10

chore(store): update license

67f8334

fix(rocksdb-provider): remove the verification of the data path in th…

65d7a77

…e provider

fix(pd): fix dependency issues in CI

3704d0a

fix(pd): add install deps in pd-store ci

d75dbfd

fix(pd): fix environment variable issues in ci

57400d2

fix(pd): fix the snakeyaml dependency issue between rocksdb-provider …

435729d

…and pd

chore(rocksdb-provider): adopt the recommendations from ai review

7d13ae6

chore(docs): trim ToplingDB trailing whitespace

94c7f17

fix(pd): correct ToplingDB option path comment

56e667e

docs(toplingdb): align config examples with implementation

3a51167

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. dependencies Incompatible dependencies of package feature New feature pd PD module store Store module labels May 12, 2026

github-project-automation Bot added this to HugeGraph PD-Store Tasks May 12, 2026

github-project-automation Bot moved this to In progress in HugeGraph PD-Store Tasks May 12, 2026

WaterWhisperer marked this pull request as draft May 13, 2026 07:38

WaterWhisperer and others added 8 commits May 17, 2026 20:13

fix(ci): use current server dist name

f4c495c

Merge branch 'master' into toplingdb-rebase

4418643

Merge remote-tracking branch 'upstream/master' into toplingdb-rebase

f82a3ce

fix(toplingdb): address ci review issues

77acd32

fix(ci): skip ToplingDB native preload on non-Linux

368aef1

fix(ci): skip unsupported ToplingDB native preload

90b07f1

Merge remote-tracking branch 'upstream/master' into toplingdb-rebase

8799d62

# Conflicts: # install-dist/scripts/dependency/known-dependencies.txt

fix(ci): align dependency metadata after upstream merge

49637fc

WaterWhisperer marked this pull request as ready for review May 24, 2026 16:49

imbajin requested a review from Copilot May 27, 2026 10:19

Copilot started reviewing on behalf of imbajin May 27, 2026 10:19 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(rocksdb): support ToplingDB provider#3024

feat(rocksdb): support ToplingDB provider#3024
WaterWhisperer wants to merge 53 commits into
apache:masterfrom
WaterWhisperer:toplingdb-rebase

WaterWhisperer commented May 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		} catch (Exception e) {
		LOG.warn("Standard RocksDB is not available: {}", e.getMessage());

Conversation

WaterWhisperer commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the PR

Main Changes

Verifying these changes

Does this PR potentially affect the following parts?

Documentation Status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WaterWhisperer commented May 12, 2026 •

edited

Loading