Skip to content

Performance regression: v5 is 2–7× slower than v3.8.0 for local evaluation #198

@gagantrivedi

Description

@gagantrivedi

Benchmarked flagsmith 5.0.1 vs 3.8.0 against the same environment (edge API, enable_local_evaluation=True, 262 features, single-threaded, after 200-iter warmup).

p50 per call

Operation v3.8.0 v5.0.1 Regression
get_identity_flags 0.39 ms 0.82 ms 2.1×
get_environment_flags 0.11 ms 0.79 ms 7.4×
is_feature_enabled (cached Flags) 0.08 µs 0.08 µs

Throughput drops from ~2,570 → ~1,215 identity evals/sec per core. Latency variance also grew (stdev ~10 µs → ~220 µs).

Repro

from time import perf_counter
from flagsmith import Flagsmith

client = Flagsmith(
    environment_key="<server-side-key>",
    api_url="https://edge.api.flagsmith.com/api/v1/",
    enable_local_evaluation=True,
    environment_refresh_interval_seconds=3600,
)
client.update_environment()

# warmup
for _ in range(200):
    client.get_identity_flags(identifier="anonymous", traits={"venue_id": "12345"})

for n in (100, 1_000, 10_000):
    t0 = perf_counter()
    for _ in range(n):
        client.get_identity_flags(identifier="anonymous", traits={"venue_id": "12345"})
    elapsed = perf_counter() - t0
    print(f"{n:>6} iters: {elapsed*1000:8.1f} ms ({elapsed/n*1e6:7.2f} µs/call)")

Environment: flag-engine 10.0.1, Python 3.12, macOS.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions