Extracted from a Pulsar Slack thread:
Another detail worth noting is that the Pulsar codebase doesn't have a clearly established model for handling concurrency. Without a documented model, it's hard to evaluate whether any given piece of code conforms to it. For comparison, in Netty there's a clear rule: all handling on the IO thread should be non-blocking, which by extension means avoiding synchronization and locks on that path. We don't strictly follow rules like this in Pulsar code. Modern JVMs (and hardware) are remarkably good at optimizing synchronized code, so in practice this hasn't been a blocker for high performance — but it does make reasoning about correctness harder than it needs to be.
Guidelines that would benefit from being written down include which work belongs on the network connection event loop versus other threads, how the various thread pools are intended to be used and what kind of work belongs on each, how threads are expected to hand off state to each other, when the completion thread of a CompletableFuture should be switched to another thread (and which one), concurrency limits for asynchronous tasks, and preferring the single-writer principle in designs to avoid concurrent state mutations. None of this is currently documented. I think LLMs open up real opportunities here: once we've defined a concurrency model, it becomes much more tractable to "lift and shift" the codebase toward it and enforce the rules consistently, rather than relying on each contributor to rediscover the conventions case by case.
A related area that should be tied into this model is backpressure — how the system avoids accepting more work than it can handle, particularly memory-wise. Regarding memory backpressure in Pulsar specifically, there's some existing documentation in PIP-442: Existing broker memory management. That section covers the memory side and is a good starting point, though backpressure more broadly (including non-memory dimensions) would benefit from being documented besides defining the concurrency model.
Extracted from a Pulsar Slack thread: