Fix compound dictionary integer overflows

[ZX41R]

Fix two signed integer overflows in compound dictionary decoding that survived the partial fix in 4792c8e (#1438 / #1450).

What overflows

EnsureCompoundDictionaryInitialized (c/dec/decode.c:1556) iterates cursor in steps of 1 << block_bits, where block_bits is derived from addon->total_size. The post-4792c8e BROTLI_SAFE_ADD only clamps total_size <= INT_MAX = 2,147,483,647. For any total_size > 255 * (1 << 23) = 2,139,095,040, block_bits settles at 23 and the cursor reaches 255 * 8388608 = 2139095040; the next cursor += 1 << block_bits is 2139095040 + 8388608 = 2147483648, which exceeds INT_MAX. Under -fwrapv, cursor wraps to INT_MIN, the loop guard cursor < total_size stays true, and the next body executes addon->block_map[cursor >> block_bits] = (uint8_t)index at a negative index — a 1-byte heap write 36 bytes before the 480-byte BrotliDecoderCompoundDictionary allocation. The clamp commit 4792c8e does not protect this site because the loop's overflow threshold is lower than INT_MAX by 8,388,607.

The same address + length pattern appears in InitializeCompoundDictionaryCopy (c/dec/decode.c:1573) — address + length is signed int addition with no overflow check. In normal flow this is shadowed by the cursor-loop overflow above, but a fix that addresses only the cursor loop would expose this site.

What this patch changes

• EnsureCompoundDictionaryInitialized: change cursor to size_t, cast addon->total_size and chunk_offsets[index + 1] to size_t for comparison, cast 1 << block_bits to (size_t)1 << block_bits for the increment. Cursor arithmetic now lives entirely in the non-negative size_t domain.
• InitializeCompoundDictionaryCopy: replace addon->total_size < address + length with BROTLI_SAFE_ADD(int, address, length, &end) followed by addon->total_size < end. Reuses the helper introduced in 4792c8e.

Diff is +7 −5 in c/dec/decode.c. No public API change.

Verification

• Reproducer (UBSan, unpatched): decode.c:1568:12: runtime error: signed integer overflow: 2139095040 + 8388608 cannot be represented in type 'int'
• Reproducer (ASan + -fwrapv, unpatched): heap-buffer-overflow on address 0x...02dc, WRITE of size 1, located 36 bytes before 480-byte region allocated by AttachCompoundDictionary at c/dec/decode.c:1533.
• Patched build (this PR), same reproducer:

  $ ./poc_compound_dict_ubsan
  [+] encoded 10 bytes: 1b1f00f82500a2987841
  [+] attached compound dictionary: addressable size=2139095041 (honest), RSS commit ~1 page
  [+] feeding stream -- expect UBSan: signed integer overflow at decode.c:1568
  [!] decoder returned without UBSan abort: r=1, error_code=1 (_SUCCESS)
  exit=0
  

• ctest --output-on-failure on patched ASan+UBSan build: 100% tests passed, 0 tests failed out of 28.

Reachability

This is a hardening fix. The trigger requires an embedder that calls BrotliDecoderAttachDictionary with total_size > 2,139,095,040. Chromium's Compression Dictionary Transport caps each shared dictionary at 100 MiB, so that path is unaffected. The Java JNI wrapper caps individual attach calls below 1 GiB. The brotli CLI (c/tools/brotli.c) and the C API itself accept any size up to INT_MAX and are the structural exposure for this loop.

I am happy to add a libFuzzer harness (e.g., extending the harness in #1443 to cover this size range) if useful.

Reported under Google's OSS VRP for hardening of the unaddressed half of #1438.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[ZX41R]

Closing in favor of a cleaner replacement PR with corrected commit authorship.