fix: skip metadata-only SSE chunks instead of aborting stream in StreamingResponseAggregator#783
Open
anishesg wants to merge 1 commit into
Open
fix: skip metadata-only SSE chunks instead of aborting stream in StreamingResponseAggregator#783anishesg wants to merge 1 commit into
anishesg wants to merge 1 commit into
Conversation
…amingResponseAggregator ## Link to Issue or Description of Change Signed-off-by: anish k <ak8686@princeton.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
Closes: #782
Vertex AI occasionally emits a leading SSE chunk that contains only response metadata (
createTime,modelVersion,usageMetadata, etc.) with noCandidatesfield. This happens consistently withgemini-3-flash-preview+googleSearchgrounding and caused roughly 40–50% of streaming calls to fail with"empty response", even though the actual content arrived in subsequent chunks.The root cause is in
internal/llminternal/stream_aggregator.goinProcessResponse: whenlen(genResp.Candidates) == 0, the function calledyield(nil, fmt.Errorf("empty response"))and returned, aborting the iterator. The fix changes this to a silentreturn, allowing the aggregator to skip metadata-only chunks and continue processing subsequent chunks that carry real content. This matches the behavior ofadk-python'sPROGRESSIVE_SSE_STREAMINGpath which handles identical chunks gracefully.The unused
fmtimport was also removed as a consequence.Testing Plan
Unit Tests
TestMetadataOnlyChunkDoesNotAbortStreaminstream_aggregator_test.gothat feeds a metadata-only chunk (noCandidates) followed by a real content chunk and asserts the aggregated response contains the expected text without any error.Manual E2E Tests
The reproduction script from the issue (10 runs of
gemini-3-flash-preview+googleSearchwithStreamingModeSSE) now produces 10/10 successes with this patch applied, matching the behavior ofStreamingModeNoneandadk-python.Checklist
Fixes #782