Skip to content

Refactor prism and go sdk logging and clean up messages#36484

Merged
shunping merged 4 commits intoapache:masterfrom
shunping:prism-logging-setup
Oct 14, 2025
Merged

Refactor prism and go sdk logging and clean up messages#36484
shunping merged 4 commits intoapache:masterfrom
shunping:prism-logging-setup

Conversation

@shunping
Copy link
Copy Markdown
Collaborator

@shunping shunping commented Oct 12, 2025

A cleaner log by default is more user-friendly. It also helps to check test output without losing in the swarm of messages.

Before:

2025/10/11 23:56:26 INFO Serving JobManagement endpoint=localhost:52057
2025/10/11 23:56:26 starting Loopback server at 127.0.0.1:52058
2025/10/11 23:56:26 components:
... (LONG proto which is hard to parse)
2025/10/11 23:56:26 Prepared job with id: job-001 and staging token: job-001
2025/10/11 23:56:26 Staged binary artifact with token: job-001
2025/10/11 23:56:26 Submitted job: job-001
2025/10/11 23:56:26  (): starting job-001[go-testtimers_eventtime_bounded-632]
2025/10/11 23:56:26  (): running job-001[go-testtimers_eventtime_bounded-632]
2025/10/11 23:56:26 Job[job-001] state: RUNNING
2025/10/11 23:56:26 starting worker job-001[go-testtimers_eventtime_bounded-632]_go
2025/10/11 23:56:26 INFO log from SDK worker worker.ID=job-001[go-testtimers_eventtime_bounded-632]_go worker.endpoint=localhost:52057 sdk.transformID="" sdk.location=/Users/shunping/Projects/beam-dev-go-prism/sdks/go/pkg/beam/core/runtime/harness/harness.go:104 sdk.time=2025-10-12T03:56:26.673Z sdk.msg="Failed to parse element_processing_timeout: time: invalid duration \"\", there will be no timeout for processing an element in a PTransform operation"
2025/10/11 23:56:26 stopping worker job-001[go-testtimers_eventtime_bounded-632]_go
2025/10/11 23:56:26  (): pipeline completed job-001[go-testtimers_eventtime_bounded-632]
2025/10/11 23:56:26  (): terminating job-001[go-testtimers_eventtime_bounded-632]
2025/10/11 23:56:26 control response channel closed

After:

time=2025-10-11T23:53:50.182-04:00 level=INFO msg="Serving JobManagement" endpoint=localhost:51959
time=2025-10-11T23:53:50.183-04:00 level=INFO msg="starting Loopback server at 127.0.0.1:51960"
time=2025-10-11T23:53:50.187-04:00 level=INFO msg="Prepared job with id: job-001 and staging token: job-001"
time=2025-10-11T23:53:50.189-04:00 level=INFO msg="Staged binary artifact with token: job-001"
time=2025-10-11T23:53:50.189-04:00 level=INFO msg="Submitted job: job-001"
time=2025-10-11T23:53:50.189-04:00 level=INFO msg="starting job-001[go-testtimers_eventtime_bounded-151]"
time=2025-10-11T23:53:50.189-04:00 level=INFO msg="running job-001[go-testtimers_eventtime_bounded-151]"
time=2025-10-11T23:53:50.189-04:00 level=INFO msg="Job[job-001] state: RUNNING"
time=2025-10-11T23:53:50.302-04:00 level=INFO msg="[SDK] pipeline completed job-001[go-testtimers_eventtime_bounded-151]"
time=2025-10-11T23:53:50.302-04:00 level=INFO msg="[SDK] terminating job-001[go-testtimers_eventtime_bounded-151]"
time=2025-10-11T23:53:50.302-04:00 level=INFO msg="[SDK] stopping worker job-001[go-testtimers_eventtime_bounded-151]_go"

@shunping shunping requested a review from liferoad October 12, 2025 04:38
@shunping shunping marked this pull request as ready for review October 12, 2025 04:38
@github-actions
Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label go.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@liferoad
Copy link
Copy Markdown
Contributor

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively refactors the logging system for Prism and the Go SDK to use Go's structured logging library, slog. The changes centralize logging configuration, reduce log verbosity by default, and improve the structure and clarity of log messages, aligning with the goal of a more user-friendly logging experience. The refactoring is well-executed. I have a couple of suggestions to further improve the clarity and consistency of the new logging output.

Comment on lines +226 to +235
case "dev":
logHandler =
devslog.NewHandler(loggerOutput, &devslog.Options{
TimeFormat: "[" + time.RFC3339Nano + "]",
StringerFormatter: true,
HandlerOptions: handlerOpts,
StringIndentation: false,
NewLineAfterLog: true,
MaxErrorStackTrace: 3,
})
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The configuration for the dev log kind uses devslog, which produces a human-readable format (e.g., [<timestamp>] INFO ...). However, the 'After' log example in the pull request description shows a key=value format, which corresponds to the text log kind (slog.TextHandler). Since the default log_kind for Prism is dev, this discrepancy could be confusing. To align the default behavior with the example, you might consider changing the default log_kind flag in sdks/go/cmd/prism/prism.go to text, or updating the PR description to show an example of the devslog output.

Comment on lines +142 to +151
var b strings.Builder
if resp.GetTime() != "" {
fmt.Fprintf(&b, "(time=%v)", resp.GetTime())
}
if resp.GetMessageId() != "" {
fmt.Fprintf(&b, "(id=%v)", resp.GetMessageId())
}
b.WriteString(resp.GetMessageText())
text := b.String()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using strings.Builder is efficient, but the resulting log format (time=...)(id=...)message can be a bit hard to read, especially with no space between the parts. For improved readability and a more standard log appearance, consider building the message from parts with spaces in between. A slice-based approach can be more readable here, and the performance difference is likely negligible in this context.

Suggested change
var b strings.Builder
if resp.GetTime() != "" {
fmt.Fprintf(&b, "(time=%v)", resp.GetTime())
}
if resp.GetMessageId() != "" {
fmt.Fprintf(&b, "(id=%v)", resp.GetMessageId())
}
b.WriteString(resp.GetMessageText())
text := b.String()
var parts []string
if resp.GetTime() != "" {
parts = append(parts, fmt.Sprintf("time=%v", resp.GetTime()))
}
if resp.GetMessageId() != "" {
parts = append(parts, fmt.Sprintf("id=%v", resp.GetMessageId()))
}
if resp.GetMessageText() != "" {
parts = append(parts, resp.GetMessageText())
}
text := strings.Join(parts, " ")

@shunping shunping merged commit 2b9827b into apache:master Oct 14, 2025
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants