app/vlinsert/syslog: add an ability to override the default list of _msg fields for syslog ingestion protocol

[vadimalekseev]

It might have been better not to add a new flag and to keep only cef.name and message as default _msg fields,
But since the syslog format has structured fields that can include more detailed messages, like cef.extension.msg, I think adding a flag to control that behavior would be a better idea. Plus, it lines up with how you can override the _msg field list in other input methods.

Updates #1362

[vadimalekseev]

As I understand it, cef.extension.msg is an optional field with a more detailed description of the log entry, while cef.name is the event name and it's present in every CEF-formatted log entry.

[func25]

while cef.name is the event name and it's present in every CEF-formatted log entry.

I think we shouldn't invent our own rule, using CEF name as the default message field seems unexpected(?). AFAIK Elasticsearch, Logstash, NXLog, etc map CEF msg to message.

[vadimalekseev]

So, which field do you suggest using as the default?

As I mentioned above, cef.extension.msg is an optional field with details, while cef.name is the same thing as cef.extension.msg, but without the details and it is present in every log:

name is a string representing a human-readable and understandable description of the event.

src: https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.3/cef-implementation-standard/Content/CEF/Chapter%201%20What%20is%20CEF.htm

msg - An arbitrary message giving more details about the event

src: https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.3/cef-implementation-standard/Content/CEF/Chapter%202%20ArcSight%20Extension.htm#_Toc494359739

So, if we pick the cef.extension.msg field as the default, users will probably have to change that flag, but we want to avoid this to simplify usage of VictoriaLogs.

I think relying on cef.name is a more universal solution for typical scenarios. If the user wants to change the _msg field, we provide the new flag for that.

[func25]

Yeah... I understand. My point is that using cef.name (kind of title) as the default is not a universal convention in other systems, so it may be unexpected.

So, which field do you suggest using as the default?

Probably message,cef.extension.msg (since they are "message").

Also a user may see some logs with a cef.name field if message there and some logs without it since cef.name become _msg. If they explicitly configure name, then they understand their own logs better.

About adding the flag, I have no strong concerns. Either adding it now or waiting until users ask for it is fine, since message field is already one of the most customizable fields across the project

[vadimalekseev]

My point is that using cef.name (kind of title) as the default is not a universal convention in other systems, so it may be unexpected.

I see your point and agree. Just found that ELK-stack has common-schema convention: https://www.elastic.co/elasticsearch/common-schema

tldr: they rename similar fields to a universal format, like: src.ip, src_ip, source_ip -> source.ip. Or CEF's msg -> message. I think this is not the same as our _msg, since Elasticsearch does not have the concept of a _msg field the way it works in VictoriaLogs

[cubic-dev-ai]

No issues found across 3 files