Windows Event Log acquisition for Splunk using OpenTelemetry Collector

Windows Event Logs are a primary data source for SIEM systems and various setups can be used to collect these logs. One option is to configure Windows Event Forwarding to send the logs from the source computer to servers. From there, the logs are pushed to a SIEM using a 3rd party forwarding software.

Problem #

Splunk offers Universal or Heavy Forwarders for this task. They can monitor The Event Log channels, register new logs and send them to the index servers. However, fast moving channels, such as the Sysmon target performed poorly in a larger environment. I tried different configuration tweaks and adding hardware resources, but to no avail. The Universal Forwarder coldn’t keep up with the log volume, fell behind which ultimatle caused delayed and dropped events.

Solution #

The simplest solution is to avoid Splunks Event Log monitor. Most of the tools I tested did a better job of keeping up with high traffic Event Channels. But to integrate with Splunks provided Technical AddOns the format of the logs must be closely emulated.

Recently I tested the OpenTelemetry Collector and so far it is doing a good job at delivering the logs in a timely fashion. It monitors the provided Event Log Channel, sets metadata attributes as required by Splunks TA and delivers the logs to a HEC endpoint. I am using locally installed Universal Forwarders as HEC endpoint, but a load-balanced central receiver should also work.
Splunk provides their own OpenTelemetry Collector distribution that is compiled with all required modules.

Configuration #

The following configuration can be used as a reference point. It is designed to be easily adaptable and to mimic the Universal Forwarders behaviour as closely as possible.

You have to adjust the following options:

• L5: Event Channel to monitor
• L9: Target index
• L18: Location for bookmarks
• L43-47: Memory limiter parameters
• L53: HEC token
• L54: HEC endpoint

The COMPUTERNAME environment variable is used as hostname, to match possible transformation rules on the indexers that were setup for the Event Forwarding hosts. The standard behaviour is to extract the event sources hostname.

# OpenTelemetry WEF Collector Configuration

receivers:
  windowseventlog/forwarded_events:
    channel: ForwardedEvents
    raw: true
    attributes:
    - host.name: ${env:COMPUTERNAME}
    - com.splunk.index: target_index_name
    - com.splunk.sourcetype: XmlWinEventLog
    storage: file_storage/bookmark_sysmon
    retry_on_failure:
      enabled: true

extensions:
  file_storage/bookmark_sysmon:
    # Adjust the path according to your environment
    directory: "C:/Program Files/otel/storage/bookmark_sysmon"
    timeout: 1s
  health_check:
    endpoint: "0.0.0.0:8090"
    path: /health/status
    response_body:
        healthy: Healthy
        unhealthy: Sick
    check_collector_pipeline:
        enabled: true
        interval: 1m
        exporter_failure_threshold: 2

processors:
  batch:
    send_batch_size: 10000
    timeout: 10s
  transform:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - set(cache, ExtractPatterns(body, ".*?\\<System\\>.*?\\<Channel\\>(?P<channel>.*?)\\<\\/Channel\\>.*?\\<\\/System\\>"))
      - set(attributes["com.splunk.source"], Concat(["XmlWinEventLog", cache["channel"]], ":"))
      - keep_keys(attributes, ["com.splunk.source", "com.splunk.index", "com.splunk.sourcetype", "host.name"])
  memory_limiter:
    # Adjust these values according to your environment, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md
    limit_mib: 1512
    spike_limit_mib: 320
    check_interval: 1s

exporters:
  splunk_hec/output:
    # The token needs write access to splunks '_internal' index if you enable the heart beat verification
    # See: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7d62ad92dfff8ad4d17a5fc82b6eebc1cc48c2a9/exporter/splunkhecexporter/heartbeat.go#L129
    token: 'CHANGEME'
    endpoint: http://127.0.0.1:8088/services/collector
    sourcetype: XmlWinEventLog
    source: wec
    export_raw: false
    health_check_enabled: true
    index: default
    heartbeat:
      startup: true
    telemetry:
      enabled: true
  debug:
    verbosity: detailed

service:
  telemetry:
    logs:
      level: info
  pipelines:
    logs:
      receivers:
      - windowseventlog/forwarded_events
      processors:
      - memory_limiter
      - batch
      - transform
      exporters:
      - splunk_hec/output
      # - debug
  extensions:
    - health_check
    - file_storage/bookmark_sysmon

Caveats #

Using the OpenTelemetry Collector to ingest logs into Splunk will result into two additional indexed fields being saved, otel.log.severity.number and otel.log.severity.text.

Earlier Workaround #

Before migrating to the OpenTelemetry Collector I was using NXLog Community Edition to monitor the Event Channels, write the event into a plain text file on disk and read said file using Splunks monitor. This also worked reasonably well, but caused data duplication on my Forwarders.