Skip to content

Event Normalization

Important

All the PunchPlatform Standard log parsers conform to the normalization presented here. If you create new parser on your own, make sure follow this rules. This is the best way to avoid data indexing issues to Elasticsearch.

Rationale

The aim of normalization in the PunchPlatform is to get common fields, nonwithstanding the log format of each equipment or software vendor. It helps creating queries and dashboards for complete log categories, encompassing the many and various constructors and models.

In security log management, we conform to an open and standard taxonomy, XDAS ( for eXchange Distributed Audit Service, see the OpenXDAS project ). This taxonomy defines five main components that are key to understand:

image

Where:

  • The initiator (field [init]) is the component (host or user) which was the source of the event (e.g. for a firewall, the source host);
  • The target (field [target]) is the component (host or user) which was the destination of the event (e.g. for a firewall, the destination host);
  • The observer (field [obs]) is the host which handled the security function (e.g. for a firewall, the appliance itself);
  • The reporter (field [rep]) is the host(s) handling the event generated (e.g. for a firewall, a network orchestrator);
  • The collector (field [col]) is the entry point of the Log Management system. In PunchPlatform\'s LMC scheme, it is the LTR itself.

This taxonomy is crucial. If the observer field is mandatory (the event comes always from somewhere), so as the collector (where it went), the other fields are optional. For instance, * there is no initiator or target after an antivirus scan report viruses: we don\'t know who put this in. But if the trigger comes from an HIDS, this information is available; * there is no reporter is your device is directly sending the syslog to your collector, such as Juniper switches.

Mandatory fields

Most of the mandatory fields are already provided the standard/common/input.punch punchlet at the beggining of each channel processing workflow.

Tuple Field Value Type Description
[channel] String Channel is a text string that identifies the channel where the log will be processed.
[message] String The raw log message collected.
[size] Integer Size of the raw log message.
[tenant] String Tenant is a text string that identifies the customer in the PunchPlatform.
[kv] Tuple Result of processing error. (by convention)
[vendor] String Vendor is a text string that identifies the vendor
[type] String Type of log produced by the device. For example, fw, web, mx, sys, ids, etc.. By default this value is set to unknown, but must be overriden.

But some of them need to be define at the Punchlet level itself. These are the following:

Tuple Field Value Type Description
[parser][name] String Name of the dedicated processing Punchlet. Must match the parser directory name in lower case (i.e. apache_httpd).
[parser][version] String Version of this Punchlet. It must follow the semver pattern "2.0.1".

Metadata fields

Metadata fields are set by the punchlets processing the log to identify which server processed the log and when.

Tuple Field Value Type Description
[lmc][enrichment][global][host] Tuple The IP address or the host name of the server running the global enrichment punchlet.
[lmc][enrichment][global][ts] String The date and timestamp of the server receiving the log to run the global enrichment punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
[lmc][enrichment][tenant][host] Tuple The IP address or the host name of the server running the customer enrichment punchlet.
[lmc][enrichment][tenant][ts] String The date and timestamp of the server receiving the log to run the tenant enrichment punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
[lmc][enrichment][channel][host] Tuple The IP address or the host name of the server running the channel enrichment punchlet.
[lmc][enrichment][channel][ts] String The date and timestamp of the server receiving the log to run the channel enrichment punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
[lmc][error] String Texts string that identify errors encountered during punchlet execution.
[lmc][input][host] Tuple The IP address or the host name of the server running the input punchlet.
[lmc][input][ts] String The date and timestamp of the server receiving the log to run the input punchlet. This field should conform to the 'yyyy-mm-dd hh:mm:ss' format.
[lmc][output][host] Tuple The IP address or the host name of the server running the output punchlet.
[lmc][output][ts] String The date and timestamp of the server receiving the log to run the output punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
[lmc][parse][host] Tuple The IP address or the host name of the server running the parser punchlet.
[lmc][parse][ts] String The date and timestamp of the server receiving the log to run the parser punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.

Normalized common fields

General event normalization

Tuple Field Value Type Description
[action] String Action of the event.
[alarm][name] String Event alarm defined by the vendor.
[alarm][description] String Event alarm additional elements provided by the vendor.
[alarm][facility] String Event facility defined by the vendor
[alarm][id] String Event severity ID defined by the vendor.
[alarm][impact] String Event impact defined by the vendor.
[alarm][sev] String Event severity defined by the vendor.
[app][method] String Application method.
[app][name] String Application name.
[app][proto][name] String Application protocol name.
[app][proto][num] Integer Application protocol number.
[app][return][code] String Application return or exit code.
[app][return][description] String Application return or exit code description.
[app][version] String Version number of the application.
[col][host] Tuple The IP address or the host name of the collecter is the host receiving events from the reporter.
[col][ts] String The date and timestamp of the collecter when the event is received.
[detection] Tuple Result of the detection processing.
[init][group] Tuple Group initiating the event.
[init][host] Tuple Host initiating the event.
[init][process] Tuple Process initiating the event.
[init][uri] Tuple Source uri of the event.
[init][usr] Tuple User initiating the event.
[obs][group] String The group of the equipment observing the event
[obs][host] Tuple The IP address or the host name of the observer is the device producing events.
[obs][process] Tuple Process observing the event.
[obs][ts] String The date and timestamp of the event observed by the device.
[obs][usr] Tuple User observing the event.
[rep][group] Tuple Group of reporter.
[rep][host] Tuple The IP address or the host name of the reporter is the host receiving events from the observer.
[rep][ts] String The date and timestamp of the reporter when the event is received.
[rule][id] String Rule id associated with the event.
[rule][name] String Rule name associated with the event.
[rule][uid] String Rule uid associated with the event.
[session][cipher] String The name of the cipher used for the session.
[session][cookie][client] String The client session cookie of the event.
[session][cookie][server] String The server session cookie of the event.
[session][count] Integer Number of occurences aggregated in the event.
[session][duration] Integer The session duration of the event.
[session][file][hash] String The session file hash of the event.
[session][file][name] String The session file name of the event.
[session][file][path] String The session file path of the event.
[session][file][type] String The session file type of the event.
[session][id] String The session id of the event.
[session][in][byte] Double The number indicating the byte count to the event source.
[session][in][packet] Integer The number indicating the packet count to the event source.
[session][out][byte] Double The number indicating the byte count to the event destination.
[session][out][packet] Integer The number indicating the packet count to the event destination.
[target][group] Tuple Group targeted in the event.
[target][host] Tuple Host targeted in the event.
[target][process] Tuple Target process of the event.
[target][uri] Tuple Target uri of the event.
[target][usr] Tuple User targeted in the event.

[Type] Tuple normalisation

XDAS is allowing plenty of technology types to be taken into account, however, we chose to kept some of the main ones below :

  • FW - Firewall
  • AV - Antivirus, HIDPS (Host-based Intrusion Detection & Prevention Systems)
  • IDS - NIDPS (Network-based Intrusion Detection & Prevention Systems)
  • LB - Load Balancers
  • SYS - Operating Systems
  • WEB - Web/FTP servers
  • VPN - VPN Appliances
  • AUTH - AAA Control Systems (Authenticatiion, Authorization, Access)
  • WAF - Web Application Firewalls
  • MX - Mail Transfer Agents
  • ORCH - Appliance Orchestrators
  • DNS - Name Servers
  • BAS - Bastions
  • BDD - Databases
  • PKI - Certificate managers
  • NAUTH - DHCP/RADIUS
  • RTSW - Other network equipements (routers, switches)

Location Tuple normalization

Tuple Field Value Type Description
[loc][city] String Indicates the city of a geolocation.
[loc][country] String Indicates the country of a geolocation.
[loc][country_short] String Indicates the 2-letter country code of a geolocation.
[loc][geo_point] GeoPoint Indicates the latitude and longitude of a geolocation. The format should be compliant to Elastic GeoPoint Reference

User Tuple normalization

Tuple Field Value Type Description
[usr][domain] String User domain name.
[usr][fullname] String User full name.
[usr][id] String User ID.
[usr][loc] Tuple User geolocation.
[usr][mail] String User mail.
[usr][name] String User name.
[usr][sid] String User SID.

Host Tuple normalization

Tuple Field Value Type Description
[host][if] String Host interface.
[host][ip] String Host IPv4 address.
[host][ipv6] String Host IPv6 address.
[host][loc] Tuple Host geolocation.
[host][mac] String Host MAC address.
[host][name] String Host name.
[host][nat][ip] String Host IPv4 address (NAT).
[host][nat][port] Integer Host port (NAT).
[host][port] Integer Host port.
[host][vlan] String Host VLAN.

Process Tuple normalization

Tuple Field Value Type Description
[process][exit] String Process exit code.
[process][id] String Process ID.
[process][name] String Process name.
[process][path] String Process path.
[process][ppid] String Parent process ID.
[process][status] String Process status.

URI Tuple normalization

Tuple Field Value Type Description
[uri][category] String URI category.
[uri][full] String Full Uniform Resource Identifier.
[uri][url] String Uniform Resource Locator.
[uri][urn] String Uniform Resource Name.

Web specific Tuple normalization

Tuple Field Value Type Description
[web][request][method] String The forwarding method
[web][request][icap_status] String The ICAP information forwarded by a web proxy server
[web][request][rc] Int The return code provided by the next relay. It an be protocolar (HTTP/403) or applicative (exit 2) and can be different from the real response from the web server.
[web][header][referer] String The referer of the request
[web][header][version] String Any version used in the header (i.e. HTTP version)
[web][header][args] Array Arguments added into the header, KV-style format, followinf RFC 7560
[web][header][content_type] String The content type of the response
[web][header][user_agent] String The user agent of the request

IDS specific normalization

Tuple Field Value Type Description
[ids][cnc] Tuple Like , represents a (Host) Command-n-Control

Mailer specific normalization

The fields [mx][from], [mx][to] and [mx][attachments] are now deprecated for resp. [init][usr][mail], [target][usr][mail] and [session][file][name] .

Tuple Field Value Type Description
[mx][subject] String Subject of the email

Antivirus specific Tuple normalization

Tuple Field Value Type Description
[av][threats] Int Number of threats detected
[av][infected] Int Number of actual viruses detected
[av][event_source] String Source of the threat detection
[av][infection_category] String Category of the infection
[av][signature] String Typology of the infection (ex. Virus,Trojan)
[av][virus_name] String Name of the infection
[av][infection_type] String Typology of the infection