Event Normalisation¶

Abstract

All the punch parsers conform to the normalization presented below. If you create a new cybersecurity parser on your own, make sure to follow the principles explained here.

Rationale¶

The aim of normalization in the PunchPlatform is to get common fields, notwithstanding the log format of each equipment or software vendor. It helps creating queries and dashboards for complete log categories, encompassing the many and various constructors and models.

In security log management, we conform to an open and standard taxonomy, XDAS ( for eXchange Distributed Audit Service, see the OpenXDAS project ). This taxonomy defines five main components that are key to understand:

Where:

The initiator (field [init]) is the component (host or user) which was the source of the event (e.g. for a firewall, the source host);
The target (field [target]) is the component (host or user) which was the destination of the event (e.g. for a firewall, the destination host);
The observer (field [obs]) is the host which handled the security function (e.g. for a firewall, the appliance itself);
The reporter (field [rep]) is the host(s) handling the event generated (e.g. for a firewall, a network orchestrator);
The collector (field [col]) is the entry point of the Log Management system. In PunchPlatform\'s LMC scheme, it is the LTR itself.

This taxonomy is crucial. If the observer field is mandatory (the event comes always from somewhere), so as the collector (where it went), the other fields are optional. For instance, * there is no initiator or target after an antivirus scan report viruses: we don\'t know who put this in. But if the trigger comes from an HIDS, this information is available; * there is no reporter is your device is directly sending the syslog to your collector, such as Juniper switches.

Mandatory fields¶

Most of the mandatory fields are already provided the standard/common/input.punch punchlet at the beginning of each channel processing workflow.

Tuple Field	Value Type	Description
`[channel]`	String	Channel is a text string that identifies the channel where the log will be processed.
`[message]`	String	The raw log message collected.
`[size]`	Integer	Size of the raw log message.
`[tenant]`	String	Tenant is a text string that identifies the customer in the PunchPlatform.
`[kv]`	Tuple	Result of processing error. (by convention)
`[vendor]`	String	Vendor is a text string that identifies the vendor
`[type]`	String	Type of log produced by the device. For example, fw, web, mx, sys, ids, etc.. By default this value is set to unknown, but must be overridden.

But some of them need to be define at the Punchlet level itself. These are the following:

Tuple Field	Value Type	Description
`[parser][name]`	String	Name of the dedicated processing Punchlet. Must match the parser directory name in lower case (i.e. `apache_httpd`).
`[parser][version]`	String	Version of this Punchlet. It must follow the semver pattern "`2.0.1`".

Metadata fields¶

Metadata fields are set by the punchlets processing the log to identify which server processed the log and when.

Tuple Field	Value Type	Description
`[lmc][enrichment][global][host]`	Tuple	The IP address or the host name of the server running the global enrichment punchlet.
`[lmc][enrichment][global][ts]`	String	The date and timestamp of the server receiving the log to run the global enrichment punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
`[lmc][enrichment][tenant][host]`	Tuple	The IP address or the host name of the server running the customer enrichment punchlet.
`[lmc][enrichment][tenant][ts]`	String	The date and timestamp of the server receiving the log to run the tenant enrichment punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
`[lmc][enrichment][channel][host]`	Tuple	The IP address or the host name of the server running the channel enrichment punchlet.
`[lmc][enrichment][channel][ts]`	String	The date and timestamp of the server receiving the log to run the channel enrichment punchlet. This field should conform to the yyyy-mm-dd hh:mm:ss format.
`[lmc][error]`	String	Texts string that identify errors encountered during punchlet execution.
`[lmc][input][host]`	Tuple	The IP address or the host name of the server running the input punchlet.
`[lmc][input][ts]`	String	The date and timestamp of the server receiving the log to run the input punchlet. This field should conform to the 'yyyy-mm-dd hh:mm:ss' format.
`[lmc][output][host]`	Tuple	The IP address or the host name of the server running the output punchlet.
`[lmc][output][ts]`	String	The date and timestamp of the server receiving the log to run the output punchlet. This field should conform to the `yyyy-mm-dd hh:mm:ss` format.
`[lmc][parse][host]`	Tuple	The IP address or the host name of the server running the parser punchlet.
`[lmc][parse][ts]`	String	The date and timestamp of the server receiving the log to run the parser punchlet. This field should conform to the `yyyy-mm-dd hh:mm:ss` format.

Normalized common fields¶

General event normalization¶

Tuple Field	Value Type	Description
`[action]`	String	Action of the event.
`[alarm][name]`	String	Event alarm defined by the vendor.
`[alarm][description]`	String	Event alarm additional elements provided by the vendor.
`[alarm][facility]`	String	Event facility defined by the vendor
`[alarm][id]`	String	Event severity ID defined by the vendor.
`[alarm][impact]`	String	Event impact defined by the vendor.
`[alarm][sev]`	String	Event severity defined by the vendor.
`[app][method]`	String	Application method.
`[app][name]`	String	Application name.
`[app][proto][name]`	String	Application protocol name.
`[app][proto][num]`	Integer	Application protocol number.
`[app][return][code]`	String	Application return or exit code.
`[app][return][description]`	String	Application return or exit code description.
`[app][version]`	String	Version number of the application.
`[col][host]`	Tuple	The IP address or the host name of the collector is the host receiving events from the reporter.
`[col][ts]`	String	The date and timestamp of the collector when the event is received.
`[detection]`	Tuple	Result of the detection processing.
`[init][group]`	Tuple	Group initiating the event.
`[init][host]`	Tuple	Host initiating the event.
`[init][process]`	Tuple	Process initiating the event.
`[init][uri]`	Tuple	Source uri of the event.
`[init][usr]`	Tuple	User initiating the event.
`[obs][group]`	String	The group of the equipment observing the event
`[obs][host]`	Tuple	The IP address or the host name of the observer is the device producing events.
`[obs][process]`	Tuple	Process observing the event.
`[obs][ts]`	String	The date and timestamp of the event observed by the device.
`[obs][usr]`	Tuple	User observing the event.
`[rep][group]`	Tuple	Group of reporter.
`[rep][host]`	Tuple	The IP address or the host name of the reporter is the host receiving events from the observer.
`[rep][ts]`	String	The date and timestamp of the reporter when the event is received.
`[rule][id]`	String	Rule id associated with the event.
`[rule][name]`	String	Rule name associated with the event.
`[rule][uid]`	String	Rule uid associated with the event.
`[session][cipher]`	String	The name of the cipher used for the session.
`[session][cookie][client]`	String	The client session cookie of the event.
`[session][cookie][server]`	String	The server session cookie of the event.
`[session][count]`	Integer	Number of occurrences aggregated in the event.
`[session][duration]`	Integer	The session duration of the event.
`[session][file][hash]`	String	The session file hash of the event.
`[session][file][name]`	String	The session file name of the event.
`[session][file][path]`	String	The session file path of the event.
`[session][file][type]`	String	The session file type of the event.
`[session][id]`	String	The session id of the event.
`[session][in][byte]`	Double	The number indicating the byte count to the event source.
`[session][in][packet]`	Integer	The number indicating the packet count to the event source.
`[session][out][byte]`	Double	The number indicating the byte count to the event destination.
`[session][out][packet]`	Integer	The number indicating the packet count to the event destination.
`[target][group]`	Tuple	Group targeted in the event.
`[target][host]`	Tuple	Host targeted in the event.
`[target][process]`	Tuple	Target process of the event.
`[target][uri]`	Tuple	Target uri of the event.
`[target][usr]`	Tuple	User targeted in the event.

[Type] Tuple normalisation¶

XDAS allows plenty of technology types to be taken into account, however, we cselected only the ones below :

FW - Firewall
AV - Antivirus, HIDPS (Host-based Intrusion Detection & Prevention Systems)
IDS - NIDPS (Network-based Intrusion Detection & Prevention Systems)
LB - Load Balancers
SYS - Operating Systems
WEB - Web/FTP servers
VPN - VPN Appliances
AUTH - AAA Control Systems (Authentication, Authorization, Access)
WAF - Web Application Firewalls
MX - Mail Transfer Agents
ORCH - Appliance Orchestrator
DNS - Name Servers
BAS - Bastions
BDD - Databases
PKI - Certificate managers
NAUTH - DHCP/RADIUS
RTSW - Other network equipments (routers, switches)

Location Tuple normalization¶

Tuple Field	Value Type	Description
`[loc][city]`	String	Indicates the city of a geolocation.
`[loc][country]`	String	Indicates the country of a geolocation.
`[loc][country_short]`	String	Indicates the 2-letter country code of a geolocation.
`[loc][geo_point]`	GeoPoint	Indicates the latitude and longitude of a geolocation. The format should be compliant to Elastic GeoPoint Reference

User Tuple normalization¶

Tuple Field	Value Type	Description
`[usr][domain]`	String	User domain name.
`[usr][fullname]`	String	User full name.
`[usr][id]`	String	User ID.
`[usr][loc]`	Tuple	User geolocation.
`[usr][mail]`	String	User mail.
`[usr][name]`	String	User name.
`[usr][sid]`	String	User SID.

Host Tuple normalization¶

Tuple Field	Value Type	Description
`[host][if]`	String	Host interface.
`[host][ip]`	String	Host IPv4 address.
`[host][ipv6]`	String	Host IPv6 address.
`[host][loc]`	Tuple	Host geolocation.
`[host][mac]`	String	Host MAC address.
`[host][name]`	String	Host name.
`[host][nat][ip]`	String	Host IPv4 address (NAT).
`[host][nat][port]`	Integer	Host port (NAT).
`[host][port]`	Integer	Host port.
`[host][vlan]`	String	Host VLAN.

Process Tuple normalization¶

Tuple Field	Value Type	Description
`[process][exit]`	String	Process exit code.
`[process][id]`	String	Process ID.
`[process][name]`	String	Process name.
`[process][path]`	String	Process path.
`[process][ppid]`	String	Parent process ID.
`[process][status]`	String	Process status.

URI Tuple normalization¶

Tuple Field	Value Type	Description
`[uri][category]`	String	URI category.
`[uri][full]`	String	Full Uniform Resource Identifier.
`[uri][url]`	String	Uniform Resource Locator.
`[uri][urn]`	String	Uniform Resource Name.

Web specific Tuple normalization¶

Tuple Field	Value Type	Description
`[web][request][method]`	String	The forwarding method
`[web][request][icap_status]`	String	The ICAP information forwarded by a web proxy server
`[web][request][rc]`	Int	The return code provided by the next relay. It can be protocolar (HTTP/403) or applicative (exit 2) and can be different from the real response from the web server.
`[web][header][referer]`	String	The referer of the request
`[web][header][version]`	String	Any version used in the header (i.e. HTTP version)
`[web][header][args]`	Array	Arguments added into the header, KV-style format, followinf RFC 7560
`[web][header][content_type]`	String	The content type of the response
`[web][header][user_agent]`	String	The user agent of the request

IDS specific normalization¶

Tuple Field	Value Type	Description
`[ids][cnc]`	Tuple	Like , represents a (Host) Command-n-Control

Mailer specific normalization¶

The fields [mx][from], [mx][to] and [mx][attachments] are now deprecated for resp. [init][usr][mail], [target][usr][mail] and [session][file][name] .

Tuple Field	Value Type	Description
`[mx][subject]`	String	Subject of the email

Antivirus specific Tuple normalization¶

Tuple Field	Value Type	Description
`[av][threats]`	Int	Number of threats detected
`[av][infected]`	Int	Number of actual viruses detected
`[av][event_source]`	String	Source of the threat detection
`[av][infection_category]`	String	Category of the infection
`[av][signature]`	String	Typology of the infection (ex. Virus,Trojan)
`[av][virus_name]`	String	Name of the infection
`[av][infection_type]`	String	Typology of the infection