You may want to filter your logs collected, or even log content, before the data is ingested into Microsoft Sentinel. For example, you may want to filter out logs that are irrelevant or unimportant to security operations, or you may want to remove unwanted details from log messages. Filtering message content may also be helpful when trying to drive down costs when working with Syslog, CEF, or Windows-based logs that have many irrelevant details.
Filter your logs using one of the following methods:
-
The Azure Monitor Agent. Supported on both Windows and Linux to ingest Windows security events. Filter the logs collected by configuring the agent to collect only specified events.
-
Logstash. Supports filtering message content, including making changes to the log messages. For more information, see Connect with Logstash.
More on this page: https://learn.microsoft.com/en-us/azure/sentinel/best-practices-data
- On-premises Windows log collection
- On-premises Linux log collection
- Endpoint solutions
- Office data
- Cloud platform data
Check Your Sentinel Log Ingestion and Cost
Log Type Ingestion Details on Each Table
Syslog Forwarder (AMA Installation)
Basically, you deploy your Linux machine (I used Ubuntu Azure VM). In many cases you’re machine is on-premises, you need to onboard it using Azure Arc. Mind the pre-requisites (especially network connectivity).
Since you’ve Arc’d your on-premises collector machine, you can just enable syslog/CEF collection from Sentinel: Stream CEF logs to Microsoft Sentinel with the AMA connector
Sentinel’s “Common Event Format (CEF) via AMA” -connector page explains pretty much all you need.
Overview how the data collection works:
Ultimately, all the logs at this point go into CommonSecurityLog table – which is Analytics tier table, there’s nothing you can do. Here is an example for enabled local4 facility where my syslog/CEF will flow:
Obviously you need to enable syslog/CEF forwarding in your firewall(s) and make sure it’s a) in correct format and b) communication works.
1 rsyslog configuration:
root@NETSEC-syslog:~# cd /etc/rsyslog.d/
root@NETSEC-syslog:/etc/rsyslog.d# ls
05-filterlogs.conf 10-azuremonitoragent-omfwd.conf 10-vcenter.conf.old 20-ufw.conf 50-default.conf
05-filterlogs.conf.20240628.bk 10-azuremonitoragent-omfwd.conf.20240628.bk 11-meraki.conf.old 21-cloudinit.conf 95-omsagent.conf
root@NETSEC-syslog:/etc/rsyslog.d# cat 05-filterlogs.conf
# Forward vCenter Logs to OMS
if ($inputname == ‘udp_vcenter’ or $inputname == ‘tcp_vcenter’) then @@127.0.0.1:23033;vcenter
& stop
# Forward Meraki Logs to OMS
if ($inputname == ‘udp_meraki’ or $inputname == ‘tcp_meraki’) then @@127.0.0.1:22033;meraki
& stop
# FIlter Fortinet logs
if ($rawmsg contains “traffic:forward accept”) then stop
if ($rawmsg contains “traffic:local accept”) then stop
if ($rawmsg contains “traffic:forward start”) then stop
if ($rawmsg contains “traffic:forward close”) then stop
if ($rawmsg contains “traffic:forward server-rst”) then stop
if ($rawmsg contains “traffic:forward timeout”) then stop
if ($rawmsg contains “traffic:forward client-rst”) then stop
if ($rawmsg contains “traffic:local close”) then stop
if ($rawmsg contains “traffic:forward ip-conn”) then stop
if ($rawmsg contains “traffic:local server-rst”) then stop
if ($rawmsg contains “traffic:local client-rst”) then stop
if ($rawmsg contains “utm:ssl ssl-anomaly”) then stop
if ($rawmsg contains “traffic:local timeout”) then stop
if ($rawmsg contains “event:switch-controller”) then stop
if ($rawmsg contains “event:connector”) then stop
if ($rawmsg contains “event:ha”) then stop
if ($rawmsg contains “event:security-rating”) then stop
if ($rawmsg contains “event:vpn failure”) then stop
if ($rawmsg contains “event:vpn success”) then stop
if ($rawmsg contains “event:system clash”) then stop
if ($rawmsg contains “event:vpn negotiate_error”) then stop
if ($rawmsg contains “event:vpn esp_error”) then stop
if ($rawmsg contains “event:system update”) then stop
if ($rawmsg contains “event:switch-controller none”) then stop
if ($rawmsg contains “event:user logout”) then stop
if ($rawmsg contains “utm:anomaly anomaly”) then stop
if ($rawmsg contains “event:switch-controller None”) then stop
if ($rawmsg contains “event:switch-controller down”) then stop
if ($rawmsg contains “event:switch-controller up”) then stop
if ($rawmsg contains “event:system success”) then stop
if ($rawmsg contains “event:system failed”) then stop
if ($rawmsg contains “event:vpn dpd_failure”) then stop
root@NETSEC-syslog:/etc/rsyslog.d#
2 Restart rsyslog service and check status
root@NETSEC-syslog:/etc/rsyslog.d# systemctl restart rsyslog
root@NETSEC-syslog:/etc/rsyslog.d# systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2024-07-06 14:45:15 UTC; 12s ago
TriggeredBy: ● syslog.socket
Docs: man:rsyslogd(8)
man:rsyslog.conf(5)
https://www.rsyslog.com/doc/
Main PID: 3418874 (rsyslogd)
Tasks: 13 (limit: 9387)
Memory: 7.0M
CPU: 122ms
CGroup: /system.slice/rsyslog.service
└─3418874 /usr/sbin/rsyslogd -n -iNONE Jul 06 14:45:15 NETSEC-syslog systemd[1]: Starting System Logging Service...
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0]
Jul 06 14:45:15 NETSEC-syslog systemd[1]: Started System Logging Service.
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: rsyslogd's groupid changed to 113
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: rsyslogd's userid changed to 107
Jul 06 14:45:15 NETSEC-syslog rsyslogd[3418874]: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="3418874" x-info="https://www.rsyslog.com"] start
root@NETSEC-syslog:/etc/rsyslog.d#
3 Azuremonitoragent-omfwd.conf
root@NETSEC-syslog:/etc/rsyslog.d# cat 10-azuremonitoragent-omfwd.conf # Azure Monitor Agent configuration: forward logs to azuremonitoragent template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
# queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity
# Forwarding all events through TCP port
*.* action(type="omfwd"
template="AMA_RSYSLOG_TraditionalForwardFormat"
queue.type="LinkedList"
queue.filename="omfwd-azuremonitoragent"
queue.maxFileSize="32m"
action.resumeRetryCount="-1"
action.resumeInterval="5"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
queue.size="25000"
queue.workerThreads="100"
queue.dequeueBatchSize="2048"
queue.saveonshutdown="on"
target="127.0.0.1" Port="28330" Protocol="tcp")
root@NETSEC-syslog:/etc/rsyslog.d#
4 Default rules for rsyslog
root@NETSEC-syslog:/etc/rsyslog.d# cat 50-default.conf
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Some “catch-all” log files.
#
#*.=debug;\
# auth,authpriv.none;\
# news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
root@NETSEC-syslog:/etc/rsyslog.d#
Rsyslog configuraiton
Commands:
- rsyslogd -v
- systemctl status rsyslog
- systemctl start rsyslog
- vi /etc/rsyslog.conf
root@NETSEC-syslog:~# cat /etc/rsyslog.conf
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load=”imuxsock”) # provides support for local system logging
#module(load=”immark”) # provides –MARK– message capability
# provides UDP syslog reception
module(load=”imudp”)
input(type=”imudp” port=”514″)
# provides TCP syslog reception
module(load=”imtcp”)
input(type=”imtcp” port=”514″)
input(type=”imudp” port=”1514″ name=”udp_meraki”)
input(type=”imudp” port=”2514″ name=”udp_vcenter”)
input(type=”imtcp” port=”1514″ name=”tcp_meraki”)
input(type=”imtcp” port=”2514″ name=”tcp_vcenter”)
$template meraki,”%timestamp% %hostname% %msg%”
$template vcenter,”%timestamp% %hostname% %msg%”
# provides kernel logging support and enable non-kernel klog messages
module(load=”imklog” permitnonkernelfacility=”on”)
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
root@NETSEC-syslog:~#
Use Data Collection Rules to Filter Logs
Azure Portal – Monitor – Data Collection Rules
Updating Data Collection Rule using Powershell
$ResourceId = “/subscriptions/d95-245-48c-a50-b1a7b9/resourceGroups/SEN-RG/providers/Microsoft.Insights/dataCollectionRules/dcr-dcwindowssecurityeventcommon” # Resource ID of the DCR to edit
$FilePath = “tmp1.dcr” # Store DCR content in this file
$DCR = Invoke-AzRestMethod -Path (“$ResourceId”+”?api-version=2021-09-01-preview”) -Method GET
$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath
$ResourceId = “/subscriptions//d95-245-48c-a50-b1a7b9/resourceGroups/SEN-RG/providers/Microsoft.Insights/dataCollectionRules/dcr-dcwindowssecurityeventcommon” # Resource ID of the DCR to edit
$FilePath = “tmp1.dcr” # Store DCR content in this file
$DCRContent = Get-Content $FilePath -Raw
Invoke-AzRestMethod -Path (“$ResourceId”+”?api-version=2021-09-01-preview”) -Method PUT -Payload $DCRContent
ResourceID
DCR Details in Export Template
Run in Cloud Shell
Fortinet Devices (FortiGate)
Local Facility: Changed to Local 7
Log Filtering on Syslog Forwarder Server:
# Azure Monitor Agent configuration: forward logs to azuremonitoragent
# Filter Fortinet logs
:msg, contains, “traffic:forward accept” ~
:msg, contains, “traffic:local accept” ~
:msg, contains, “traffic:forward start” ~
:msg, contains, “traffic:forward close” ~
:msg, contains, “traffic:forward server-rst” ~
:msg, contains, “traffic:forward timeout” ~
:msg, contains, “traffic:forward client-rst” ~
:msg, contains, “traffic:local close” stop
:msg, contains, “traffic:forward ip-conn” stop
:msg, contains, “traffic:local server-rst” stop
:msg, contains, “traffic:local client-rst” stop
:msg, contains, “utm:ssl ssl-anomaly” stop
:msg, contains, “traffic:local timeout” stop
:msg, contains, “event:switch-controller” stop
:msg, contains, “event:connector” stop
:msg, contains, “event:ha” stop
:msg, contains, “event:security-rating” stop
:msg, contains, “event:vpn failure” stop
:msg, contains, “event:vpn success” stop
:msg, contains, “event:system clash” stop
:msg, contains, “event:vpn negotiate_error” stop
:msg, contains, “event:vpn esp_error” stop
:msg, contains, “event:system update” stop
:msg, contains, “event:switch-controller none” stop
:msg, contains, “event:user logout” stop
:msg, contains, “utm:anomaly anomaly” stop
:msg, contains, “event:switch-controller None” stop
:msg, contains, “event:switch-controller down” stop
:msg, contains, “event:switch-controller up” stop
:msg, contains, “event:system success” stop
:msg, contains, “event:system failed” stop
:msg, contains, “event:vpn dpd_failure” stop
template(name=”AMA_RSYSLOG_TraditionalForwardFormat” type=”string” string=”<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%”)
# queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity
# Forwarding all events through TCP port
*.* action(type=”omfwd”
template=”AMA_RSYSLOG_TraditionalForwardFormat”
queue.type=”LinkedList”
queue.filename=”omfwd-azuremonitoragent”
queue.maxFileSize=”32m”
action.resumeRetryCount=”-1″
action.resumeInterval=”5″
action.reportSuspension=”on”
action.reportSuspensionContinuation=”on”
queue.size=”25000″
queue.workerThreads=”100″
queue.dequeueBatchSize=”2048″
queue.saveonshutdown=”on”
target=”127.0.0.1″ Port=”28330″ Protocol=”tcp”)
Replace /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf file with above content. Do not forgot to restart rsyslog service:
- systemctl start rsyslog
- systemctl status rsyslog
let Watchlist = datatable(Priority:string, Activity:string) [
‘1’,’event:system’,
‘1’,’event:user’,
‘1’,’event:user logon’,
‘1’,’event:vpn’,
‘1’,’utm:anomaly’,
‘1’,’utm:dlp’,
‘1’,’utm:dlp dlp-docsource’,
‘1’,’utm:dns’,
‘1’,’utm:dns dns-query’,
‘1’,’utm:dns dns-response’,
‘1’,’utm:emailfilter’,
‘1’,’utm:emailfilter bannedword’,
‘1’,’utm:emailfilter spam’,
‘1’,’utm:emailfilter webmail’,
‘1’,’utm:ips’,
‘1’,’utm:ips botnet’,
‘1’,’utm:ips malicious-url’,
‘1’,’utm:ips signature’,
‘1’,’utm:ssh ssh-channel’,
‘1’,’utm:ssh ssh-command’,
‘1’,’utm:ssh ssh-hostkey’,
‘1’,’utm:waf’,
‘1’,’utm:waf waf-address-list’,
‘1’,’utm:waf waf-custom-signature’,
‘1’,’utm:waf waf-http-constraint’,
‘1’,’utm:waf waf-http-method’,
‘1’,’utm:waf waf-signature’,
‘1’,’utm:waf waf-url-access’,
‘2’,’event:cifs-auth-fail’,
‘2’,’event:endpoint’,
‘2’,’event:rest-api’,
‘2’,’event:router’,
‘2’,’event:sdwan’,
‘2’,’event:sdwan down’,
‘2’,’event:sdwan up’,
‘2’,’event:webproxy’,
‘2’,’event:wireless’,
‘2’,’traffic:forward deny’,
‘2’,’traffic:ztna’,
‘2’,’utm:app-ctrl’ ,
‘2’,’utm:app-ctrl port-violation’,
‘2’,’utm:app-ctrl protocol-violation’,
‘2’,’utm:app-ctrl signature’,
‘2’,’utm:file-filter’,
‘2’,’utm:virus’,
‘2’,’utm:virus analytics’,
‘2’,’utm:virus command-blocked’,
‘2’,’utm:virus content-disarm’,
‘2’,’utm:virus ems-threat-feed’,
‘2’,’utm:virus exempt-hash’,
‘2’,’utm:virus infected’,
‘2’,’utm:virus inline-block’,
‘2’,’utm:virus malware-list’,
‘2’,’utm:virus outbreak-prevention’,
‘2’,’utm:virus oversize’,
‘2’,’utm:voip’,
‘2’,’utm:webfilter’,
‘2’,’utm:webfilter ftgd_blk’,
‘2’,’utm:webfilter ftgd_err’,
‘2’,’utm:webfilter urlfilter’,
‘2’,’utm:webfilter webfilter_command_block’,
‘3’,’event:connector’,
‘3’,’event:fortiextender’,
‘3’,’event:ha’,
‘3’,’event:switch-controller’,
‘3’,’event:wanopt’,
‘3’,’traffic:forward’,
‘3’,’traffic:forward accept’,
‘3’,’traffic:forward client-rst’,
‘3’,’traffic:forward close’,
‘3’,’traffic:forward dns’,
‘3’,’traffic:forward ip-conn’,
‘3’,’traffic:forward server-rst’,
‘3’,’traffic:forward timeout’,
‘3’,’traffic:local’,
‘3’,’traffic:local accept’,
‘3’,’traffic:local client-rst’,
‘3’,’traffic:local close’,
‘3’,’traffic:local deny’,
‘3’,’traffic:local server-rst’,
‘3’,’traffic:local timeout’,
‘3’,’traffic:multicast’,
‘3’,’traffic:sniffer’,
‘3’,’utm:casb’,
‘3’,’utm:emailfilter email’,
‘3’,’utm:emailfilter ftgd_err’,
‘3’,’utm:forti-switch’,
‘3’,’utm:forti-switch fsw-flow’,
‘3’,’utm:gtp’,
‘3’,’utm:gtp gtp-all’,
‘3’,’utm:gtp pfcp-all’,
‘3’,’utm:icap’,
‘3’,’utm:ssl ssl-anomaly’,
‘3’,’utm:ssl ssl-exempt’,
‘3’,’utm:ssl ssl-handshake’,
‘3’,’utm:ssl ssl-negotiation’,
‘3’,’utm:ssl ssl-server-cert-info’,
‘3’,’utm:virtual-patch’,
‘3’,’utm:virus filename’,
‘3’,’utm:virus filetype-executable’,
‘3’,’utm:virus mimefragmented’,
‘3’,’utm:virus scanerror’,
‘3’,’utm:virus switchproto’,
‘3’,’utm:webfilter activexfilter’,
‘3’,’utm:webfilter antiphishing’,
‘3’,’utm:webfilter appletfilter’,
‘3’,’utm:webfilter content’,
‘3’,’utm:webfilter cookiefilter’,
‘3’,’utm:webfilter ftgd_allow’,
‘3’,’utm:webfilter ftgd_quota’,
‘3’,’utm:webfilter ftgd_quota_counting’,
‘3’,’utm:webfilter ftgd_quota_expired’,
‘3’,’utm:webfilter http_header_change’,
‘3’,’utm:webfilter scriptfilter’,
‘3’,’utm:webfilter ssl-exempt’,
‘3’,’utm:webfilter unknown-ce’,
‘3’,’utm:webfilter urlmonitor’,
‘3’,’utm:webfilter videofilter-category’,
‘3’,’utm:webfilter videofilter-channel’,
‘3’,’event:security-rating’
];
CommonSecurityLog
| where DeviceVendor contains “Forti”
| summarize TotalActivity = count() by Activity
| lookup Watchlist on Activity
- sudo tcpdump -i any dst port 28330 -Ann | grep “traffic:forward accept”
SonicWall
Local Facility: Changed to Local 7
Log Filtering on Syslog Forwarder Server: