Settings

The Arkime system utilizes configuration files using INI format by default. With the release of Arkime 5, support was introduced for configuration files in JSON and YAML formats using json or yaml extensions, respectively. When utilizing JSON or YAML, arrays can be specified either natively or using separators like INI.

The configuration file location can be specified using the -c command-line option. Since Arkime 5 the configuration doesn't have to be local, it can also be an url, OpenSearch/Elasticsearch, or Valkey/Redis. Currently capture doesn't support Valkey/Redis. Formats:

• URL - https://[user:password@]example.com/config.ini
• OpenSearch - opensearchs://[user:password@]example.com/INDEX/_doc/DOC
• Elasticsearch - elasticsearchs://[user:password@]example.com:9200/INDEX/_doc/DOC
• Redis - redis://[:password@]host:port/db-number/DOC
• Redis Sentinel - redis-sentinel://[[sentinelPassword]:[password]@]host[:port]/redis-name/db-number/DOC

Overriding Configuration File

The Arkime configuration file can be overriden using two different methods, the command line or environment variables. The command line takes precedence over the environment variables, which in turn takes precedence over the configuration file. Both of these solutions replace the entire value of the variable, so if you want to append to a list you need to specify the entire list.

Command Line

Arkime applications support overriding configuration files via a command line option using either -o variable=value or -o section.variable=value.

Environment Variables

Since Arkime 5.6, environment variables can also be used to override configuration settings. Environment variables are specified in the format ARKIME__variable=value (for default section) or ARKIME_section__variable=value. If the section or variable needs a DOT, DASH, COLON, or SLASH character you can use those keywords since environment variables can't have those characters. It is not possible to use an environment inside the configuration file for partial replacement.

Examples

To replace this entire configuration file

[default]
elasticsearch=https://localhost:9200
[url:foobar]
url=http://localhost:9200
[arkime-clusters]
example=url:http://localhost:8124

-o elasticsearch=https://localhost:9200 -o url:foobar.url=http://localhost:9200 -o arkime-clusters.example=url:http://localhost:8124

ARKIME__elasticsearch=https://localhost:9200 
ARKIME_urlCOLONfoobar__url=http://localhost:9200
ARKIME_arkimeDASHclusters__example=url:http://localhost:8124

Debugging & Logging

There are two methods for enabling debugging to Arkime applications, either the command line --debug option or a setting debug=N in the configuration file. If any command line options are present, then the configuration file is ignored. To increase the debug level on the command line use multiple --debug options, so for example --debug --debug will set the debug level to 2, so will debug=2 in the configuration file.

These settings should live in the [cont3xt], [parliament], [wiseService] or [default] sections depending on the Arkime application.

The capture application has additional settings for logging, see the Capture Debug section.

Web Settings

For Arkime applications that have a web interface these are the common settings for listening for web traffic.

These settings should live in the [cont3xt], [parliament], [wiseService] or [default] sections depending on the Arkime application.

Auth & Security Settings

Arkime applications have common authorization and security settings.

These settings should live in the [cont3xt], [parliament], [wiseService] or [default] sections depending on the Arkime application.

requiredAuthHeader=company-x-role
requiredAuthHeaderVal=arkime,admin

Auth OIDC Settings

(Since 4.2.0) Arkime now support direct OIDC authentication when setting userNameHeader to oidc. Make sure that each user has the "Web Auth Header" checkbox selected.

These settings should live in the [cont3xt], [parliament], [wiseService] or [default] sections depending on the Arkime application.

Example:

authDiscoverURL=[DISCOVER or ISSUER or WELLKNOWN URL]
authClientId=[CLIENTID]
authClientSecret=[CLIENTSECRET]
authUserIdField=preferred_username
authRedirectURIs=http://ARKIMEHOST:PORT/auth/login/callback
# Optional to auto create users, make sure userId/userName variables are right
#userAutoCreateTmpl={"userId": "${this.preferred_username}", "userName": "${this.name}", "enabled": true, "webEnabled": true, "headerAuthEnabled": true, "emailSearch": true, "createEnabled": false, "removeEnabled": false, "packetSearch": true, "roles": ["arkimeUser", "cont3xtUser"] }
# Optionally set a logout URL from provider
#logoutUrl=https://the.oidc.provider.com/logout

Users Database

The Arkime applications store all the users in OpenSearch/Elasticsearch. Since you will have multiple Arkime clusters and applications, Arkime allows you to share the database to use. When these settings aren't use Arkime will fallback to using a non shared Users Database for viewer/capture.

These settings should live in the [cont3xt], [parliament], [wiseService] or [default] sections depending on the Arkime application.

user-role-mappings

Starting with 5.5.0 it is now possible to dynamically update the roles associated with a user when using header or oidc authMode. This is done by adding a [user-role-mappings] section to the configuration file. When using the user-role-mappings section, all roles you are using must be defined, one per line, and any role no longer enabled for the user will be removed. The rules are evaluated on login, and the user's roles are updated accordingly.

To ensure immediate role changes, manually edit the user's roles on the Users tab, as automated updates require a user login.

Each line is of the format rolename=javascript expression. The javascript expression can use two variable, this which allows you to reference any Arkime DB user fields and vals which allows you to access any of the request headers for header auth or the oidc info for oidc auth.

Depending on what IAM system you are using, you may need to enable the sending of userinfo to Arkime.

For user defined roles you must use the format role:a-user-defined-role and have created the role a-user-defined-role in the Users page UI. The leading role: is required to distinguish it from the built-in roles. Using user defined roles is broken before 5.7.0, so please upgrade to take advantage of this feature.

Important:

• Nodejs lowercases all HTTP header names, since HTTP headers are case insensitive. You should use lowercase keys in your javascript expressions. For example vals['x-roles'] NOT vals['X-Roles'].
• Javascript keys can't contain dashes, so you must put them in quotes. For example vals['x-roles'] NOT vals.x-roles.
• A role that isn't defined will be removed from the user.

Example:

[user-role-mappings]
# Everyone is arkimeUser
arkimeUser=true
# usersAdmin have -svc extension in userId
usersAdmin=this.userId.endsWith('-svc')
# fred and wilma are superAdmin
superAdmin=this.userId === 'fred' || this.userId === 'wilma'
# custom role based on header 'roles' containing the string 'special' anywhere
role:a-user-defined-role=vals.roles.includes('special')
# If you need to debug what is in this or vals you can use console.log
# wiseUser=console.log(this, vals),true
# Checks a comma separated x-roles header for the string arkime_wise_admin
wiseAdmin=vals['x-roles'] && vals['x-roles'].split(',').map(s => s.trim()).includes('arkime_wise_admin')

Capture/Viewer

The default confguration file for capture and viewer is /opt/arkime/etc/config.ini. When searching for a setting, capture and viewer employ a tiered system for configuration variables. This tiered system enables a single configuration file to be used across multiple servers. The order of sections within the configuration file is not significant; however, the specific section in which a variable resides can be crucial. A frequent issue encountered is the misplacement of a variable in an incorrect section.

Order of config variables for capture and viewer:

1. [optional] The section titled with the node name is given priority and used first. Arkime will always tag sessions with node: <node name>.
2. [Optional] If a node section possesses a nodeClass variable, the section titled with the nodeClass name is utilized next. Sessions will be tagged with node:<node class name>, which is beneficial for monitoring different network classes.
3. The section titled "default" is considered last.

OpenSearch/Elasticsearch Settings

Arkime leverages OpenSearch/Elasticsearch both as a database and as time-based storage for all saved sessions. The settings pertain to the communication between Arkime and OpenSearch/Elasticsearch. Although the terms "elasticsearch" and "es" appear in the variable names, both OpenSearch and Elasticsearch are compatible with most settings.

Arkime does NOT support having pcapDir and the OpenSearch/Elasticsearch data directory on the same file system. Arkime will NOT work in this configuration. It is strongly recommended running capture and OpenSearch/Elasticsearch on different hosts, if that is not possible, use different disks or file systems.

PCAP Storage Settings

Arkime does NOT support having pcapDir and the OpenSearch/Elasticsearch data directory on the same file system. Arkime will NOT work in this configuration. It is strongly recommended running capture and OpenSearch/Elasticsearch on different hosts, if that is not possible, use different disks or file systems.

Capture Command Line

The capture tool supports several important command line options, please use the --help command line option for the most up-to-date options supported by your version.

/opt/arkime/bin/capture --scheme --command-socket /var/run/arkime.socket --command-wait
nc -U /var/run/arkime/socket
help
exit

--op tags={tag}

Capture Settings

Packet Deduplication Settings

Arkime since version 2.7.1 has supported basic packet deduplication. The deduplication is done early in the pipeline before a packet queue is assigned. Currently only the iphdr + tcphdr or iphdr + udphdr is used to find duplicate packets, ignoring the TTL field. The deduplication will only look back a configured number of seconds.

The packet deduplication is an expensive CPU task, so you should monitor the "Dup Drops/s" column in capture stats to see if worthwhile.

Viewer Settings

Advanced Settings

Capture Debug Settings

These are the capture specific debug settings. They control the logging of various events during the capture process. See the Debug Settings section for general debug settings that apply to all Arkime applications.

Arkime Default User Settings

(Since 5.0.2) Must be in the [user-setting-defaults] section in the viewer config. These options override the default Arkime user settings. They are used when a new user is created.

Readers/Writers

Capture supports several different methods for reading and writing packets during a live capture. These are selected by setting the and config settings.. We suggest using pcapReadMethod=tpacketv3 and the default pcapWriteMethod=simple for best performance.

For offline pcap processing see the command line options of capture and FAQ - How do I import existing PCAPs?.

Reader - AFPacket - Settings

AFPacket v3 also known as Tpacketv3, is the preferred reader for Arkime and can be used on most 3.x or later kernels. Configure capture to use tpacketv3 as the reader method with pcapReadMethod=tpacketv3 in your configuration file.

Example:

[default]
pcapReadMethod=tpacketv3
tpacketv3BlockSize=2097152
interface=eth0
tpacketv3NumThreads=2

Reader - DAQ - Settings

To use daq:

• load the daq plugin by changing configuration file so it has reader-daq.so as a rootPlugins rootPlugins=reader-daq.so
• tell capture to use daq as the reader method with pcapReadMethod=daq in your configuration file
• optionally change interface to any special daq interface value

Reader - PF_RING - Settings

• We strongly suggest you try tpacketv3 (afpacketv3) first if available BEFORE using PF_RING since Arkime supports afpacket better. Note that Arkime only supports vanilla PF_RING and NOT PF_RING DNA nor PF_RING ZC. Since vanilla PF_RING and afpacket are about the same, we recommend you use afpacket. Usually the only time you should use PF_RING is when you have other tools on the same machine also using vanilla PF_RING.
• install the pfring package on all hosts that will run capture from http://packages.ntop.org/
• load the pfring plugin by changing configuration file so it has reader-pfring.so as a rootPlugins rootPlugins=reader-pfring.so
• Configure capture to use pfring as the reader method with pcapReadMethod=pfring in your configuration file
• optionally change interface to any special pfring interface value

Reader - PCAP-Over-Ip - Settings

Since version 2.7.0, Arkime supports processing PCAP-Over-Ip requests. Two modes are support client and server. Client mode will connect to a pcap-over-ip service, server will listen for pcap-over-ip connections. Read more information about PCAP-Over-Ip.

Client mode is set by using pcapReadMethod=pcap-over-ip-client. The interface setting must have a list of hosts and optional ports to connect to.

Server mode is set by using pcapReadMethod=pcap-over-ip-server. The interface setting is ignored and should just be set to dummy.

Reader - SNF - Settings

• install the snf package on all hosts that will run capture
• load the snf plugin by changing configuration file so it has reader-snf.so as a rootPlugins rootPlugins=reader-snf.so
• Configure capture to use snf as the reader method with pcapReadMethod=snf in your configuration file
• optionally change interface to any special snf interface value

Reader - TZSP - Settings

Since version 4.1.0, Arkime supports listening for TZSP forwarded packets.. You can enable TZSP support by using pcapReadMethod=tzsp. The interface setting is ignored and should just be set to dummy.

Reader - Offline Scheme

Arkime 5 introduces the concept of reader schemes to process offline pcap files from multiple local and remote locations. Support for new schemes can be built into capture/viewer or added with a plugin. When a scheme is used to process an offline pcap file, which scheme is used and information about the file is stored in Arkime files index for viewer to use. The schemes run in a dedicated thread, which will make file processing faster then previous versions. Switching to "scheme" mode vs libpcap-file mode is done either by using -r with any scheme OR the --scheme option. It is all or nothing, once in scheme mode, non scheme supported command line options will be ignored with no warning.

file:///fullpath

Process a file on the local disk

http(s)://host(:port)/path

Process a file using http range requests

s3://bucket/key

Process a file that lives in s3, more info

s3http(s)://host(:port)/bucket/key

Process a file using s3 protocol but use a certain host, useful with MinIO/LocalStack for example, more info

sqs://host(:port)/account/queue

Process SQS queue of s3 pcap files more info

sqshttp(s)://host(:port)/account/queue

Process SQS queue of s3 pcap files, useful with LocalStack more info

So for example ./capture -r http://hostname/foo.pcap -r s3://pcapbucket/file.pcap will process 2 files.

Reader - Scheme - S3

Arkime 5.0 added the ability to process S3 files directly without having to first download the files. Arkime 5.3 enhance the capability to process multiple S3 files using a prefix

Two forms of the S3 URL are supported:

• s3://bucket/key
• s3http(s)://host(:port)/bucket/key

Process a single file

capture -r s3://testbucket/filepath/file.pcap

Process files with a matching prefix

capture -R s3https://host:443/testbucket/filepath/abc

Reader - Scheme - SQS

Arkime 5.3 added the ability to process a SQS queue for pcap files added to S3. As files are added to S3, S3 will automatically add a message to the SQS queue. Arkime will monitor the SQS queue and process the S3 files. By default Arkime will exit after all the messages are processed, use the -m option to keep running.

Two forms of the SQS URL are supported:

• sqs://sqs.REGION.amazonaws.com/ACCOUNT/QUEUE
• sqshttp(s)://sqs.REGION.DOMAIN(:PORT)/ACCOUNT/QUEUE

Process an AWS queue and exit

capture -R sqs://sqs.us-east-1.amazonaws.com/00000000/testqueue

Process a LocalStack queue forever

capture -m -R sqshttp://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-queue

Writer - Simple - Settings

Writer - S3 - Settings

Special Sections

Besides the normal [default] and [node] sections, Capture/Viewer support several special configuration file sections. These sections are used to configure custom fields, views, and other more complex features. Be careful to not place normal config items in these special sections.

Special Sections - custom-fields

You can add custom fields to Arkime several ways, including the wise and tagger plugins. Since Arkime 1.5.2 the easiest way is to use a [custom-fields] section in the ini file. At capture startup it will check to make sure all those fields exist in the database. The format of the line is the similar as used in wise and tagger, except you use expression=definition. You will need to also create a [custom-views] section to display the data on the Sessions tab.

Example:

[custom-fields]
# Format is FieldExpr=text format
theexpression=db:theexpression

sample.md5=db:sample.md5;kind:lotermfield;friendly:Sample MD5;count:true;help:MD5 of the sample

Special Sections - custom-fields-remap

Starting with 5.0 it is now possible to remap results turned by wise or the tagger plugin to set different fields based on what field matched. For example instead of only being able to set the asset field based on if the ip.src or ip.dst field matched, it is now possible to set asset.src when ip.src matches or asset.dst when ip.dst matches. This is done by remapping the asset field.

Example:

[custom-fields-remap]
asset=ip.src=asset.src;ip.dst=asset.dst

[custom-fields]
asset.src=kind:lotermfield;count:true;friendly:Asset Src;db:assetName.src;help:Asset Name Src
asset.dst=kind:lotermfield;count:true;friendly:Aseet Dst;db:assetName.dst;help:Asset Name Dst

Special Sections - custom-views

With Arkime "views" are how the SPI data is displayed in the Sessions tab. Usually there is a unique "view" for each category of data. You can add custom views to Arkime several ways, including the wise and tagger plugins. Since Arkime 1.5.2 the easiest way is to use a [custom-views] section in the ini file. At viewer startup, a new section will be created for each entry. The format of the line is name=definition. Viewer will sort all views by name when choosing the order to display in.

Example:

[custom-views]
sample=title:Samples;require:sample;fields:sample.md5,sample.house

[custom-fields]
sample.md5=db:sample.md5;kind:lotermfield;friendly:Sample MD5;count:true;help:MD5 of the sample
sample.house=db:sample.house;kind:termfield;friendly:Sample House;count:true;help:House the sample lives in

Special Sections - headers-email

This section makes it easy to specify email headers to index. They will be searchable in the UI using email.[HEADERNAME]

Example:

[headers-email]
x-priority=type:integer

Special Sections - headers-http-request

This section makes it easy to specify HTTP Request headers to index. They will be searchable in the UI using http.[HEADERNAME]

Example:

[headers-http-request]
referer=type:string;count:true;unique:true

Special Sections - headers-http-response

This section makes it easy to specify HTTP Response headers to index. They will be searchable in the UI using http.[HEADERNAME]

Example:

[headers-http-response]
location=type:string
server=type:string

Special Sections - override-ips

override-ips is a special section that overrides the MaxMind databases for the fields set, but fields not set will still use MaxMind (example if you set tags but not country it will use MaxMind for the country) Spaces and capitalization are very important.

Since 5.0.0 you can now create secondary ini files that will auto reload on changes by setting the overrideIpsFiles setting.

Example:

[override-ips]
10.1.0.0/16=tag:ny-office;country:USA;asn:AS0000 This is neat

Special Sections - packet-drop-ips

This section allows you to specify ips or cidrs to drop from being processed. This is different from a bpf filter since the packets will actually reach capture (and counted) but won't be fully processed. However if you have many ranges/ips to drop it can be more efficient then bpfs. It is possible to also allow ranges inside of dropped ranges using the "allow" keyword. Order added doesn't matter, searching always finds the best match.

Since 5.0.0 you can now create secondary ini files that will auto reload on changes by setting the packetDropIpsFiles setting.

[packet-drop-ips]
10.0.0.0/8=drop
10.10.0.0/16=allow
10.10.10.0/24=drop
10.10.10.10=allow

Special Sections - remote-clusters

The remote-clusters (formerly moloch-clusters) section is used to describe the various Arkime clustersthat are available to forward traffic to either manually or through the periodic query functionality. Each line represents a single cluster, with the name just being any unique string.

Example:

[remote-clusters]
cluster1=url:https://arkime.example.com:8005;serverSecret:password;name:Cluster
cluster2=url:https://cluster2.example.com:8005;serverSecret:foo;name:Test Cluster

Special Sections - vlan-vni-collapse

(Since 5.2)When using the sessionIdTracking setting this section allows you to collapse VLAN or VNI into a single value. This is useful when multiple VLANs or VNIs might be used for the same session.

In this example the values 50, 100, 300, 400, and 500 will be collapsed into a single value of 0 and the values 125 and 325 will be collapsed into a single value of 25.

[vlan-vni-collapse]
0=50,100,300,400,500
25=125,325

Special Sections - wise-types

WISE also lets you configure which fields are used for the standard wise types and you can add your own wise types. You do this by creating a [wise-types] section in the capture configuration file AND listing the fields using `{type}={expression};{db:dbfield}...`. The type field must be less then 12 characters, and is the same type field you would use in the wise service.

Special Sections - Multi Viewer Settings

The multi viewer is useful when you have multiple Arkime clusters that you want to search across. To use the multi viewer, an extra viewer process and a multies process must be started. The viewer process works like a normal viewer process, except instead of talking to a OpenSearch/Elasticsearch server, it talks to a multies server. The multies server proxies the queries to all the real OpenSearch/Elasticsearch servers. These two processes can share the same config file and node name section. The viewer part uses the SAME configuration values as above if you need to set anything.

A sample configuration for multi viewer (the elasticsearch variable points to the multies.js process):

[multi-viewer]
elasticsearch=127.0.0.1:8200
viewPort = 8009
#viewHost = localhost
multiES = true
multiESPort = 8200
multiESHost = localhost
multiESNodes = http://escluster1.example.com:9200,name:escluster1,prefix:PREFIX,elasticsearchAPIKey:testkey1;http://escluster2.example.com:9200,name:escluster2

Which you would then use by starting both the multiviewer and multies. This is a sample for running manually (but you should setup startup scripts to run for real):

cd /opt/arkime/viewer
/opt/arkime/bin/node multies.js -n multi-viewer
/opt/arkime/bin/node viewer.js -n multi-viewer

Special Sections - Central Viewer Settings

Using a central viewer isn't required, but is useful when you want to funnel all your users thru a single viewer. This is especially useful when using a reverse proxy or the network topology requires it. There is no special configuration for the central viewer, just start it with the same configuration as the other viewers. Remember, you must still run a viewer on each capture node, unless you are using S3 storage.

A sample configuration for central viewer

[default]
passwordSecret =CHANGEME
serverSecret=CHANGEME
cronQueries=auto

[central-viewer]
elasticsearch=https://127.0.0.1:9200

# If using a reverse proxy make sure viewHost is localhost and viewPort is the port the reverse proxy will communicate to viewer on
viewHost = localhost
viewPort = 8009

cd /opt/arkime/viewer
/opt/arkime/bin/node viewer.js -n central-viewer

Value Actions (previously right-click)

It is possible to configure right click actions on any SPI data fields. Right click actions can be added based on either the field name or the category of the data. They can be added either in the configuration file or by enabled WISE data sources.

Configuration File Sample:

[value-actions]
VTIP=url:https://www.virustotal.com/en/ip-address/%TEXT%/information/;name:Virus Total IP;category:ip
VTHOST=url:https://www.virustotal.com/en/domain/%HOST%/information/;name:Virus Total Host;category:host
VTURL=url:https://www.virustotal.com/latest-scan/%URL%;name:Virus Total URL;category:url
KBN=url:https://localhost:5601/app/discover#/?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'%ISOSTART%',to:'%ISOSTOP%'))&_a=(columns:!(_source),index:'filebeat-*',interval:auto,query:(language:kuery,query:'network.community_id: "%URIEncodedText%"'));name:Discover communityId;fields:communityId

Each line in the valueactions section contains possible actions:

The possible URL Substitutions are:

The categories that Arkime uses are:

Field Actions

It is possible to configure actions to add to any SPI data field label. Field actions can be added based on either the field name or the category of the data. They can be added either in the configuration file or by enabled WISE data sources.

Configuration File Sample:

[field-actions]
TEST=url:https://www.test.com/?query=%EXPRESSION%:Test Action %FIELNAME%/;category:ip;users:testuser
      

Each line in the fieldactions section contains possible actions:

The possible URL Substitutions are:

The possible Field Action Menu Name Substitutions are:

PCAP at Rest Encryption

Arkime provides support for PCAP at rest encodings. Two forms of encodings are currently supported: "aes-256-ctr" and "xor-2048". Please note that xor-2048 is not actually secure and is only for testing.

The current implementation is based around openssl. The encoding that you wish to use is configured in the configuration file by setting the simpleEncoding variable. The simpleEncoding may be set either globally or per node.

Each file on disk is encoded by a unique data encryption key (dek). The dek is encrypted using a key encryption key (kek) when stored. The encrypted dek, the id of the kek, and the initialization vector (IV) are all stored per file in OpenSearch/Elasticsearch. Which kek is used when creating files, is selected with the simpleKEKId variable. The simpleKEKId may be set either globally or per node.

The kek passwords that may be used should be placed in a [keks] section of the configuration file. There is one line for each kekid to kek mapping. An easy way to create kek passwords is openssl rand -base64 30. Remember a kek is password used to encrypt the dek and NOT the password used to encrypt the files. The dek is what is used to encrypt the files, and is unique per file.

You MUST secure your configuration file.

You are not required to use the same keks on all nodes, however, you can if you wish. It is recommended that you rotate your keks occasionally (timing dependent on your risk tolerance) and create new keks to be used. Do NOT delete the old keks until all pcaps which have been encoded with those keks have been expired.

Currently it is not possible to re-encrypt a data encryption key, however, this should be possible in the future with a db.pl command.

Example:

[default]
pcapWriteMethod=simple
simpleEncoding=aes-256-ctr
simpleKEKId=kekid1

[keks]
kekid1=Randomkekpassword1
kekid2=Randomkekpassword2

Advantages:

• Don't need to mess with disk encryption
• Access to just the host doesn't give you access to the pcap file

Disadvantages:

• Normal pcap tools will no longer work
• If the files index in OpenSearch/Elasticsearch gets corrupt, you can not reingest the files

Plugins

Arkime supports several different plugins that can be added to the capture process. Plugins are enabled by setting the plugins variable in the configuration file. For exmple, to enable the ja4plus plugin, you would add plugins=ja4plus.amd64.so to the configuration file.

Plugins - CHAD

CHAD creates hashes for http and smtp headers.

Plugins - Kafka

(Since 4.2.0) Arkime can be decoupled from directly sending sessions to OpenSearch/Elasticsearch by leveraging Kafka as a message queue. This approach offers greater flexibility and scalability. Arkime will continue to interact directly with OpenSearch/Elasticsearch for other data storage needs.

To integrate Kafka:

1. Producer: Configure Arkime to publish session data to a specific Kafka topic.
2. Consumer: Develop a Kafka consumer that subscribes to the session topic. This consumer will be responsible for processing the session data and indexing it into OpenSearch/Elasticsearch. The consumer can determine the target OpenSearch/Elasticsearch index based on either:
   • Bulk Header: This can be used directly to index the data.
   • Document Field: Extracting the index name from a specific field within the session data.

Configuration:

• Update the Arkime configuration file by adding kafka.so to the plugins= line to enable the Kafka plugin.
• Add any of the following settings to the Arkime configuration file.
• (Since 5.0.0) It is possible to have full control of the librdkafka configuration by creating a new [kafka-config] section in the Arkime configuration file. Each item is an entry from the Global Configuration section and is applied AFTER the Arkime settings below.

Plugins - Lua

Arkime allows you to create simple lua scripts. See the Lua README for more information

Plugins - JA4+

Arkime supports JA4 and JA4+ algorithms starting from version 5 onwards.

JA4 algorithm

JA4, the TLS Client Fingerprinting portion, is built into the Arkime capture binary. After installing Arkime 5 or later, you will automatically see tls.ja4 field show up.

JA4+ algorithms

JA4+ algorithms have licensing requirements. Please familiarize yourself with them before installing/enabling the JA4+ portions of Arkime.

To install/enable the JA4+ algorithms, you need to:

• Download the ja4plus.amd64.so or ja4plus.arm64.so plugin. Please ensure to download it from the same location/version as the Arkime version you are using.
• Copy the downloaded plugin to the /opt/arkime/plugins directory on all machines that run capture.
• Edit the /opt/arkime/etc/config.ini file, modifying the plugins= line to include the downloaded filename, for example, plugins=ja4plus.amd64.so.
• Restart capture, which will automatically create and start using the new JA4 fields.

Plugins - Netflow

Arkime can generate netflow for all sessions it saves SPI data for. Add netflow.so to the plugins= line in the config file.

Plugins - ScrubSPI

Allows you to scrub data before sending to OpenSearch/Elasticsearch. To use add to plugins and create a configuration called scrubspi with each entry being a field with regex replace statement.

[scrubspi]
http.uri=/github/foohub/
asn.dst=:FOO:BAR:

Plugins - Suricata

Arkime can enrich sessions with Suricata alerts. Suricata and Arkime must see the same traffic and share the same eve.json/alert.json for the plugin to work. Sessions that have been enriched will have several new fields, all starting with suricata, and will be displayed in a Suricata section of the standard Arkime session UI. Arkime matches sessions based on the 5 tuple from the alert.json or eve.json file, only using the items with event_type of alert. A very simple query to find all sessions that have Suricata data is suricata.signature == EXISTS!.

Note: there isn't a special Suricata UI inside Arkime, this is just adding new fields to Arkime sessions like wise or tagger do. The Suricata enrichment is done by capture, so capture must see the traffic.

Add suricata.so to the plugins= line in the config file.

Sample Config:

# Add suricata.so to your plugins line, or add a new plugins line
plugins=suricata.so

# suricataAlertFile should be the full path to your alert.json or eve.json file
suricataAlertFile=/nids/suricata/eve.json

Plugins - Tagger

See the Tagger page and the Tagger Format page

Plugins - TCP Health Check

TCP Health Check plugin enables the capture module to listen on a specific TCP port and immediately close accepted connections. This is useful to keep load balancer health checks informed about the capture module working properly. One specific use case is Health Checks for Target Groups in AWS Network Load Balancer.

Add tcphealthcheck.so to the plugins= line in the config file.

Plugins - WISE

The With Intelligence See Everyting (WISE) plugin is used to communitcate to the wiseService Arkime application. Learn more about the wiseServce.

Each capture node needs to have the wise plugin enabled. You will need to make a few changes to the [default] section of the configuration file.

1. Add wiseURL=http://WISEHOST:8081
2. Enable the plugin by adding wise.so to the `plugins=` variable, it should be the last entry.

Each viewer node that a operator uses needs to have the wise plugin enabled. You will need to make a few changes to the [default] section of the configuration file.

1. Add wiseURL=http://WISEHOST:8081
2. Enable the plugin by adding wise.js to the viewerPlugins= variable.

Usually just setting wiseURL is enough.

WISE also lets you configure which fields are used for the standard wise types and you can add your own wise types. You do this by creating a [wise-types] section in the capture configuration file AND listing the fields using `{type}={expression};{db:dbfield}...`. The type field must be less then 12 characters, and is the same type field you would use in the wise service.

High Performance Settings

Sample config items for max performance. Most of the defaults are fine. Reading Arkime FAQ - Why am I dropping packets and https://github.com/pevma/SEPTun or https://github.com/pevma/SEPTun-Mark-II may be helpful.

# MOST IMPORTANT, use basic magicMode, libfile kills performance
magicMode=basic

# Optionally disable compression, this will use more disk space but less CPU.
# simpleCompression=none
# Switch from gzip to zlib if you want compression, zstd uses less CPU than gzip
# simpleCompression=zstd

pcapReadMethod=tpacketv3
tpacketv3BlockSize=8388608

# Increase by 1 if still getting Input Drops
tpacketv3NumThreads=2

pcapWriteMethod=simple
pcapWriteSize=4194304

# Start with 5 packet threads, increase by 1 if getting thread drops.  Should be about 2 x Gbps that need to be captured
packetThreads=5

# Increase the size of ES messages and compress them for lower traffic rates
dbBulkSize=4000000
compressES=true
dbEsHealthCheck=false

# Approximate max streams that will be monitored
maxStreams=2000000

# Set to number of packets a second, if still overflowing try 400k
maxPacketsInQueue = 300000

# Uncomment to disable features you don't need
# parseQSValue=false
# parseCookieValue=false

The following rules can help greatly reduce the number of SPI sessions being written to disk:

---
version: 1
rules:
- name: "Dont write tls packets or check yara after 10 packets, still save SPI"
  when: "fieldSet"
  fields:
    protocols:
    - tls
  ops:
    _dontCheckYara: 1
    _maxPacketsToSave: 10

- name: "Dont save SPI sessions to ES with only 1 src packet"
  when: "beforeFinalSave"
  fields:
    packets.src: 1
    packets.dst: 0
    tcpflags.syn: 1
  ops:
    _dontSaveSPI: 1

- name: "Dont save SPI data for listed hostnames tracked by dst ip:port, use on cloud destinations"
  when: "fieldSet"
  fields:
    host.http:
    - ad.doubleclick.net
    protocols:
    - tls
  ops:
    _dontSaveSPI: 1
    _maxPacketsToSave: 1
    _dropByDst: 10

Lab Settings

Arkime by default is not configured to monitor low bandwidth networks. You'll want to update the configuration to have a better experience when testing in a lab or low bandwidth setting.

# Only use 1 packet thread so only 1 file is written at a time
packetThreads=1

# Disable compression since this will cause buffering and
# cause viewer to not show packets until buffer is written
simpleCompression=none

# Lower pcap buffer to smallest possible, although Arkime will write pageSize
# blocks if a timer fires.
pcapWriteSize=65536

# Lower the number of streams expected for lower memory usage
maxStreams=1500

# Decrease the size of writes to DB
dbBulkSize=100000
maxESConns=2

Cont3xt

Cont3xt, by default, utilizes configuration files using INI format. The configuration file is located at /opt/arkime/etc/cont3xt.ini, but its location can be altered using the -c command-line option. With the release of Arkime 5, support was introduced for configuration files in JSON and YAML formats. These must be have json or yaml extensions, respectively. When utilizing JSON or YAML, arrays can be specified either natively or using separators like INI.

The configuration file can contain sections for Cont3xt itself and then sections for each integration. The sections for each integration are optional and can be used to set specific settings for each integration. Usually, integration keys will be set up by each user, but it is possible to have global keys.

General

DB

Caching

Common settings per integration

Arkime Integration

Clickhouse Integration

CSV Integration

Databricks Integration

Example config that will query Databricks for all the IPs that match.

  [databricks:users]
  itypes = ip
  name = DataBricks Users
  host = abc-123456-789.cloud.databricks.com
  path = /sql/1.0/warehouses/abcxyz125789
  token = THESUPERSECRETTOKEN
  statement=SELECT * FROM catalog.schema.table WHERE ip = :SEARCHTERM

OpenSearch/Elasticsearch Integration

JSON Integration

Redis Integration

Parliament

Parliament contains a grouped list of your Arkime clusters with links, Elasticsearch/OpenSearch health, and issues for each. You can use Parliament as a landing page for all of your Arkime clusters and as a status page to monitor the health of your clusters. See Parliament for more information.

Parliament Settings

(Since 5.0.0) These settings must be in the [parliament] section, either in their own file or in shared configuration file. The issues file setting must exist and is set to your Parliament JSON file.

When upgrading to 5.x of Parliament there are some important changes. If you were passing in port/certFile/keyFile into the command line arguments when starting Parliament, you must also include those in this configuration file. It is recommended that you configure the Auth section in the Parliament UI on the Setting page before upgrading to 5.x. The upgrade will update the configuration with your authentication settings, otherwise you will need to input them manually as basic passworth auth has been disabled for Parliament.

Sample configuration:

[parliament]
# Where to store issues
file=../etc/parliament.issues.json

# Port to listen on
#port=8008

### Parliament supports authentication like viewer/cont3xt, make sure to use the same settings across all tools
### Important to use the usersElasticsearch/usersPrefix settings and not the elasticsearch/prefix settings
#usersElasticsearch=http://localhost:9200
#usersPrefix=arkime
#authMode=digest
#passwordSecret=password
#httpRealm=Moloch

WISE Service

WISE Service, by default, utilizes configuration files using INI format. The configuration file is located at /opt/arkime/etc/wiseService.ini, but its location can be altered using the -c command-line option. With the release of Arkime 5, support was introduced for configuration files in JSON and YAML formats. These must be have json or yaml extensions, respectively. When utilizing JSON or YAML, arrays can be specified either natively or using separators like INI.

The configuration file can contain sections for wiseService itself and then sections for each WISE source. WISE will not try and query sources until a section is created for them.

General

General settings for WISE, located in a [wiseService] section.

Caching

The wiseService caches all results returned by external sources. An external source is something like OpenDNS or Reverse DNS, where it is impossible to load the entire data set into memory and WISE needs to make a external query to obtain results. All caching is done in memory, however Redis can be used for a larger shared cache.
Create a [cache] section

Common Source Settings

Glob Rules:

• To match zero or more characters either `*` or `%` may be used
• To match a single digit `#` may be used
• To match a character with special meaning, you must precede it with "~". The "~" also serves as the escape character
• To match one of a specific list of characters, you may surround the character with square brackets
• Globs are anchorced on both end, use `*` to unanchor
• String case doesn't matter

Alien Vault

The Alien Vault data source currently uses the downloadable database that is updated often. Requires that access be purchased and configured.
Create a [alienvault] section to configure

Emerging Threats Pro

The Emerging Threats Pro data source currently uses the downloadable database that is updated often. Requires that access be purchased and configured.
Create a [emergingthreats] section to configure

OpenDNS Umbrella

The OpenDNS source currently uses the bulk query api and does live queries. Requires that access be purchased and configured.
Create a [opendns] section to configure

ThreatQ

The ThreatQ export interval time should be configured depending on process requirements and indicator volume.
Create a [threatq] section to configure

ThreatStream

WISE can integration with Anomali Threatstream. If doing a large amount of queries, one of the download methods is recommended. Requires that access be purchased and configured.
Create a [threatstream] section to configure.

Splunk

The Splunk wise service can run in two different modes. It can query Splunk for every value or it can query Splunk periodicly for a table of values. Many Splunk operators prefer the periodic query since they can scale for it.
Create a [splunk:UNIQUENAME] section to configure

Example config that will query Splunk for all the vpn_ip to user name mappings during the last 24 hours every 60 seconds. It will then set the user field for any ip that matches.

  [splunk:users]
  type = ip
  format = json
  host = spunk.example.com
  port=5500
  username=theuser
  password=thepassword
  periodic=60
  query=search index="THEINDEX" sourcetype="vpn" assigned earliest=-24h | rex "User <(?[^>]+)>.*IPv4 Address <(?[^>]+)>" | dedup vpn_ip | table user, vpn_ip
  keyColumn=vpn_ip
  fields=field:user;shortcut:user

Databricks

(Since 5.5.1) The Databricks wise service can run in two different modes. It can query Databricks for every value or it can query Databricks periodicly for a table of values. Many Databricks operators prefer the periodic query since they can scale for it.
Create a [databricks:UNIQUENAME] section to configure

Example config that will query Databricks for all the vpn_ip to user name mappings during the last 24 hours every 60 seconds. It will then set the user field for any ip that matches.

  [databricks:users]
  type = ip
  host = abc-123456-789.cloud.databricks.com
  path = /sql/1.0/warehouses/abcxyz125789
  token = THESUPERSECRETTOKEN
  periodic=60
  query=SELECT * FROM catalog.schema.table WHERE ip = :SEARCHTERM
  keyPath=vpn_ip
  fields=field:user;shortcut:user

Elasticsearch

The Elasticsearch wise service can query OpenSearch or Elasticsearch for fields to set
Create a [elasticsearch:UNIQUENAME] section to configure

Example config that will query OpenSearch/Elasticsearch for an ip that is in the 10.172/16 space, in the index TheIndex-\*, only looking at records that have a \@timestamp field newer than 86400000ms. It looks at the `cef_ext.src` field and only looks at records that has a cef_ext.suser field set. Once it has a result it sets the user field in arkime to whatever the `cef_ext.suser` field is in the document.

  type=ip
  onlyIPs=10.172.0.0/16
  elasticsearch=http://ELKCLUSTERHOST1:9200,http://ELKCLUSTERHOST2:9200
  esIndex=TheIndex-*
  esTimestampField=@timestamp
  esQueryField=cef_ext.src
  esMaxTimeMS=86400000
  esResultField=cef_ext.suser
  fields=field:user;shortcut:cef_ext.suser

File

Like the url source, use a single elasticsearch document as the file. The document can be periodically reloaded.
Create a [elasticsearchfile:UNIQUENAME] section to configure

File

The wiseService can monitor multiple files. Each file needs to have its own section, with the section name starting with `file:`. The wiseService automatically notices if the file changes and reloads it.
Create a [file:UNIQUENAME] section to configure

CSV Example

Config File

  [file:ipcsv]
  file=./ip.wise.csv
  tags=ipwisecsv
  type=ip
  column=1
  format=csv
  #Asset field already exist, use field 0 for value. extra field is new, use field 2 for value
  fields=field:asset;shortcut:0\nfield:extra;kind:lotermfield;count:true;friendly:extra;db:extra;help:Help for Extra;shortcut:2

The CSV File

  blah1,10.0.0.3
  blah2,10.0.0.2,foo

Tagger Example

Config File

  [file:iptagger]
  file=./ip.wise.tagger
  tags=ipwisetagger
  type=ip
  format=tagger

The Tagger File

  #field:extra;kind:lotermfield;count:true;friendly:extra;db:extra;help:Help for Extra
  10.0.0.3;asset=blah1
  10.0.0.2;asset=blah2;extra=foo

JSON Example

Config File

  [file:ipcsv]
  file=./ip.wise.json
  tags=ipwisejson
  type=ip
  keyColumn=theip
  format=json
  #Asset field already exist, use field asset for value. extra field is new, use field extra for value
  fields=field:asset;shortcut:asset\nfield:extra;kind:lotermfield;count:true;friendly:extra;db:extra;help:Help for Extra;shortcut:extra\n

The JSON File

  [{"asset": "blah", "theip": "10.0.0.3"},
   {"asset": "blah2", "theip": "10.0.0.2", "extra": "foo"}]

**Note:** you use shortcut to match between fields in the JSON dictionary and the properties in OpenSearch/Elasticsearch.

JSONL Example

Added in 5.0, WISE now supports jsonl files with one full json object per line.

Config File

  [file:ipcsv]
  file=./ip.wise.jsonl
  tags=ipwisejsonl
  type=ip
  keyColumn=theip
  format=jsonl
  #Asset field already exist, use field asset for value. extra field is new, use field extra for value
  fields=field:asset;shortcut:asset\nfield:extra;kind:lotermfield;count:true;friendly:extra;db:extra;help:Help for Extra;shortcut:extra\n

The JSON File

  {"asset": "blah", "theip": "10.0.0.3"}
  {"asset": "blah2", "theip": "10.0.0.2", "extra": "foo"}

**Note:** you use shortcut to match between fields in the JSONL dictionary and the properties in OpenSearch/Elasticsearch.

Subnets Example

More complex example of the above where you want to create a new section

  [file:subnets]
  type=ip
  format=json
  file=subnets.json
  keyColumn=ip
  fields=field:subnets.description;kind:termfield;count:true;friendly:Description;db:subnets.description;help:Description;shortcut:description\nfield:subnets.securityzone;kind:termfield;count:true;friendly:Security Zone;db:subnets.securityzone;help:Security Zone;shortcut:securityZone\nfield:subnets.vlan;kind:integer;count:true;friendly:Vlan;db:subnets.vlan;help:Vlan;shortcut:vlan\nfield:subnets.site;kind:termfield;count:true;friendly:Site;db:subnets.site;help:Site;shortcut:site
  # Jade view method
  view=if (session.subnets)\n    +arrayList(session.subnets, 'description', 'Description', 'subnets.description')\n    +arrayList(session.subnets, 'label', 'Label', 'subnets.label')\n    +arrayList(session.subnets, 'securityzone', 'Security Zone', 'subnets.securityzone')\n    +arrayList(session.subnets, 'vlan', 'Vlan', 'subnets.vlan')\n    +arrayList(session.subnets, 'site', 'Site', 'subnets.site')
  # Simple view method
  #view=require:subnets;title:Subnets;fields:subnets.description,subnets.label,subnets:securityzone,subnets.vlan,subnets.site

The JSON File

  [{"description": "the description", "label": "interesting label", "securityzone": "hot", "vlan": 123, "site": "secret",  "ip": "10.0.0.3"},
   {"description": "the description2", "label": "interesting label2", "securityzone": "cold", "vlan": 123, "site": "secret",  "ip": "10.0.0.2"}]

Passive Total

WISE can integration with Passive Total to provide additional threat intelligence.
Create a [passive] section to configure.

Redis

Redis can be used as a wise data source. If using redis as a wise cache you'll probably want to use a different redis "database" by specifying the database number in the url. Example: `url=redis://localhost/1` Each redis source can only handle one type of data, although multiple redis sources can be configured and they can use the same redis database.
Create a [redis:UNIQUENAME] section to configure.

File

Like the url source, use a single elasticsearch document as the file. The document can be periodically reloaded.
Create a [elasticsearchfile:UNIQUENAME] section to configure

ReverseDNS

For IPs that are included by the ips setting, do a reverse lookup and place everything before the first dot in the field specified.
Create a [reversedns] section to configure

Virus Total

WISE can integration with Virus Total to provide additional threat intelligence.
Create a [virustotal] section to configure.

URL

The wiseService can monitor and download URL. Each url needs to have its own section, with the section name starting with `url:`. The wiseService can automatically download and reload the files.
Create a [url:UNIQUENAME] section to configure

Value Actions

Not really a WISE data source, this source monitors configured files, redis urls, or OpenSearch/Elasticsearch urls for valueactions (previously right-click) to send to all the viewer instances that connect to this WISE Server. Each file needs to have its own section, with the section name starting with `valueactions:`. The format of the monitored files is the same as [WISE](settings#wise). It will auto reload the valueactions files if they change.
Create a [valueactions:UNIQUENAME] section to configure

So for example you might have

  [valueactions:virustotal]
  url=/opt/arkime/etc/valueactions-virustotal.ini

and then /opt/arkime/etc/valueactions-virustotal.ini could contain

  VTIP=url:https://www.virustotal.com/en/ip-address/%TEXT%/information/;name:Virus Total IP;category:ip
  VTHOST=url:https://www.virustotal.com/en/domain/%HOST%/information/;name:Virus Total Host;category:host
  VTURL=url:https://www.virustotal.com/latest-scan/%URL%;name:Virus Total URL;category:url

FIeld Actions

Not really a WISE data source, this source monitors configured files, redis urls, or OpenSearch/Elasticsearch urls for field actions to send to all the viewer instances that connect to this WISE Server. Each file needs to have its own section, with the section name starting with `fieldactions:`. The format of the monitored files is the same as [WISE](settings#wise). It will auto reload the fieldactions files if they change.
Create a [fieldactions:UNIQUENAME] section to configure