SimpleSource (class)

The SimpleSource base class implements some common functions for sources that only have one type. Sources that extend this class only need to worry about fetching the data, and only need to implement the constructor and simpleSourceLoad.

Sources need to

  • implement WISESource#initSource
  • implement SimpleSource#simpleSourceLoad
  • they can optionally call this.load() if they want to force a reload of data

Extends: WISESource

new SimpleSource(api, section, options) (constructor)

Create a simple source. The options dontCache, formatSetting, tagsSetting, typeSetting will all be set to true automatically.

Parameters:

Param Type Description
api WISESourceAPI the api when source created passed to initSource
section string the section name
options object see WISESource constructor for common options
options.reload integer If greater than zero, call simpleSourceLoad every options.reload minutes

simpleSource.tagsResult (member)

The encoded tags result if options.tagsSetting was set to true

Overrides: tagsResult

simpleSource.format (member)

The format of the source if options.formatSetting was set to true.

Overrides: format

simpleSource.parse (member)

{function} The parser function of the source if options.formatSetting was set to true.

Overrides: parse

simpleSource.type (member)

The wise item type of the source if options.typeSetting was set to true.

Overrides: type

simpleSource.emptyResult (member)

A simple constant that should be used when needing to represent an empty result

Overrides: emptyResult

simpleSource.dump() (function)

Implemented for simple sources

Overrides: dump

simpleSource.itemCount() (function)

Implemented for simple sources

Overrides: itemCount

simpleSource.load() (function)

This loads the data for the simple source. SimpleSource will call on creation and on reloads. It can also be called by the source to force a reload of the data.

simpleSource.simpleSourceLoad(cb) (function)

Each simple source must implement this method. It should call the callback with either the error or the entire body of the text to parse. The SimpleSource class will take care of parsing the data. It will be called after the constructor and periodically if reloading is enabled.

Parameters:

Param Type Description
cb function (err, body)

simpleSource.parseFieldDef(line) (function)

Parse a field definition line and call the addField or addView as needed

Overrides: parseFieldDef

Parameters:

Param Type Description
line string the line to parse

simpleSource.parseCSV(body, setCb, endCb) (function)

Util function to parse CSV formatted data

Overrides: parseCSV

Parameters:

Param Type Description
body string the raw CSV data
setCb function the function to call for each row found
endCb function all done parsing

simpleSource.parseTagger(body, setCb, endCb) (function)

Util function to parse tagger formatted data

Overrides: parseTagger

Parameters:

Param Type Description
body string the raw tagger data
setCb function the function to call for each row found
endCb function all done parsing

simpleSource.parseJSONArray(json, setCb, endCb) (function)

Util function to parse JSON formatted data

Overrides: parseJSONArray

Parameters:

Param Type Description
json array the parsed JSON array
setCb function the function to call for each row found
endCb function all done parsing

simpleSource.parseJSONElement(json) (function)

Util function to walk a JSON object using the configured shortcuts and build an args array suitable for encodeResult.

Each shortcut path may traverse intermediate arrays as well as a final array value: e.g. with shortcut a.b.c, the input { a: [ { b: { c: "v1" } }, { b: { c: "v2" } } ] } yields both v1 and v2. With shortcut d.e, the input { d: { e: ["v3", "v4"] } } yields both v3 and v4.

Overrides: parseJSONElement

Parameters:

Param Type Description
json object the JSON object (single element) to extract from

Returns:

Name Type Description
  Array alternating pos/value entries for encodeResult

simpleSource.parseJSON(body, setCb, endCb) (function)

Util function to parse JSON formatted data

Overrides: parseJSON

Parameters:

Param Type Description
body string the raw JSON data
setCb function the function to call for each row found
endCb function all done parsing

simpleSource.parseJSONL(body, setCb, endCb) (function)

Util function to parse JSONL formatted data

Overrides: parseJSONL

Parameters:

Param Type Description
body string the raw JSONL data
setCb function the function to call for each row found
endCb function all done parsing

simpleSource.getSourceRaw(cb) (function)

Get the raw source data for editing. Source should implement this method if they want to support editing the data for a source.

Overrides: getSourceRaw

Parameters:

Param Type Description
cb function (err, data)

simpleSource.putSourceRaw(data, cb) (function)

Put the raw source data after editing. Source should implement this method if they want to support editing the data for a source.

Overrides: putSourceRaw

Parameters:

Param Type Description
data string The full data for the source from UI
cb function (err)

WISESourceAPI (class)

When sources are created they get an api object to interact with the wise service.

wiseSourceAPI.debug (member)

Current debug level of wiseService

wiseSourceAPI.insecure (member)

Is wiseService running in insecure mode

wiseSourceAPI.getConfig (member)

Get from the config section a value or default

Parameters:

Param Type Description
section string The section in the config file the key is in
sectionKey string The key to get from the section
[default] string the default value to return if key is not found in section

Returns:

Name Type Description
  string The value found or the default value

wiseSourceAPI.getConfigSections (member)

Get a list of all the sections in the config file

Returns:

Name Type Description
  string | Array A list of all the sections in the config file

wiseSourceAPI.getConfigSection (member)

Get the full config for a section

Parameters:

Param Type Description
section string The section of the config file to return

Returns:

Name Type Description
  object The full config for the section

wiseSourceAPI.addField(field) (function)

Add a field

Parameters:

Param Type Description
field string An encoded field definition

wiseSourceAPI.addView(viewName, view) (function)

Add a view

Parameters:

Param Type Description
viewName string Name of the new view
view string An encoded view definition

wiseSourceAPI.addSource(section, src, types) (function)

Activate a section of a source. Must be called if you want wise to query the source. A section is an instance of a source, some sources can have multiple sections.

Parameters:

Param Type Description
section string The section name
src WISESource A WISESource object
types Array An array of the types that this source supports

wiseSourceAPI.addSourceConfigDef(sourceName, configDef) (function)

Add for each source config definition for the UI to use.

Parameters:

Param Type Description
sourceName string The source name
configDef SourceConfig The configuration of this source type

wiseSourceAPI.addValueAction(actionName, action) (function)

Add a value action set

Parameters:

Param Type Description
actionName string The globally unique name of this action, not shown to user
action ValueAction The action

wiseSourceAPI.addFieldAction(actionName, action) (function)

Add a field action set

Parameters:

Param Type Description
actionName string The globally unique name of this action, not shown to user
action ValueAction The action

WISESourceAPI~SourceConfigField Type

Define all configuration for a field for a source

Properties

Name Type Default Description
name string   The name of the field
[required] boolean false Must the config value be filled out
[password] boolean false Is it a password type field that should be hidden
[multiline] string   If set this should be split using the value and shown in the UI as a text area
help string   The help text to show the user about the field
[ifField] string   Only show the field if the ‘ifField’ field is set and is equal to ‘ifValue’
[ifValue] string   Only show the field if the ‘ifValue’ field is set and is equal to ‘ifValue’
[regex] string   The value must match the regex to be considered valid

WISESourceAPI~SourceConfig Type

Define all the configuration for a source. This is used by the UI to generate what to display to the admin.

Properties

Name Type Default Description
name string   The name of the source
singleton boolean   Only one instance of this source can be configured
description string   Friendly text about the source
types Array   List of WISE types the source supports
[cacheable] boolean true Can the source be cached by WISE
fields SourceConfigField | Array   The fields for the source

WISESourceAPI~ValueAction Type

Define a value action (right click menu item)

Properties

Name Type Description
key string The key must be unique and is also used as the right click menu name if the name field is missing
name string The name of the value action to show the user
[url] string The url to send the user, supports special substitutions, must set url or func
[func] string A javascript function body to call, will be passed the name and value and must return the value, must set url or func
[actionType] string If set to ‘fetch’ this will replace the menu option with the results of url or func
[category] string Which category of fields should the value action be shown for, must set fields or category. View available categories
[fields] string Which fields to show the value action for, must set fields or category
[regex] string When set replaces %REGEX% in the url with the match
[users] string A comma separated list of user names that can see the right click item. If not set then all users can see the right click item.
[notUsers] string (Since 3.0) A comma separated list of user names that can NOT see the right click item. This setting is applied before the users setting above.

WISESource (class)

All sources need to have the WISESource as their top base class.

new WISESource(api, section, options) (constructor)

Should only be created by super(api, section, options) call

Parameters:

Param Type Default Description
api WISESourceAPI   the api when source created
section string   the section name
options object   All the options
[options.dontCache] boolean false do not cache this source, the source handles itself
[options.cacheTimeout] integer cacheAgeMin*60 or 60 override the cacheAgeMin setting, -1 same as dontCache
[options.tagsSetting] boolean false load the optional tags setting
[options.typeSetting] boolean false load the required type setting
[options.formatSetting] boolean false load the format setting with default the provided value if not false
[options.fullQuery] boolean false for MD5/SHA, query will be query.value and query.contentType

wiseSource.tagsResult (member)

The encoded tags result if options.tagsSetting was set to true

wiseSource.format (member)

The format of the source if options.formatSetting was set to true.

wiseSource.parse (member)

{function} The parser function of the source if options.formatSetting was set to true.

wiseSource.type (member)

The wise item type of the source if options.typeSetting was set to true.

wiseSource.emptyResult (member)

A simple constant that should be used when needing to represent an empty result

wiseSource.parseFieldDef(line) (function)

Parse a field definition line and call the addField or addView as needed

Parameters:

Param Type Description
line string the line to parse

wiseSource.parseCSV(body, setCb, endCb) (function)

Util function to parse CSV formatted data

Parameters:

Param Type Description
body string the raw CSV data
setCb function the function to call for each row found
endCb function all done parsing

wiseSource.parseTagger(body, setCb, endCb) (function)

Util function to parse tagger formatted data

Parameters:

Param Type Description
body string the raw tagger data
setCb function the function to call for each row found
endCb function all done parsing

wiseSource.parseJSONArray(json, setCb, endCb) (function)

Util function to parse JSON formatted data

Parameters:

Param Type Description
json array the parsed JSON array
setCb function the function to call for each row found
endCb function all done parsing

wiseSource.parseJSONElement(json) (function)

Util function to walk a JSON object using the configured shortcuts and build an args array suitable for encodeResult.

Each shortcut path may traverse intermediate arrays as well as a final array value: e.g. with shortcut a.b.c, the input { a: [ { b: { c: "v1" } }, { b: { c: "v2" } } ] } yields both v1 and v2. With shortcut d.e, the input { d: { e: ["v3", "v4"] } } yields both v3 and v4.

Parameters:

Param Type Description
json object the JSON object (single element) to extract from

Returns:

Name Type Description
  Array alternating pos/value entries for encodeResult

wiseSource.parseJSON(body, setCb, endCb) (function)

Util function to parse JSON formatted data

Parameters:

Param Type Description
body string the raw JSON data
setCb function the function to call for each row found
endCb function all done parsing

wiseSource.parseJSONL(body, setCb, endCb) (function)

Util function to parse JSONL formatted data

Parameters:

Param Type Description
body string the raw JSONL data
setCb function the function to call for each row found
endCb function all done parsing

wiseSource.itemCount() (function)

For sources that support it, get the number of items loaded into memory.

Returns:

Name Type Description
  integer the number of items loaded into memory

wiseSource.getSourceRaw(cb) (function)

Get the raw source data for editing. Source should implement this method if they want to support editing the data for a source.

Parameters:

Param Type Description
cb function (err, data)

wiseSource.putSourceRaw(data, cb) (function)

Put the raw source data after editing. Source should implement this method if they want to support editing the data for a source.

Parameters:

Param Type Description
data string The full data for the source from UI
cb function (err)

wiseSource.dump(res) (function)

Dump the sources data to caller. Source should implement this method if they want to support displaying the current state.

Parameters:

Param Type Description
res object The express res object

WISESource.encodeResult() (function)

Convert field ids and string values into the encoded form used in WISE.

This method takes a variable number of arguments, each in a pair of field id and string value.

Returns:

Name Type Description
  buffer the encoded results

WISESource.combineResults(results) (function)

Combine an array of encoded results into one encoded result

Parameters:

Param Type Description
results object | array Array of results

Returns:

Name Type Description
  Buffer A single combined result

WISESource.result2JSON(results) (function)

Convert an encoded combined results binary buffer into JSON string

Parameters:

Param Type Description
results Buffer The combined results from encode

Returns:

Name Type Description
  string The JSON string

WISESource.isSafeFile(file) (function)

Is the file a plain file that we own, so safe to use in a shared directory.

Parameters:

Param Type Description
file string The file to check

Returns:

Name Type Description
boolean boolean  

WISESource.request(url, file, cb) (function)

Download a url and save to a file, if we already have the file and less than a minute old use that file.

Parameters:

Param Type Description
url string The URL to download
file string The file to save the results to
cb function (statusCode) The status code result from the download

WISESource.initSource(api) (function)

Every source needs to implement this method. If a singleton it will just create the source object directly. If not it should loop through all keys that start with sourcekind:

Parameters:

Param Type Description
api WISESourceAPI The api back into the WISE Service

Example

exports.initSource = function (api) {
  api.addSourceConfigDef('sourcename', {
    singleton: false,
    name: 'sourcename',
    description: 'This is the best source ever',
    fields: [
      { name: 'type', required: true, help: 'The wise query type this source supports' },
      { name: 'tags', required: false, help: 'Comma separated list of tags to set for matches', regex: '^[-a-z0-9,]+' }
    ]
  });
  new TheSource(api);
}

/_ns_/nstest.html API

GET - Health check URL

/fields API

GET - Used by capture to retrieve all the fields created by wise sources

Parameters:

Param Type Default Description
[ver] integer 0 Version of the encoded binary to return

Returns:

Name Type Description
binary binary  

/views API

GET - Used by viewer to retrieve all the views being created by wise sources

Returns:

Name Type Description
  object All the views

/valueActions API

GET - Used by viewer to retrieve all the field value actions created by wise sources

Returns:

Name Type Description
  object | array All the actions

/fieldActions API

GET - Used by viewer to retrieve all the field actions created by wise sources

Returns:

Name Type Description
  object | array All the field actions

/get API

POST - Used by capture to lookup all the wise items

Parameters:

Param Type Default Description
ver integer 0 The format of the post data, version 0 and 2 supported
hashes string | array   A comma separated list of md5 hashes of field arrays that the client knows about. If one of the hashes matches the current field array, then we don’t send the field array.

Returns:

Name Type Description
  binary The encoded results

/sources API

GET - Used by wise UI to retrieve all the sources (unauthenticated).

Returns:

Name Type Description
  string | array All the sources

/config/defs API

GET - Used by wise UI to retrieve all the configuration definitions for the various sources (unauthenticated).

Returns:

Name Type Description
object object  

/api/locales API

GET - Retrieve all available locale files for internationalization

Returns:

Name Type Description
  object Object containing all locale data

/types API

GET - Used by the wise UI to retrieve all the types known (unauthenticated).

Returns:

Name Type Description
  string | array all the types

/types/:source API

GET - Used by the wise UI to retrieve all the types for a source, or if no source all the types known.

Parameters:

Param Type Description
{:source} string the source to get the types for

Returns:

Name Type Description
  string | array all the types for the source

/:source/:type/:key API

GET - Query a single source for a key

Parameters:

Param Type Description
{:source} string The source to get the results for
{:type} string The type of the key
{:key} string The key to get the results for

Returns:

Name Type Description
  object | array The results for the query

/:type/:key API

GET - Query all sources for a key

Parameters:

Param Type Description
{:type} string The type of the key
{:key} string The key to get the results for

Returns:

Name Type Description
  object | array The results for the query

/stats API

GET - Query for the stats

Returns:

Name Type Description
  object Object with array of stats per type and array of stats per source

/api/user API

GET - /api/user

Fetches the currently logged in user, so the UI can gate admin-only actions. This is an authenticated API and requires wiseService to be started with –webconfig.

Returns:

Name Type Description
  ArkimeUser The currently logged in user.

/source/:source/get API

GET - Used by wise UI to retrieve the raw file being used by the section. This is an authenticated API and requires wiseService to be started with –webconfig.

Parameters:

Param Type Description
:source string The source to get the raw data for

Returns:

Name Type Description
{success, raw} object The raw source data

/source/:source/put API

PUT - Used by wise UI to save the raw file being used by the source. This is an authenticated API and requires wiseService to be started with –webconfig.

Parameters:

Param Type Description
:source string The source to put the raw data for

Returns:

Name Type Description
{success, text} object The result of saving the raw source data

/config/get API

GET - Used by wise UI to retrieve the current config. This is an authenticated API and requires wiseService to be started with –webconfig.

Returns:

Name Type Description
object object  

/config/save API

PUT - Used by wise UI to save the current config. This is an authenticated API, requires the pin code, and requires wiseService to be started with –webconfig.