Searchable custom stats metadata in Assets Server – Help Center

Custom stats records can be stored in the Assets Server stats index through the logUsage API. The metadata that is stored in these stats records can also contain custom metadata.

Searching and especially aggregating (reporting) on this data can be very useful to perform reporting on the data stored in the stats index.

This article describes how to use the logUsage API to properly store your custom data.

Examples of querying or aggregating on stats data can be found in the source files of the Assets Server stats project.

Field type mapping

Defining how the field should be indexed by Assets Server is done by specifying a postfix in the following format:

field_name__xx

Where 'xx' points to a specific analyzer with which the data is analyzed. (For an overview of the postfixes that can be used, see below.)

Example: Assume you want to store statistics information about where and when an asset was used (published). In a report, you want to show how many assets where published in a certain period. To make such a report, you would have to store the date on which an asset was published so that you can later on search on a period.

To achieve this, the field is configured with the postfix __dt ('dt' stands for 'date/time', see supported field type mappings below).

The logUsage API call would be as follows:

http://demo.assets-server.com/services/logUsage
    ?assetId=5LMAQTW9qfhANn7f6JJVpl
    &action=CUSTOM_ACTION_PUBLISH
    &published__dt=2018-02-24

This indexes a new stats record with a custom metadata field called 'published_dt'. The field is indexed as a date field as it uses the '__dt' postfix. There is no additional configuration needed here, the type definition is configured automatically with the first logUsage call where the field is provided.

Important: As mapping types are fixed, it is advised to consider beforehand how your metadata should be mapped before indexing any stats data. Once you data is indexed for a field for the first time, the type mapping is configured and cannot be changed anymore. When data is indexed in a field which cannot be analyzed, the stats record entry will be ignored and an error is logged in the Assets Server log file.

Supported field type mappings

The following overview shows all supported mapping types and a description of their behavior.

Note: Elasticsearch supports a wide range of query types; it is out of scope to discuss them here in detail. The examples are limited to term, range and match queries.

default (no postfix)

Description: When there is no postfix specified, Assets Server will interpret the field as a standard text field. This means the text is tokenized into separate tokens (words or part of words) using the Elasticsearch English analyzer. Querying is possible on individual tokens.

Example values: "The quick brown fox jumps over the lazy dog"

Elasticsearch query examples:

Example 1: This term query will find the example value:

{
 "query": {
     "term": {
        "test": "fox"
     }
  }
}

Other term queries that work:

"quick"
"jumps"

Example 2: This match query will find the example value:

{
 "query": {
     "match": {
        "test": "The quick brown"
     }
  }
}

Other match queries that work:

"The quick brown fox jumps over the lazy dog"
"The QUICK BROWN"
"lazy fox"

Queries that will not work:

slow cat

Date-time

Postfix: __dt

Description: Date and date-time field. Supported date-time formats are:

yyyy-MM-dd'T'HH:mm:ss.SSSZ

yyyy-MM-dd'T'HH:mm:ssZ

yyyy-MM-dd'T'HH:mm:ss

yyyy-MM-dd HH:mm:ss.SSS Z

yyyy-MM-dd HH:mm:ss.SSS

yyyy-MM-dd HH:mm:ss Z

yyyy-MM-dd HH:mm:ss

yyyy-MM-dd

yyyy/MM/dd HH:mm:ss.SSS Z

yyyy/MM/dd HH:mm:ss.SSS

yyyy/MM/dd HH:mm:ss Z

yyyy/MM/dd HH:mm:ss

yyyy/MM/dd

HH:mm:ss.SSS

HH:mm:ss

Example values:

2018-06-24T12:34:56.543-0800

2018-06-24T12:34:56-0800

2018-06-24T12:34:56

2018-06-24 12:34:56.543 -0800

2018-06-24 12:34:56.543

2018-06-24 12:34:56 -0800

2018-06-24 12:34:56

2018-06-24

2018/06/24 12:34:56.543 -0800

2018/06/24 12:34:56.543

2018/06/24 12:34:56 -0800

2018/06/24 12:34:56

2018/06/24

12:34:56.543

12:34:56

Elasticsearch query examples:

Example: Find all stats record between January 1st 2018 and October 31st 2018:

{
 "query": {
     "range": {
        "test__dt": {
           "gte": "01-01-2018",
           "lte": "31-10-2018",
           "format": "dd-MM-yyyy"
        }
     }
  }
}

Number

Postfix: __nr

Description: Number field.

Example values: 123456789012345

Elasticsearch query examples:

Example: Find all stats records with value 123456789012345:

{
 "query": {
     "term": {
        "test__nr": 123456789012345
     }
  }
}

Decimal

Postfix: __dc

Description: Decimal field.

Example values: 12345.678

Elasticsearch query examples:

Example: Find all stats records with value greater than 50"

{
 "query": {
     "range": {
        "test__dc": {
            "gte": 50  
        }
     }
  }
}

Boolean

Postfix: __b

Description: Boolean field.

Example values:

true

false

Elasticsearch query examples:

Example: Find all stats records with value true:

{
 "query": {
     "term": {
        "test__b": true
     }
  }
}

Exact text

Postfix: __s

Description: Exact text field. The text is not analyzed and not tokenized. You can only query for the exact value that was stored.

Example values: "The quick brown fox jumps over the lazy dog"

Elasticsearch query examples:

This term or match query will find the example value:

"The quick brown fox jumps over the lazy dog"

Queries that will not work:

"The quick brown FOX jumps over the lazy dog"

"fox"

Case insensitive text

Postfix: __si

Description: Case insensitive text field. Same as exact text __s, with the addition that queries can be case insensitive.

Example values: "The quick brown fox jumps over the lazy dog"

Elasticsearch query examples:

This term query will find the example value:

"the quick brown fox jumps over the lazy dog"

This match query will find the example value:

"The quick brown FOX jumps over the lazy dog"

Queries that will not work:

" the quick brown fox jumps over the lazy dog"

"The quick brown fox"

Trimmed and case insensitive text

Postfix: __st

Description: Trimmed and case insensitive text field. Same as case insensitive __si, with the addition that the text is trimmed, ignoring any white space at the beginning and end of the text when querying.

Example values: " The quick brown fox jumps over the lazy dog"

Elasticsearch query examples:

This term query will find the example value:

"the quick brown fox jumps over the lazy dog"

These match queries will find the example value:

"The quick brown fox jumps over the lazy dog"

" The quick brown fox jumps over the lazy dog"

Queries that will not work:

"The quick brown fox"

Case insensitive and tokenized

Postfix: __sa

Description: Case insensitive and tokenized text field that divides text at numbers, punctuation and other non-alphabetic characters.

Example values: "The 25 quick brown fox3s jumped over the lazy dog"

Elasticsearch query examples:

These term queries will find the example value:

"brown"

"fox"

These match queries will find the example value:

"The 25 quick brown fox3s jumped over the lazy dog"

"fox3s"

Queries that will not work:

"25"

Case insensitive and tokenized (divided at path delimiter)

Postfix: __sh

Description: Case insensitive and tokenized text field that divides on a path delimiter: "/". Used for searching on part of a path or on path ancestors.

Example values: "/This is/a typical/path"

Elasticsearch query examples:

These term queries will find the example value:

"/this is"

"/this is/a typical"

"/this is/a typical path"

These match queries will find the example value:

"/This is/a typical/path"

"/this is/a typical/path"

Queries that will not work:

"/a typical/path"

Default but with an explicit language

Postfix: __s_xx

Description: Same as default but with an explicit language. For example __s_nl maps to Dutch and __s_ja maps to Japanese.

Supported languages:

ar - Arabic

ca - Catalan

cs - Czech

da - Danish

de - German

el - Greek (modern)

en - English

es - Spanish

fi - Finnish

fr - French

ga - Irish

hi - Hindi

hu - Hungarian

in - Indonesian

it - Italian

ko - Korean

lv - Latvian

nl - Dutch

no - Norwegian

pl - Polish

pt - Portuguese

ro - Romanian

ru - Russian

sv - Slovak

th - Thai

tr - Turkish

zh - Chinese

Field type mapping

Supported field type mappings

default (no postfix)

Date-time

Number

Decimal

Boolean

Exact text

Case insensitive text

Trimmed and case insensitive text

Case insensitive and tokenized

Case insensitive and tokenized (divided at path delimiter)

Default but with an explicit language

Related articles