3 Using AMDL Expressions

This section shows you how to use various syntax in AMDL expressions for creating rules.

AMDL has features that enable you to use advanced AMDL functionality for maintaining rules. These include:

Further Information

For more information about:

• The syntax used by AMDL, see Getting Started with AMDL Syntax.

• How to use AMDL to write rules, see: Getting Started and Writing Business Rules in AMDL.

• How to write tests to check your AMDL rules, see Unit Tests for AMDL .

• Details of the types, scopes, methods and annotations in AMDL, see the Appendices.

3.1 AMDL Expressions

AMDL includes expressions, which are the basic building blocks of Business Rules. The general form of any AMDL expression is as follows:

@annotation
scope.name: definition

An AMDL expression consists of a scope, name, definition and, optionally, one or more annotations:

• scope –— the scope of the AMDL expression, denoting the type of expression that is being defined. Some scopes allow you to define expressions, while some are only for referencing values. Appendix 3: AMDL Scopes lists all the possible scopes you can use in Business Rules. For example, you can define a scope for a state or for lists.

• name –— the name by which the definition is referenced in output and other AMDL expressions. A name must be unique within a scope, but can be shared by AMDL expressions of different scopes (for instance, state.foo and values.foo could both distinctly exist).

• definition –— the value to which the AMDL expression evaluates if run.

• annotation –— additional information about the AMDL expression that can influence its evaluation behaviour, which are optional. AMDL expression can have no, one, or several annotations. For more details. refer to Using Annotations and Appendix 4: AMDL Annotations (which lists all the annotations that can be used in AMDL).

3.2 Comments

You can write comments in AMDL using //... and /*...*/. These are ignored by the parser. // ensures that everything remaining on that line is interpreted as a comment. /* ensures that everything is treated as a comment until a closing */ is found.

// This is a single-line comment
rules.thisIsAMDL: event.amount.baseValue > 100 // This is a comment too
/* This is a multi-line comment.
This is still part of the comment.
So is this. */

3.3 Conditions

In AMDL, a condition is a Boolean expression that evaluates to either true or false.

You can use conditions in various types of AMDL expressions. A condition consists of an operator and one or more operands. For a full list of AMDL operators, see AMDL Operators.

An example of a simple condition is the following, which is true if the value in the "country" field in the event is "GB":

event.country == "GB"

Conditions are also numerical comparisons, such as:

event.amount.baseValue > 100

In place of a condition, you can use a Boolean variable, or something that returns Boolean value. For example, using the example event from section Event Data, you can write this condition to check if a transaction was accepted:

event.accepted == true

Or you can write:

event.accepted

You can link conditions using the Boolean operators AND, OR and NOT. In AMDL, these are represented by && (AND), || (OR), and ! (NOT).

For example, this condition is true if the transaction event took place in Cambridge, UK (the country is "GB" and the city is "Cambridge"):

event.country == "GB" && event.city == "Cambridge"

The following condition is true if the event took place in Cambridge, or in the UK (or both):

event.country == "GB" || event.city == "Cambridge"

This condition is only true if the event neither took place in the UK, nor in Cambridge:

!( event.country == "GB" || event.city == "Cambridge" )

Note the use of parentheses to ensure the comparisons are executed in the correct order. The parameters ensure that the combination of conditions using OR is evaluated first, followed by the NOT. Parentheses are needed when you combine different Boolean operators, such as AND and OR, to ensure the logic is evaluated correctly. For example, this condition is true if the city is Cambridge and the country is GB or the US:

event.city == "Cambridge" &&
( event.country == "GB" || event.country == "US" )

However, this condition is true if the city is Cambridge, UK (city is Cambridge, country is GB). It is also true if the country is the US (if the country is the US the city has no effect):

(
event.city == "Cambridge" && event.country == "GB" ) ||
event.country == "US"

You write rules using conditions. The condition enables a rule to evaluate to true or false. If evaluated to true, a rule trigger actions such as alerts and tags (see Creating a Rule).

3.4 Using Annotations

There are a variety of annotations that you can use in AMDL to change when an expression is evaluated, or the effects of evaluating an expression Appendix 4: AMDL Annotations includes a complete list of all the annotations that you can use in AMDL. You can use annotations in the following ways:

@rollingAverage(12h)

3.5 Static Values

You can use the AMDL "values" scope to define static values, which are constants that do not change unless a user edits the expression. For example, you can define the following threshold value for use in other expressions:

values.threshold: 50

These values can then be referenced in other expressions using the standard reference syntax of values.<nameOfExpression>. Typically, you use the standard reference syntax to prevent repetition of a value across multiple AMDL definitions, which allows easier updates. For instance, you can have a list of countries that are shared by several rules, where storing these in a values definition allows easy modification.

3.6 Transient Variables

Transient variables are AMDL expressions which are re-calculated for each event, and not stored for future events. You declare these variables in a similar way to entity state, but with the scope "var". For example, you could calculate the current exchange rate used in each event by finding the ratio of the value in the original currency to the value in the base currency:

var.fxRate: event.amount.value / event.amount.baseValue

Other AMDL expressions, in the "'var" scope and other scopes, can reference the results of these calculations using the standard reference syntax of var.<nameOfVariable>. This appears in the following example:

var.thresholdInOriginalCurrency: values.threshold * var.fxRate

In Business Rules, transient variables can help tidy a set of expressions which all contain similar expressions with slight variations, or which contain references to the result of a common calculation. For example, many expressions might refer to the exchange rate defined above as var.fxRate.

3.7 Conditional Assignment

For many use cases, you need to be able to conditionally assign a value or update a state expression. For example, you might want to store the value of the most recent accepted transaction, the last unsuccessful transaction, or the merchant of the most recent transaction of over $1000 in value.

AMDL uses the conditional operator, ?, for conditional assignment (see Ternary operator for other uses). This operator acts like an "if/then" statement in other languages, providing a way to evaluate some expression only if a certain condition is true. The ? operator must be preceded by a Boolean expression, or a condition (see Conditions) or some other expression that evaluates to true or false. The operator is followed by the expression or value to evaluate if the condition is true.

The following examples is for creating a state expression that stores the value of the most recent card not present (CNP) transaction:

state.lastCNPTransactionValue:
event.transactionType == "CNP" ? event.amount.baseValue

Note that this uses the Business Rules syntax for state expressions (see Adding State Information to a Rule).

This state variable is only updated for events where the transactionType field is "CNP". For other events, the condition before the ? operator evaluates to false (or evaluation stops altogether if the transactionType field is missing from that event). When this happens, the expression stops being evaluated and any value currently in the state variable is therefore not overwritten.

When using the conditional operator, you can also supply an expression to evaluate if the condition is false. This enables you to implement an "if/then/else" logic, storing one value if the condition is true and other if it false. If the evaluation of the initial condition stops, the expression will not be updated at all.

For example, for a deposit and withdrawal event types, you might want to calculate a signed version of the event value, which is - for a withdrawal and + for a deposit. Signed values for an event make it easier to keep track of a customer's balance. You can enter the following code using a transient variable:

var.signedAmount: event.eventType == "deposit" ?
event.amount.baseValue :
-1 * event.amount.baseValue

If the event type is "deposit", this variable takes its value from event.amount.baseValue. Otherwise (if the initial condition is false), it takes the value of this field multiplied by -1.

3.7.1 Switch Case Operator

In a situation where a variable can take multiple values, you can create an expression that evaluates differently depending on the value. You could do this by chaining multiple conditional statements together (see Conditional Assignment). However, you can create more readable expressions using the AMDL switch case operator in this example:

@eventType
(deposit)
state.lastAmount:
event.country ~?
"GBR": event.currencyGBP;
"IRL": event.currencyEUR;
default: event.currencyUSD;

The ~? operator initiates the switch on the value of the country field. Each case is terminated by a semicolon. The values switched on must be either a fixed value or default.

The default value is optional, where including the default keyword means that if the rule engine gets to the end of a switch case execution without detecting a match, it uses the default case. If no default is included and the rules engine does not find a matching case, AMDL execution of the expression stops.

3.7.2 Switch Case Operator Usage

The ~? operator allows one of many operations to be evaluated based upon a switch expression. This is useful if you have several special cases for the same variable, each of which needs different treatment.

Example 1:

Here, you check the event amount against a different threshold based on the mcc field.

event.mcc ~? "7995": event.amount.baseValue > 150;
"5912": event.amount.baseValue > 200;
"4722": event.amount.baseValue > 5000;
default: event.amount.baseValue > 500;

Example 2:

Here, the 'default' case is optional. If the expression does not evaluate to one of the case labels, the expression evaluation stops.

event.mcc ~? "7995": event.amount.baseValue > 150;
"5912": event.amount.baseValue > 200;
"4722": event.amount.baseValue > 5000;

Note that case labels must be a fixed value or 'default'. Currently, switching based upon variable expressions is not supported. Also note that every switch case must be terminated by a semicolon.

3.8 Dates, Times and Durations

Date-times and durations in AMDL make it simple to compare times and base AMDL logic on the elapsed time between two events. Date-times in system events, and in AMDL, must be written in ISO-8601 format (http://www.w3.org/TR/NOTE-datetime), such as "2020-02-01T12:34:56Z", and must have a time zone designator. The designator can be:

• Z for UTC

• an offset in the format + (for time zones head of UTC)

• followed by hh for an integer number of hours (for time zones behind UTC)

• an offset in hours and minutes expressed as hhmm orhh:mm.

You can enter durations, which represent elapsed time, by appending a unit to an integer. For example, entering 7d represents 7 days. The following table shows the duration units that you can use in AMDL.

In AMDL, you can subtract one date-time from another to give a duration. You can also add durations to and subtract durations from a date-time to get another date-time. For example, this expression gives the date-time three hours after the date-time of the current event:

event.eventTime + 3h

This expression gives the date-time 30 minutes before the current event:

event.eventTime - 30m

The expression below shows the use of date-times and durations in a condition. The condition is true if less than 60 days have elapsed between the current event and the time the customer registered for their account. The first part of the expression evaluates to a duration representing the amount of time elapsed between the eventTime and the accountOpenDateTime; this is then compared to a fixed duration of 60 days (60d).

event.eventTime - event.accountOpenDateTime < 60d

Note the order of the two fields. If the account open date is known to be before the time of the current event (which seems likely), this order results in a positive duration. Inverting the order results in a negative duration, which is always less than 60 days (resulting in incorrect logic).

3.9 Collections

In AMDL, collections are data structures that contain multiple values. The simplest type of collection is an array. An array consists of multiple individual elements, which may be of any type. For example, this expression defines an array of strings:

values.dwarfs:
[ "Sleepy", "Dopey", "Happy", "Grumpy", "Sneezy", "Bashful", "Doc" ]

You can check whether a collection contains a given value using the "contains" operator, ~#. For example, given the collection above, this condition is true:

values.dwarfs ~# "Doc"

AMDL also has a "does not contain" operator, !# where, for example, the following condition is true:

values.dwarfs !# "Gandalf"

You can define arrays in-line as part of your AMDL expression. For example, the following is true if the value of the field merchantCategoryCodeis listed as follows:

[ "5122", "5912", "5993", "7841", "7995" ] ~# event.merchantCategoryCode

You can also use annotations to transform AMDL expressions, such as state and transient variable definitions, into collections. By default, these expressions store a single value, and when the expression is evaluated the stored value is updated and overwritten with the latest value. Collections stored in state allow you to store, for example, the values of a merchant's 10 most recent transactions, or the unique IDs of all the mobile devices used to access an account.

3.9.1 Arrays

Annotations let you change the way that data is stored in state and global expressions. The @array annotation let you turn a state or global expression from one that stores a single scalar value into one that stores multiple values. You must provide an argument to the @array annotation, which can be either a number or a duration. If the argument is a number it specifies the maximum number of values to store in the collection. If the argument is a duration, values are retained for that duration in the collection. Values that have been in the collection for longer than the specified duration are deleted when that expression is referenced or updated.

For example, this rules expression, defined for the 'consumer' entity type, stores the values of the last 10 transactions for each customer entity:

@array(10)
@eventType("transaction")
state.last10TransactionValues: event.amount.baseValue

This rules expression stores the values of all the transactions seen for each entity over the last 24 hours:

@array(24h)
@eventType("transaction")
state.transactionValuesLast24h: event.amount.baseValue

3.9.2 Sets

Whilst an array is an ordered collection of any values, a set is an unordered collection of unique values; for example, the collection of all unique IP addresses from which an account has been logged in to. Sets are declared in exactly the same manner as arrays, except using the @set annotation.

Sets have varied use cases. For example, you can use sets to:

• Store the different payment methods of a customer over the last 30 days.

• Store the last 10 mobile device IDs seen logging into a particular account.

• Store the IP addresses a device has logged in from in the course of an hour.

The following example shows the different payment methods used by a customer over the last month.

@set(30d)
state.methodIdsLastMonth: event.methodId

3.9.3 Using Collections

A single value can be checked against the elements of these collections using the ~# and !# operators, or other collection operators. This allows you to check whether a value is contained in an array or set, or check all the elements in a collection against a single value. For example, you can check if all the elements in the array are greater than 100, or if all the elements in the array are less than the value of the current transaction.

You can use the collection methods described in Appendix 5: AMDL Methods to extract data from a collection. For example, there are methods for calculating the number of elements in a collection, the total value, the mean, median and mode, and other statistical measures. Methods also exist for sorting collections alphabetically or numerically, or combining collections.

There are also more complex types of collection, including histograms (see Using Histograms) and maps (see Using Lookup Tables).

3.9.4 Collection membership operators usage

These operators are used to test membership of a collection, and return a Boolean value. In usage, the left-hand operand must always be some form of collection, and the right-hand operand the element of the collection whose membership is being tested.

Example 1:

Checking that a list in the event contains the value 20.00.

event.transactionAmounts ~# 20

Example 2:

Checking whether a string in the event is contained within a pre-defined collection of values.

values.declineStatusCodes ~# event.responseMessage

Example 3:

Checking that the country field in the event is not one of "GB", "US" or "IS".

{ "GB", "US", "IS" } !# event.country

3.9.5 Collection comparison operators usage

Operators: ==# !=# <# <=# ># >=#

These operators perform comparison operations upon all elements of a collection and return the Boolean combination of the results. The left operand must always be a collection and the right operand that which is to be compared.

Example 1:

All elements of the ordered collection are equal to 1, so this is true.

[ 1, 1, 1, 1, 1 ] ==# 1

Example 2:

None of the elements in the collection are equal to "strawberry", so this is true.

{ "apple", "pear", "banana" } !=# "strawberry"

Example 3:

All values are less than or equal to some pre-defined threshold.

event.transactionAmounts <=# values.threshold

Example 4

> operator for comparing datetimes, ensuring that ># checks that all datetimes exceed the right operand.

state.transactionTimes ># event.accountOpenDateTime

3.10 Manipulating Strings

In AMDL, you can manipulate strings using:

• String concatenation

• Regular expressions

3.10.1 String Concatenation

You can concatenate strings using the .. operator. For instance, to store a full name as state, you could write:

@eventType(registration)
state.fullName: event.firstname .. " " .. event.surname

String Concatenation Usage

Operator: ..

This operator concatenates any two strings, numbers, durations, or times as strings. For non-strings this is done in such a format that they can be coerced back to their original types.

Example 1:

Evaluates to "Hello World".

"Hello " .. "World"

Example 2:

Produces a full name from component parts.

event.firstName .. " " .. event.lastName

3.10.2 Regular Expressions

By default, regular expression matching and replacement are disabled in Solution and Thredd-level AMDL. This means that you cannot use the regular expression operators described below in Solution or Program Manager level Business Rules.

AMDL supports two regex operators: the regex match operator ~=, and the regex substitution operator ~:

The regex match operator ~= takes a match regex as its right-hand argument. This is a string enclosed by forward slashes (for example, "/.*/"). The operator results in a Boolean value true if the regex matches a substring of the left-hand argument. Full-text matches can be implemented by adding ^ to the beginning and $ to the end of the match regex, in order to match the beginning and end of the string. For example, you may wish to detect whether an email address field ends in "protonmail.com". This could result in a rule such as:

rules.protonmailDomain: event.email ~= "/protonmail\.com$/"

Use of the Backslash

You can use the backslash to escape the period. Internally, the system evaluates AMDL using Java. This means the allowed regex syntax is identical to Java's Pattern syntax, except that you escape forward slashes with a backslash. For example, to match "/", you use the following regular expression: "/\//".

Regex Substitution

The regex substitution operator takes a substitution regex as its right-hand argument. This is two strings, a match regex and a replacement string, enclosed and separated by forward slashes. For example, "/\s/_/" replaces space characters by an underscore. The result of the operator is a copy of the left-hand argument, except with each substring that matches the match regex replaced by the replacement string. For example, "Hello world!" ~: "/l/LL/" returns the string "HeLLLLo worLLd!". You can refer to matching groups in the matching regex using $ followed by the match group number (for example $1 for the first match group). To replace each letter in a string with the same letter followed by an asterisk for example, you can use the regex "/(.)/$1*/" - "Hello world!" ~: "/(.)/$1*/" returns "H*e*l*l*o* *w*o*r*l*d*!*".

For example, we can write a rule that removes any initial titles (e.g. "Mr", "Ms", "Mx" or "Prof.") from the cardholder name in the event, and compares the remaining name to the stored customer name:

rules.cardholderNameWithoutTitle_DoesNotMatchStoredName:
( event.cardholderName ~: "/^(([DdMm][RrXx]?[Ss]?|Prof)\.?\s+)+//" ) !=
state.customerName

Regular Expression Match Operator Usage

Operator: ~=

This operator checks whether the left string operand matches the regular expression contained in the right operand. Note that you can only apply the operator to strings. For more details on regular expressions, see Outputting Values.

Example 1:

Checking whether the postcode starts with "CB".

event.address.postcode ~= "/^CB/"

Regular Expression Substitution Operator Usage

Operator: ~:

This operator performs a substitution specified in the right operand on the left string operand. Note that you can only apply the operator to strings. For more details on regular expressions, see Outputting Values.

Example 1:

This expression substitutes dots with dashes, resulting in "1970-01-01". Note that it does not matter whether regexes are quoted or unquoted.

"1970.01.01" ~: "/-/./"

3.11 Specifying a Default Value

If a reference within an AMDL expression cannot be found (for example, an event field which is not present, or a state which hasn't been updated yet), it returns a null value. If the null value is not handled in the logic of the expression, the AMDL execution of the expression stops and the value of the expression being evaluated also becomes null.

For scenarios where the expression is a rule, the rule does not trigger. If the expression is a variable update expression such as a state, global or transient variable, that variable is not updated. Although this is the desired behaviour, you need to be careful you handle such possibilities when needed.Note that executing AMDL does not short-circuit Boolean operators and evaluates all references. The evaluation of expressions stops if any of these references evaluates to null.

For example, even if the field accepted has a value in the processed event and var.acceptedTransaction is undefined, evaluation of this rule ceases. As a result, the rule does not trigger:

rules.accepted: event.accepted == true || var.acceptedTransaction == true

The 'default value' operator ?? automatically replaces a preceding reference with the value following it, if that reference is found to be null during execution. For example, to ensure that the rule above executes even if the accepted field is not present:

rules.accepted:
( event.accepted ?? false ) == true || var.acceptedTransaction == true

This ensures that the evaluation of the field accepted always returns a value. If the field is missing, the default value false is returned. Note that the parentheses ensure that the components of the expression are evaluated in the correct order (see Operator Precedence for more information).

3.11.1 Usage

Operator: ??

In AMDL, expression evaluation stops if a referenced variable does not exist. The ?? operator lets you specify a default value if an expression cannot be evaluated.

Example 1:

If the event does not contain a field called amount.baseValue, this evaluates to 0 instead. This expression is particularly useful when dealing with optional fields that may or may not be present in the event.

( event.amount.baseValue ?? 0 ) > 100

Example 2:

You can default to the value of another field, or a state, or other variable as well as a fixed value.

( event.chargebackTime ?? event.eventTime ) < state.prevTime + 90d

3.12 Evaluating if an Expression Returns a Non-Null Value

You can use the 'exists' operator ~ to query whether the result of an expression exists or not. This allows you to create conditions which are true if an optional event field exists within the specific event being processed, or if a specific state variable, transient variable or values expression has a value. For instance, you can check whether a field exists in the event by using the following syntax:

rules.fieldExists: ~event.field

You can construct an inverse using the "does not exist" operator. The inverse is a condition which is true if a field does not exist. This is a combination of the 'exists' operator and the 'NOT' Boolean operator.

3.12.1 Usage

Operator: ~

This operator checks whether evaluation of a reference returns a non-null value.

Example 1:

Evaluates to true if optionalField is present in the event being processed.

~event.optionalField

Example 2:

Evaluates to true if previousTransactionValue is defined in state for the entity being processed.

~state.previousTransactionValue

Due to the wide-varieties of regions in which Threddcustomers operate in, there may be unreliability in some of the above data.

3.13 Storing the First Value for an Expression

You can capture a state once without it being updated again, for instance, the first login date of a user. You can use the 'exists' operator and the ternary operator. However, because it is a relatively common occurrence a special annotation is provided: @firstValue.

When applied to an AMDL expression, once the expression evaluates to a value, it maintains that value and is not updated any more.

@eventType(login)
@firstValue
state.firstLoginTime: event.eventTime

3.14 Using Histograms

Arrays and sets store all values they are updated with for a duration. When you expect to use a large number of values for updating the collection in the period of the duration, this could produce excessive storage requirements and adversely affect the engine performance. For instance, globally storing the value of all transactions over a month could result in a very large collection.

3.14.1 Histogram Buckets for Time-Series Analysis

To facilitate analysis of time-series data, AMDL provides a histogram type. A histogram consists of a group of buckets, each of which represents a group of updated values that occurred during a specific time window. A histogram is granular on the level of the buckets, not the values. Therefore, a histogram can be a powerful approximation to expedite performance.

You can provide any state-like AMDL expression in a histogram annotation which has the form @histogram(historyLength=<duration>, bucketSize=<bucket duration>). The duration indicates the period for which the histogram should store buckets before expiring them. The second argument determines the number of buckets in the histogram and is optional. If left unspecified, a default is chosen to ensure at least 10 buckets are used. More buckets mean more accurate summary figures but a greater storage/performance impact. Bucket size is limited to a series of acceptable durations. If not one of the acceptable bucket sizes (see below), the nearest acceptable bucket size is used. This is ensures the histogram uses the expected number of buckets.

Histogram Bucket Data

Acceptable bucket sizes for histograms include:

• Monthly buckets: 12 months, 6 months, 4 months, 3 months, 2 months, 1 month

• Fixed-length buckets: 7 days, 1 day, 12 hours, 6 hours, 3 hours, 2 hours, 1 hour, 30 mins, 20 mins, 15 mins, 10 mins, 5 mins, 1 min, 30 seconds, 5 seconds, 1 second.

Bucket durations are expressed in months (e.g. 6 months, 12 months). The durations use ‘M’ as their unit (for example, 6 months is represented as ‘6M’).

Histogram buckets align with calendar time periods. For example, monthly buckets align with calendar months, so the first relevant event that occurs after midnight (UTC) on the first day of the month contributes the first datapoint to that month's bucket. Buckets with a shorter duration (expressed in days, hours, minutes or seconds) align with calendar days. All non-monthly histogram buckets align at midnight (UTC) each Monday, so buckets with a duration of 7 days start at midnight on Monday; buckets with a duration of 1 day start at midnight each day; buckets with a duration of 6 hours start at midnight, 6am, noon, and 6pm each day.

Like arrays, you can call the size, total and mean methods on a histogram state.

3.14.2 Expressions and Rules

You can define a global histogram that stores transactions over the last week as shown in the following example:

@eventType
(transaction)
@histogram (historyLength=7d, bucketsize=1d)
globals.txAmounts: event.amount

You can then write a rule which triggers if an entity performs a transaction of an amount greater than 1000, but normalised for the day's global average transaction amount in relation to the week's.

@eventType
(transaction)
rules.highTx:
event.amount *
( globals.txAmounts.mean(7d) / globals.txAmounts.mean(1d) ) > 1000

Note that you add a duration argument to the mean method. This limits the values of the histogram to those within that passed period, so far as the granularity of the bins allow. In the example above, the mean(1d) method returns the mean of values from today. The mean(7d) method returns the mean of values from today, and the previous 6 days.

Monthly durations (such as 1 month, 6 months) use calendar months, which can be of different durations. To avoid issues when comparing histogram data between months of different lengths, normalised versions of these methods are also provided (normalisedSize, normalisedTotal, normalisedMean). These methods normalise the total, size or mean to a month length of 30 days.

Used in this way, these methods can access data from the most recent bucket, or from that bucket and previous buckets. However, for comparison purposes or for defining a baseline, you can refer to previous buckets. You can provide a date-time as an argument to the size(), mean() or total() method which allows you to access the bucket that contains that date-time. This date-time can be derived from event data through subtraction of a duration, to return data from the bucket from a specified duration in the past.

For example, the expressions below enables you to compare the total value of a customer's transactions this week with the total value last week.

@eventType
(transaction)@histogram(bucketSize=7d, historyLength=28d)
state.TransactionValueWeekly: event.amount

@alert
@eventType
(transaction)
rules.txnValueThisWeekGtr3timesLastWeek:
event.amount + state.TransactionValueWeekly.total(7d) >
3 * state.TransactionValueWeekly.total( event.eventTime - 7d )

This example rule triggers an alert if the total value of a customer's transactions so far this week is greater than three times the total value of the customer's transactions last week. Data is returned for last week's bucket instead of for the latest bucket by passing the date-time exactly 7 days ago. This is in the format of event.eventTime - 7d as the argument to the total() method.

However, this approach only allows you to access data from a single bucket. To access data from multiple previous buckets, you can use the atTime() method. This method only applies to histograms, and takes a date-time as an argument. This date-time can be derived by subtracting a duration from a date-time.

Example of an Alert

You can modify the rule in the previous example to generate an alert. In this example, the total value of a customer's transactions this week is greater than the total value of a customer's transactions over the previous 2 weeks.

@alert@eventType(transaction)
rules.txnValueThisWeekGtrTotal2PreviousWeeks:
event.amount + state.TransactionValueWeekly.total(7d) >
state.TransactionValueWeekly.atTime( event.eventTime - 7d).total(14d)

To access data in this example, you use the atTime() method with an argument 7 days in the past in combination with total(14d). The total value is in two buckets that does not include the current one. It includes the bucket with the date-time supplied as an argument (7 days ago), and the previous bucket. This is because the bucket size is 7 days and the duration supplied as an argument to the total() method is twice the duration.

Time Field

By default, histograms use the event data field eventTime to determine the time at which the current event occurs. This therefore determines which bucket to update and/or which bucket(s) to access.

The @histogram annotation can take an optional argument, timeField, which allows you to define a non-default time field to use. This can be an event data field or a transient variable, specified as a string. For example, for an event data field called "realTime", you write timeField="event.realTime"). The eventTime might not signify the time that the original transaction took place, particularly with batch processed event, so you might want to use another field as the time field:

@histogram(bucketSize = 1d, historyLength = 7d, timeField =
"event.realTime")
state.transactionValues: event.amount.baseValue

Alternatively, you can use a derived time calculated in a transient variable. For example, if the transient variable "newTime" contains the date-time in the histogram time field:

@histogram(bucketSize = 1d, historyLength = 7d, timeField =
"var.newTime)
state.transactionValues: event.amount.baseValue

3.14.3 Testing Histograms

To test an expression that refers to a histogram, you must specify a simulated version of the referred histogram. For example, the following rule displays with the initial state form a unit test.

@alert
@eventType(transaction)
rules.highTotalTransactionValueToday:
event.amount + state.txnValueDaily.total(1d) > 5000

@histogram
(bucketSize = 1d, historyLength = 3d)
state.txnValueDaily: {
"data": {
"2019-03-31": { "size":17, "total":5325 },
"2019-04-01": { "size":10, "total":3575 },
"2019-04-02": { "size":14, "total":4500 },
}
}

The state definition in the 'Initial State' test panel must begin with a copy of the histogram annotation from the state being referred to. The histogram is defined as a JSON map with a single key, "data", within which is a series of JSON objects representing the buckets in the histogram. Each bucket has a key, which is the start date(-time) of that bucket, and the value is a JSON object containing two properties, "size" and "total".

3.15 Using Lookup Tables

A map is a lookup table where there is a map between two values that return a resultant value. This enables the storage of a key-value state. For instance, in a transactional system every entity may have several different payment methods, and you may want to keep track of the number of transactions made using each method. You can use each method identifier as the key, and the count of the number of transactions as the value (i.e. what each key is mapped to).

The data in this map can be represented as follows:

transactionCountByPaymentMethod: {
"method1": 10,
"method2": 3
}

For "method1" as the key, transactionCountByPaymentMethod returns the value 10.

You can use AMDL to define both static maps (using the "values" scope) and dynamic maps for updating and modification with each incoming event, stored in an entity or a global state.

3.15.1 Lookup Tables in Static Maps

Static maps store lookup tables such as a currency conversion table, or a dictionary of country codes mapped to country names. For example, you can store a list of threshold values for different merchant categories, and raise an alert when the value of a transaction exceeds the threshold for the appropriate Merchant Category Code (MCC). You can implement static maps using nested conditional statements or for better readability, a switch case statement (see Switch Case Operator). However, using a static map can simplify this kind of expression by removing the thresholds into a different expression from the business logic of the rule itself. You can define a map like this:

values.MCCSpecificThresholds: {
"7999": 300,
"7995": 1000,
"5912": 200,
"5411": 450,
"5311": 750
}

A rule can then reference this map using the syntax <scope>.<map expression>[<key>] for example, values.MCCSpecificThresholds["7999"] returns 300.

For example, this rule generates an alert if the value of a transaction exceeds the threshold for the relevant MCC:

@alert
@eventType("transaction")
rules.valueOverMCCThreshold:
event.amount.baseValue >
values.MCCSpecificThresholds[ event.merchantCategoryCode ]

Using Switch Case Statements

You can write the above expression for lookup tables using a switch case statement (see Switch Case Operator), which allows you to provide a default value. A map does not provide a way of specifying a default, but you can use the default value operator (see Specifying a Default Value). The default value is returned if the referenced key is not present in the map.

This example specifies a default threshold of 500:

@alert
@eventType("transaction")
rules.valueOverMCCThreshold:
event.amount.baseValue >
(values.MCCSpecificThresholds[ event.merchantCategoryCode] ?? 500 )

3.15.2 Storing Maps

You can store maps dynamically through state or global expressions. For example, you can store the timestamp of the last time a customer has used the transaction method of the current transaction. The transaction could be from a specific payment card or tokenised payments such as Apple Pay or Google Pay. If each payment method has a unique ID in the event field event.paymentMethod.methodId, you could define a state expression that stores the timestamp of the last transaction for each unique method ID:

state.lastTimeMethodSeen[ event.paymentMethod.methodId ]: event.eventTime

Each time you receive a transaction with a particular method ID, the timestamp is stored as the value, with that method ID as the key. After receiving the following transactions:

The map stored in state looks as follows:

state.lastTimeMethodSeen: {
"method1": "2019-12-01T10:01:24Z",
"method2": "2019-12-05T08:17:54Z",
"method3": "2019-12-10T17:26:12Z"
}

If you received another transaction with the method ID "method2", at 15:26:41 on 11th Dec 2019, it overwrites the existing value for that key, and the resulting map looks as follows:

state.lastTimeMethodSeen: {
"method1": "2019-12-01T10:01:24Z",
"method2": "2019-12-11T15:26:41Z",
"method3": "2019-12-10T17:26:12Z"
}

You can then write a rule or other expression referencing this state. For example, you can reference the time a particular payment method was last seen, or check whether a payment method has been seen within a certain period of time. The following transient variable stores true if the payment method in the current event has not been seen in the last 180 days:

var.paymentMethodNotSeenInLast180days:
event.eventTime - state.lastTimeMethodSeen[event.paymentMethod.methodId] >
180d

You can also write an expression that updates values for multiple map keys for a single event, using syntax like the following:

<scope>.<expression name>[<key 1>]: <value 1>;
[<key 2>]: <value 2>;
...
[<key N>]: <value N>

The key values can be static or refer to event data fields or other AMDL expressions. For example, consider a transaction event that represents an e-commerce order. The event contains a shipping address and a billing address, uniquely identified by the shippingAddress.addressId and the billingAddress.addressId respectively.

You can store the last used billing and shipping address for each customer as follows:

state.lastAddressUsed[ "shipping" ]:
event.shippingAddress.addressId;
[ "billing" ]: event.billingAddress.addressId

You can then refer to the most recently used shipping address using state.lastAddressUsed["shipping"], or the most recently used billing address using state.lastAddressUsed["billing"].

For an example of using event data fields as keys, you can use the same transaction event that contains a billing and a shipping address. You could write an expression to store the date and time that each address was last used in an event as follows:

state.addressSeen[ event.shippingAddress.addressId ]: event.eventTime;
[ event.billingAddress.addressId ]: event.eventTime

By using the @array or @set annotation when defining a dynamic map in a state or global expression, you can create a nested dynamic map for storing a separate collection of elements against each key in the map:

@eventType("transaction")
@array(7d)
state.merchantTransactions7d[event.merchantId]: event.amount.baseValue

You can then create a rule that looks up a key in the map and uses collection methods to extract data from the corresponding collection:

@eventType("transaction")
rule.excessivePaymentsSingleMerchant:
state.merchantTransactions7d[event.merchantId].size(1d) > 6 &&
state.merchantTransactions7d[event.merchantId].total(1d) +
event.amount.baseValue > 40000

For more information about using arrays and sets, see Collections.

To avoid performance issues, the default limit for the number of keys stored in each map is 1000. Once a map reaches this limit, each time a new key is added, the oldest key is removed. However even with this limit, performance issues may occur, particularly in the case of nested maps that store a collection of elements with each key. When creating a dynamic map, you should consider the number of unique keys for storage. Note that in a nested map, keys are not removed if the corresponding collection becomes empty. When necessary, use the @mapOptions annotation to set an appropriate limit on the number of stored keys. See below for more information.

3.15.3 Specifying a Key Limit

When defining a dynamic map using a state or global expression, you can use the @mapOptions annotation and the keyDuration or keySize arguments to specify a limit on the number of keys stored in the map.

@eventType("transaction")
@mapOptions(keyDuration=14d, keySize=500)
state.merchantTransactions7d[event.merchantId]: event.amount.baseValue

You can use the keyDuration argument to set a duration limit. Using this argument removes any keys that were last updated before this duration. You can use the keySize argument to set a size limit. Once the number of stored keys reaches the size limit, each time a new key is added, the key that was last updated longest ago is removed. If you use both arguments, keys are removed according to whichever limit is reached first.

3.16 Using Data Lists in AMDL

In AMDL, you can access data in a data list defined at the same level of the expression using the lists scopes, using the 'contains' (~#) and 'does not contain' (!#) operators for collections. You can also update a data list. To create a data list, you need to use the Settings menu in the Fraud Transaction Monitoring System. For more information, see Create a Data List in the Thredd Fraud Transaction Monitoring Portal Guide). Data lists are displayed as a table, consisting of a mandatory unique ID column (_id) and one or more other optional columns.

3.16.1 Accessing a Data List

In AMDL, you can access data in a data list defined at the same level of the expression using the lists scopes, using the 'contains' (~#) and 'does not contain' (!#) operators for collections. For example, you could write a rule at the Solution Multiple product Solutions may be configured in your portal deployment. Each Solution provides a combination of UI configurations, data enrichment and analytics for detecting a specific type of risk. For example, you may have a Solution for application fraud and another for inbound/outbound payments, subject to your programs set up with Thredd and Featurespace. The same event may trigger separate alerts in different Solutions. level that checks whether a particular merchant ID is in the unique ID column of a data list with the name highRiskMerchantList defined for the same Solution:

@alert
@eventType(transaction)
rules.merchantOnHighRiskList: lists.highRiskMerchantList ~#
event.merchantId

Note that this rule refers to a data list that has already been created in the Data Lists section of the Thredd. Fraud Transaction Monitoring portal. As shown in the following error message, you cannot save a rule that contains a reference to a data list that has not yet been created.

Figure 1: Validation error message

If your data list contains multiple columns, you can access the value in another column using AMDL map syntax. Using the unique ID as the key, you can select a row from the data list, and access columns as nested sub-maps using the column heading as the key.

For example, if a negative list of customers (called 'dataList') contains the column "mobileDeviceId":

rules.dataListCheck:
lists.dataList[event.consumerId]["mobileDeviceId"] == event.deviceId

3.16.2 Data List Size Limits

The following are the size limits on data lists:

Updating Data Lists Using AMDL

You can update data lists in the same way as for state and global variables, using a similar syntax and the lists scope. To add a unique ID to a data list, the syntax is the same as adding a value to a collection in a state or global expression, but using the lists scope. In this example, you can add a merchant ID to the high-risk merchant list if that merchant is included in a confirmed fraud report:

@eventType("fraud")
lists.highRiskMerchants: event.merchantId

Note that if this data list does not already exist, it will be created when this expression is evaluated for the first time.

If your data list has multiple columns, you can update the values in those columns as well, using the AMDL multiple values map update syntax. For example, to update a data list using the event data fields deviceId and ipAddress (adding these as columns headed "device" and "ip"):

lists.customerDevices[event.customerId]["device"]: event.deviceId;
["ip"]: event.ipAddress

As with updating a map, this update expression creates a new row in the data list, with the consumer ID from the event data as the value of the '_id' column, or overwrites such a row if it already exists. Note that the data list does not update if any of the update expressions fail to evaluate.

Testing Data Lists

The AMDL unit test functionality lets you test rules that refer to data lists. However, as unit tests do not have access to real data list data, you must create a simulated data list. This is in the same way where you create a simulated state when testing a rule that refers to a state or global expression.

To test a rule that refers to a single-column data list (one that only contains unique IDs), you can simulate the data list in the same way that an array (unordered collection) of strings is defined. For example, to test high-risk merchant ID rule (above), you can simulate a data list of merchant IDs by writing the following in the 'Initial State' test panel:

lists.highRiskMerchants: [
"M1056101",
"M3651540",
"M1120129",
"M9912832",
]

To test a rule that refers to a multi-column data list, you can simulate the list as an array of strings, where each string is a row of the data list. You simulate the rows using the following format: "_ id|column1Name:column1value|column2Name:column2Value…", where the string begins with the unique identifier for that row, followed by pairs of column name: value, separated by the pipe character (|).

For example, to test the multi-column data list (above), you can write the following in the 'Initial State' text panel:

lists.dataList: [
"1056101|mobileDeviceId:A01|ip:12.5.7.89",
"3651540|mobileDeviceId:D02|ip:11.5.7.89",
"1120129|mobileDeviceId:F11|ip:10.5.7.89",
"9912832|mobileDeviceId:Z76|ip:99.5.7.89",
]

3.17 Filtering Collections

You can filter collections using a predicate expression which evaluates to true or false. This returns a collection of all elements which meet the predicate, where it acts as a filter. The syntax for filtering collections is <scope>.<expression name/reference>[ <predicate> ]. The predicate expression uses $ to signify 'for each element in the collection'. For instance, consider a collection that contains the values of each transaction a particular customer has made within the last 24 hours:

state.transactionValues: [ 101, 99.99, 125, 45.99, 37.50, 48.96, 20, 10 ]

Using a predicate filter, you can write a rule to check how many transactions the customer has made with a value in excess of 100, containing the following condition:

state.transactionValues[ $ > 100 ].size() > 0

This condition is true if there are any elements in state.transactionValues with a value greater than 100 return a collection containing 2 elements, [ 101, 125 ]. The size of this collection is 2, therefore the condition is true.

The highlighted part of the expression is the predicate. The $ character means 'for each element' in the collection, and the predicate condition evaluates for each value in the array.

You can access properties of the values within the predicate by an extension of the $ syntax. For example, [$.currency == "GBP"] gets all objects in the collection that have a currency field equal to "GBP", Or you can enter the expression directly as [currency == "GBP"]).

The predicate filter is useful if your event contains an array of objects with properties, such as this array of items purchased:

{
"eventType": "transaction",
"items": [
{
"sku": "1234567",
"description": "ARIC™ Man™ action figure",
"unitCost": 22.99,
"quantity": 1,
"totalCost": 22.99
},
{
"sku": "9876543",
"description": "ARIC™ Man™ Fraud-fighter™ costume",
"unitCost": 8.99,
"quantity": 2,
"totalCost": 17.98
},
...
],
...
}

The following condition checks whether there are any items in the items array with a "sku" of "1234567", and evaluates to true.

event.items[ sku == "1234567" ].size() > 0

The default limit for a collection is 1,000 elements.

3.18 Iterating over Elements in a Collection

AMDL expressions can iterate over all elements of a collection using the * operator. Referring to a collection using the [*] syntax performs a selector operation. For example, using the example event described in Filtering Collections above:

event.items[*].totalCost

This example returns a collection of all the values of totalCost for items within that array — in the case of the example event, [ 22.99, 17.98, ... ]. If you have several nested layers, the selector is applied to everything after the [*]. For example, event.history [*].items.consumer.id returns a collection of all the items.consumer.id values nested within any elements of the collection event.history. A collection can also be nested if your event structure contains nested arrays. The expression below expands into all the values of all the field3 fields at the specified path:

event.data.array1[*].field1.field2.array2[*].field3

If a state update expression (key or value) contains the iteration operator, state variable updates with each value in the collection iteratively. The following expression results in the total costs for each item in the collection being added to the set individually:

@set(10d)
state.itemCosts: event.items[*].totalCost

Iterating on a map key results in each key being updated with the specified value:

state.itemTimes[ event.items[*].sku ]: event.eventTime

If both the key and the value contain the iterator, both collections iterate in sequence. The first key updates with the first value, and the second key with the second value, which continues.

state.itemCostsBySKU[ event.items[*].sku ]: event.items[*].totalCost

If the two collections do not have the same number of elements, the expression does not evaluate.