This feature is only available with CaseWare AnalyticsAI, part of the CaseWare Cloud suite.
To refine the tests, select the Configuration tab.
From here you can see all the tests that are run in AnalyticsAI. You can select and deselect tests to run, configure test parameters and change the emphasis (Risk Score) on specific tests.
You can also refine the test parameters.
To view the Configuration page:
- Click on 2-2 Risk Analysis (Transaction Risk) in the Documents page.
- Select Configuration.
The Results page displays.
Deselect a test’s check box if you don’t want it to run.
You can also use this page to change the emphasis (Risk Score) on specific tests and their parameters.
To change a test's Risk Score:
- Click on the test, for example, Ends in 999.
- Enter a new Risk Score, for example, 15.
- Select Run Tests to see the updated results.
Charts and graphs
Tests include a pie chart that shows the percentage and quantity of the flagged transactions for that test. Pie charts appear when a test has flagged transactions.
Some tests, such as High Amounts include a pie chart and an additional graph that helps you configure parameters.
The test-specific bar graphs show the data distribution relative to the test, so that an auditor can have a clear picture of where to set the parameter. For example, the High Amounts bar graph shows the transactions line amounts distribution. You can adjust the Percentile to flag amounts to the right of the orange bar.
In this example, the default percentile value has been changed to 99.7. Notice that for the default percentile value of 99.1% there are 21 transactions flagged. When we change the Percentile to 99.7 and rerun the tests, the number of transactions flagged drops to five.
AnalyticsAI has a number of machine learning (ML) based outlier detection tests.
These tests are turned off by default. Use the Configuration page to turn them on as desired.
The Configuration page also shows the status of individual tests as follows:
- An In progress indicator shows for any test that is being run.
- Not run icon ()
- Precondition failed icon ()
- Pending run icon ()
- Failed run icon ()
- Successful run icon ()
The default base Risk Scores on the Configuration page have been selected to reflect the importance of a transaction that has been flagged by a given test.
For example, Complex Account Combinations has a base score of 50, as most transactions tend to be simple, and complicated transactions may require closer examination.
Conversely, Unusual Users has a base score of 1, as it’s not an immediate red flag when a user shows up in a journal they don’t frequent.
This is an area where professional judgement and knowledge of the entity are valuable. If you are concerned that one particular test may be indicative of a problem, you can increase the base score for that test to bring those transactions to the top of the list.
AnalyticsAI calculates risk level relative to the overall amount of risk in the data set. It sorts all of the transactions in order of descending risk score. It then takes the top 5% of scored transactions and labels those transactions as high risk. AnalyticsAI uses exclusive threshold logic here, so that all transactions with the same score have the same risk level. Similarly, the next 10% of transactions are labeled as medium risk. All other transactions are considered low risk.
For example, assume there are 1,000 transactions in a data set, of which five have five risk points, seven have four risk points, and so on as shown below:
|Score||Number of transactions|
There are a total of 135 scored transactions. The top 5% will be scored high. Five percent of 135 is about seven transactions, and our exclusive threshold logic will ensure that transactions with the same score will all be included at the same risk level - meaning in this case all transactions with a score of five will be labeled as high.
The next cutoff, for medium risk looks at the next 10% of transactions, which is about 14 (135 * 10%). As this would split the transactions with three points, all the transactions with four points will end up being labeled as medium risk and the transactions with three or fewer points will be low risk.
This model balances being able to categorize large sets of transactions, with varied risk levels, with being flexible enough for different distributions of risk across different transaction sets.