Corticon provides several techniques for data integration. Which ones are right for you depends on your use case – you can assemble the right mix to suit your needs.

Here is a video overview of the data access options:

When you want the Datasource to create the Vocabulary

You could choose to connect to a database or REST Datasource to populate your Vocabulary. You need not be bound to the Datasource once the Vocabulary has been generated. For more about this technique, see the section Populate a Vocabulary from a Datasource .

When to use the Enterprise Data Connector (EDC)

Corticon EDC accesses data is a relational database to augment data when processing a discrete Decision Service request. It is used by rule authors who like the user friendly and intuitive way of modeling data access and persistence in their Rulesheets to conditionally enrich transactional data, and to use reference data for rule processing using a single backend relational database.

For example, a single request to adjudicate an insurance claim tells Corticon to retrieve all required data and related data from a database to service the request. Corticon performs well in this scenario including the persistence of the claim processing result back into the database.

Corticon EDC uses an object relational model (ORM) where entities mapped to a database are automatically enriched with database data and optionally saved back to the database. This approach makes it very easy to use database data in your rules but also introduces query and data processing overhead when reading data from a database with related tables. Both read and write performance can suffer in some use cases with EDC. When performing reads, the number of queries required to retrieve an entity and all associated entities can grow exponentially with each level of associations. When performing writes, each updated object is committed to the database as a distinct update instead of a large, single pass multi-record update.

Database read and update through EDC is a good choice for a single Decision Service request scenario with limited amounts of data. Examples are individual claims processing, single client eligibility requests, and a single transaction validation request. Many Corticon users have EDC running every day with EDC deployment scenarios.

EDC's limitations are in its performance in large, data intensive operations where large chunks of data are loaded into Corticon for processing and updating the database.

EDC can be thought of as the "Easy Data Connector" because it provides the simplest means of connection to a database. This contrasts with the "Advanced Data Connector" which requires greater knowledge of SQL but provides greater flexibility.

When to use Advanced Data Connectors (ADC)

ADC also provides for accessing data in a database but takes a different approach than EDC to address different use cases. ADC is very efficient in dealing with large data sets. ADC has the ability to connect to disparate databases--you can use ADC to read from and write to multiple databases in a single decision service. With EDC you're limited to one database. Both read and write performance is much better than EDC when processing larger data sets in a Decision Service request. ADC can read related data in a few passes from the database, where EDC requires discrete queries to fetch data. And ADC can write back data in chunks, where EDC writes data as discrete updates.

Both ADC and EDC are single-threaded -- a single request is executed by a single Decision Service reactor which has access to the full collection of data included in the request payload, database exposed entities and filter criteria in the Rulesheets (EDC) or ADC queries.

Using ADC, you get great performance when processing a large dataset through a single Decision Service request. You can use ADC to quickly process a set of unrelated records such as individual customers or work orders. You can also use it when you need to do operations such as aggregations and clustering. You can build rules that operate on the full collection of data. As examples, you can quickly adjudicate all medical claims in a month or approve specific procedures across all hospitals in a specified region, or calculate sales prices for all items in stock. In some situations, it is imperative to have access to the full collection of data for your rules to work properly. For example, when sales prices are calculated based on clustering rules whereby all sales prices for products in the same cluster are based on the average purchase price of their respective product cluster.

When to use the REST Datasource

The Corticon REST Datasource provides support for accessing REST services. It allows you to retrieve REST data to enrich the payloads being processed by your rules.

The Corticon REST Datasource uses the Data Direct Autonomous REST Connector which provides the ability to access REST services as if they were databases. This is beneficial to a Corticon user because the process of mapping a vocabulary to a REST service is the same as for EDC and ADC data sources.

To configure the REST Datasource you either perform schema discovery or supply your own schema file. When using schema discovery, you supply the URL of the REST data source and query parameters and allow the Autonomous REST Connector to generate a schema for your REST service. To supply your own schema, you can either export a discovered schema from Corticon Studio and make edits or create one from scratch. See the Autonomous REST Connector documentation for details on the its schema file format.

The query parameters for a REST Datasource can either be fixed or dynamically set by data in your payload. Dynamic setting of parameters allows you to access a REST service to retrieve information about a specific entity in your payload. You can also configure the security settings for accessing a REST service.

As REST access is limited to read-only, it is ideal for data enrichment. You could have one or several REST Datasources used by a decision service. You can even mix EDC or ADC with REST depending on your data access needs.

The wealth of REST data sources exposed through APIs means that you could be touching multiple sources to build the best complete data set possible. In marketing scenarios that might mean taking sparse info on a prospect from social or business contacts to enrich the data by discovering their profile and preferences to focus campaigns and assign local reps for follow-up. In medical applications, diagnoses and treatments can be enriched with claims approval histories or related clinical trials. For mortgage lenders, quickly scanning multiple credit review resources for a prospect, and then matching their home value and loan to retrieve the best rate from multiple lenders.

When to use batch processing

Batch processing is used to process large data sets either after hours, during periods of low system usage or to meet business demands such as monthly or quarterly reporting. Corticon's batch processing can be used with ADC to efficiently process huge amounts of data. Batch decision services can also use REST Datasources.

A requirement for batch processing is that each transaction stands on its own, not needing access to the full collection of data to make decisions on single transactions. As only so much data can be loaded into Corticon working memory at once, the data would need to be fed to the rules engine in chunks to then process the chunks concurrently based on resource capacity. Note that there are no return payloads in batch processing – the result of all the rule processing is persisted in the database.

Batch processing usually runs against the same input source to process large volumes of data so it is set to run at scheduled such as nightly or monthly. Corticon's Web Console can be used to schedule batch executions or you can use external tooling to perform scheduling and call Corticon REST API to start a batch execution.