Batch processing is a server-based function that provides additional power to external data source functionality. Elevating an external data source solution to a batch job means that Corticon Server can take several patientIds from the database's Patient table at once and pass them in as a set to the Decision Service for processing – a very efficient way to perform processing – where the Decision Service retrieves additional data for each request from the Datasource for rule conditions and to enrich the record in a relational database with results. You get greater control over queries and insert statements that are used. This is beneficial when you need finer control for performance or need to retrieve large amounts of data.

How batch processing works - Fetching the transaction identifying data from the underlying data source that will be injected into the rules engine – seed data retrieval – takes place outside the Decision Service. As such, Decision Service requests that are usually individual transactions are instead fetched in chunks for the rules engine, and then dispersed across multiple processing threads to concurrently process the incoming requests. Batch processing produces no return payload per request – however, the result of each rule processing is persisted in a relational Datasource.

The Datasource for a batch processing is typically to a relational database through an ADC or EDC connection. You can use a REST service to read data that enhances the data; however, REST latency could compromise your batch processing's performance requirements. Try it! Use the Mixed Connectivity sample in a batch scenario. For each batch item, all data to drive the sample comes from a database, while the Ruleflow calls a REST service to get the rate data.

Note:
For this sample, you must have:
  1. Completed the steps in Getting Started with ADC that:
    1. Run the scripts patient and adc that set up the database with the schema and data.
    2. Imported the metadata into the Vocabulary.
    3. Set the Ruleflow's Get Patient Data query to IndicatedPatients.
  2. Packaged and saved the Decision Service file and its Datasource Configuration file where the server will be able to access them.
  3. Installed Corticon Server with the server and Web Console options, and then started the server.

To load the batch sample:

In Corticon Studio, choose the menu item Help > Samples. Select the Advanced Sample Batch Rule Processing, and then click Done. Follow the Import dialog to bring the sample into your workspace. The batch sample is just two SQL queries that will be used later in this topic, yet that step completes setup.

Note: For details on defining and running batch processes, see the section on Batch Configurations in the Web Console guide
Batch configuration depends on the Server where the Web Console connects to have access to::
  • A Decision Service that has Ruleflow Service Callouts that reference the databases and queries
  • The Decision Service project's Datasource Configuration
  • BATCH_READ queries loaded in the query service database

To run the batch sample:

  1. In your browser, connect to the Web Console, typically http://localhost:8850/corticon (use your appropriate host name and port number), using the default credentials Admin/Admin.
  2. In the Web Console, choose Decision Services, and then click Add Decision Service for just a single Decision Service. In the dialog box:
    1. Name your service. For example, myADC_Sample 1.0.
    2. Choose the Decision Service file you created.
    3. Choose its server location.
      The settings will look similar to this:


    4. On the Database tab, choose your Datasource Configuration File.


    5. Click Save and Deploy.
  3. Choose Batch Configurations, and then click New Batch Configuration. In the dialog box:
    1. Name the batch configuration. For example, myADC_Batch.
    2. Choose your deployed Decision Service.
    3. Choose its Datasource. This is a specific database name within the Datasource configuration file.

      (See Export the Datasource Configuration file)

    4. Choose the AllPatients query.
      The settings will look similar to this:


    5. Save the batch configuration.
  4. The Batch Configuration page lists the new configuration:


    Run the batch job by clicking on its Batch Configuration name, and then clicking Execute:


  5. On the Batch Configuration page detail page you can see the Processed Count each time you run it.
  6. Test that the database is getting updates by looking at the Treatment table's Approved column.
    Note: Clearing sample writes - To reset the approval values written to the database Treatment table to NULL, run the Batch sample's clear_approved script in your DB management tool.

Parameters in queries

Batch configurations can access queries that are defined to request parameter values. The BATCH_READ query PatientsByRegion runs the query:

SELECT patientId FROM Patient WHERE region IN ({Patient.region})
To run a parameterized batch job:
  1. Create batch configuration named MyADC_Batch_Regional. Select the Decision Service you deployed, its Datasource, and the query PatientsByRegion. That adds another field to accept the values of your Query Parameters. Enter the value NE. The settings will look similar to this:


  2. Save and then run this batch job. When you inspect the Treatment table you see that only patients from the NE region have been processed.
Note: For more about the format of Corticon's queries, see How Corticon is expressed in SQL.

Creating additional sample records in the database

As the sample set of patient/treatment records is very small, the efficiency of rapid batch processing can be difficult to observe. A SQL utility is provided that will generate PATIENT_COUNT additional records for your tests. It is a good idea to also adjust the FIRST_PATIENT_ID and FIRST_TREATMENT_ID so that they are not overwriting each time you execute the utility

To create large data sets and run large batch tests:
  1. Open the script in the Batch sample's generate_patients script in your DB management tool.
  2. Change the values for: 
       SET @PATIENT_COUNT=1000
   SET @FIRST_PATIENT_ID=1000
   SET @FIRST_TREATMENT_ID=1000
    These set the number of patients you want generated, and the starting ids for patients and treatments.
  3. Run the batch job with this newly generated data, and then look at the Decision Service page to see the counters and charts.