OpenEdge provides support for a feature called Change Data Capture (CDC). CDC is an OpenEdge RDBMS feature that identifies and captures data that changed in tables of a source database, as a result of CREATE, UPDATE, and DELETE (CUD) operations.

CDC is an industry term that describes the process of duplicating subsets of OLTP data in an external data source with a relatively up-to-date version of relational data. The OpenEdge implementation of CDC provides a flexible and scalable capture process to facilitate the data extraction, transformation, and eventually the loading of the data to an external data source.

Pro2 leverages this feature and replaces the ABL triggers with CDC. Pro2 CDC implementation also supports a feature called thread#0. When this feature is used, the CDC Admin thread converts all CDC_change_tracking records regardless of which thread the table is mapped to. This provides the convenience of having a single CDC thread that converts all CDC records to replication queue records.

The CDC replication process is faster than using ABL triggers and can be managed online. The CDC replication results in relatively low database downtime. The client application connection remains unchanged and there is no need to connect to the replication database. You can create a copy of the CDC batch program (CDCBatch.p) and append the file name with the thread number to run multiple threads. For example, CDCBatch0, CDCBatch1, and CDCBatch2.

Limitations

Pro2 and CDC is limited in that Pro2 handles only Level 0 policies. Policies should be created with the suffix Pro2 to be recognized by Pro2, for example, tablename_pro2. Moreover, Pro2 does not replicate any changes that are generated from non-Pro2 policies. To determine whether a record is read, Pro2 uses the _userMisc field in the change_tracking_table.

Enable CDC on the source database

In a CDC-based replication the replication queue records (also known as replQ records) are generated from the change tracking table. The change tracking table is in the source database. By extension, the change tracking records are stored in the source database. In a trigger based replication the replQ records are generated by deploying Pro2 triggers on source database (in this case, sports). However, for CDC-based replication, the source database should be CDC enabled.

Note: The same versions of OpenEdge must be loaded on both the Pro2 machine and source database machine. For example, if the OpenEdge Database release is 11.7, then 11.7 must be loaded on the Pro2 machine.

To enable CDC on the source database:

  1. Open the Proenv window.
  2. Shut down PAS for OpenEdge instance and source database. In this example, the source database is sports.
  3. Enter the following command to add the CDC structure file to the source database:
    prostrct add sports cdcadd.st
    Note: The structure file will assign particular areas in the source database for the CDC function.
  4. Enter the following command to enable CDC on source database:
    proutil sports -C enablecdc area ReplCDCArea indexarea ReplCDCarea_IDX
  5. Restart the PAS for OpenEdge instance and the source database.
  6. In the Pro2 web interface, select the source database from the Manage Replication tab.

    The Edit Replication window appears.

  7. Change the Source DB Mode to CDC.
Note: After you have enabled CDC on the source database, configure the replication process for CDC based replication.

Add, remove, and customize CDC threads

You can perform CDC-based replication after your source database is CDC enabled. Pro2 Treats the both replication threads and CDC threads the same, allowing you greater flexibility in your replication configurations. You can add, remove and customize threads as needed.

Perform the following steps to add replication threads:

  1. In the Pro2 web user interface, select Dashboard > Total Threads.

    The Total Threads drop-down list appears.

  2. Click +Add.

    The Add New Thread window appears.

  3. Enter a Replication Thread ID and set up the other controls based on your needs.
  4. Click Add.


  5. The CDC thread appears in the Total Threads list.

To delete CDC threads:

  1. In the Pro2 web interface, select Dashboard > Total Threads.

    The Total Threads drop-down list appears.

  2. Select the check box beside the CDC thread that you want to delete and click Delete.

    The Delete Threads window appears.

    Note: You can delete multiple threads by selecting multiple check boxes at the same time.
  3. Click OK. The CDC thread no longer appears on the Total Threads window.

To customize CDC threads:

  1. On the Pro2 web interface Dashboard, click the Replication Process drop-down list.
  2. Select CDC Process.

    The CDC threads will appear on screen.

  3. Customize the Logging Level, Disposition, and Replication Control as needed.

Map change data capture

To configure CDC replication, you must map your source and target database after your CDC source database is enabled. CDC database mapping is similar to trigger based mapping.

After you have enabled CDC on source database, a CDC Mapping tab appears in the Manage Replication window.

To enable CDC mapping:

  1. From the Manage Replication window on the Pro2 web interface, select the Source Database.

    The Edit Replication tab appears.

  2. Ensure that the Source DB Mode is CDC. Choose the Source DB Connection. Click Next.

    The Select Source tab appears.

  3. Enter information in the Select Source window, and click Save.

    The Set Target tab appears.

  4. Enter information in the Set Target window and click Save.

    The Generate Target Schema tab appears.

  5. Generate the target schema, download the schema files, and click Save.

    The Mapping Source to Target Tables tab appears.

  6. Select the tables from the source database that you want to map with the target database, and click Next.

    The CDC Mapping tab appears.

    Note: Mapping tables from source to target database is a prerequisite to CDC mapping. You will be able to view the tables that you have mapped from the source to the target database in the CDC Mapping tab. The tables which are not mapped will not appear in the CDC Mapping tab.
  7. Select the tables on which you want to activate CDC mapping, and click Add.
  8. Select the Policy Settings for each of the Source Table in the Add / Activate Policy column.
    Note: Pro2 replication is currently designed to capture records the have changed. In CDC, this is known as Policy. Activating a policy forces the OpenEdge database to immediately start tracking changes to the selected tables. Deactivating a policy forces the OpenEdge database to immediately stop tracking changes to the selected tables.
  9. Click Next.

    The Generate Code tab appears.

Split CDC threads

You can split each CDC thread into multiple threads regardless of whether the CDC thread serves the replication of a single table, or a group of tables. You can split a single thread into a maximum of 10 split threads. When you split a thread into n number of parts, it provides n-1 split threads. For example, if you split the thread 5 (CDCBatch5.bat) into four parts, it provides three split threads 51, 52, and 53 along with the original thread 5. This means, the four parts are threads 5, 51, 52, and 53.

To split CDC threads:
  1. Create new .bat files. For example:
    1. Create three copies of the replBatch5.bat file located in the Pro2\bprepl\Scripts folder and rename them as replBatch51.bat, replBatch52.bat, and replBatch53.bat respectively.
    2. Edit the replBatch51.bat file, locate and modify Thread=5 to Thread=51, and save your changes. Repeat this step for replBatch52.bat and replBatch53.bat files.
  2. Create new thread positions:
    1. Go to the Pro2 web interface and click the Advanced Configuration tab.
    2. For thread 51, type 51 in the field next to the Create New Thread button, and then click Create New Thread. Repeat this step for threads 52 and 53.
  3. Create a new property:
    1. In the Pro2 interface, click the Properties tab, and then click Add.
    2. In the Property Name field, specify the name of the property as CDCSPLIT_THREAD5.
    3. In the Value field, specify the number of threads as 4.
      Note: You can add up to n-1 value where n is the number of thread. The maximum value allowed per thread is 10. However, if you specify a value (characters or negative numbers) other than the numbers 1 to 10, the trigger uses only the actual thread number.

After generating the trigger code and creating new property for split threads, if you do not create the .bat files and do not schedule the task entries, then the data remains in the queue until the new thread jobs are started.

Progress recommends that you do not change the split number if there are existing records in the queue. Changing the split number might push the changes of the same record and row to a differently calculated queue.

Enable or disable queue compression for CDC

Note: This is an optional property and is disabled by default.

You can enable or disable queue compression by means of the CDC_QUEUE_COMPRESSION property. You can add this property by manual intervention from the web UI.

  1. Click Properties tab and select Add.
  2. Enter the Property Name as CDC_QUEUE_COMPRESSION.
  3. Set the Value to either YES or NO.
  4. Click Save.