Database Replication in MarkLogic Server

This section describes database replication in MarkLogic Server in general terms, and includes these sections:

• Terms Used in this Guide
• Understanding Database Replication
  • Overview of Database Replication
  • Bulk Replication
  • Bootstrap Hosts
  • Inter-cluster Communication
  • Replication Lag
  • Primary and Replica Database Index Settings
• Example Database Replication Configurations
• Database Replication for Disaster Recovery
• Rolling Back to the Non-Blocking Timestamp
• Upgrading Clusters Configured with Database Replication
• Comparison of Database Replication with Flexible Replication

Terms Used in this Guide

These are the database replication terms used in this guide:

• To replicate is to create a copy of a document in another database and to keep that copy in sync (possibly with some time-lag/latency) with the original.

• A local cluster is the cluster of MarkLogic Server hosts at your current location.

• A foreign cluster is a remote cluster of MarkLogic Server hosts.

• A primary (formerly master) database is the database being replicated. In any database replication scheme, there is a primary database and at least one replica database.

• A primary (formerly master) forest is the forest being replicated. In any database replication scheme, there is a primary forest and at least one replica forest.

• A primary (formerly master) cluster is shorthand for the cluster that hosts a primary database.

• A replica database is the database that receives replicated data from the primary database.

• A replica forest is the forest that receives replicated data from the primary forest.

• A replica cluster is shorthand for the cluster that hosts a replica database.

• A bootstrap host is the MarkLogic Server host machine used by a foreign cluster to initiate communication with the local cluster.

• A foreign bind port is the port used on each host to handle XDQP communication with foreign clusters.

• Asynchronous replication refers to a configuration in which the primary does not wait for confirmation that the update has been received by the replica before committing the transaction and proceeding with additional transactions. Database replication is asynchronous.

• Transaction-aware refers to a configuration in which all updates that make up a transaction on the primary are applied as a single transaction on the replica.

• Zero-day replication refers to replicating the data from the primary database that existed before replication was configured.

Understanding Database Replication

Database replication is the process of maintaining copies of forests on databases in multiple MarkLogic Server clusters. At a minimum, there will be a primary database and one replica database. A primary database can replicate to multiple databases. A replica database cannot serve as a primary database to another replica.

This section contains these topics:

• Overview of Database Replication
• Bulk Replication
• Bootstrap Hosts
• Inter-cluster Communication
• Replication Lag
• Primary and Replica Database Index Settings

Overview of Database Replication

Database replication operates at the forest level by copying journal frames from a forest in the primary database and replaying them on a corresponding forest in the foreign replica database.

As shown in the following illustration, each host in the primary cluster connects to the remote hosts that are necessary to manage the corresponding replica forests. Replica databases can be queried but cannot be updated by applications.

Bulk Replication

Any content existing in the primary databases before database replication is configured is bulk-replicated into the replica databases. Bulk replication is also used after the primary and foreign replica have been detached for such a long time that journal replay is no longer possible. Once bulk replication is complete, journal replication proceeds.

This is the bulk replication process:

1. The indexing operation on the primary database maintains a catalog of the current state of each fragment. The primary sends this catalog to the replica database.

2. The replica compares the primary's catalog to its own and updates its fragments using this logic:

   • If the replica has the fragment, then it updates the nascent or deleted timestamps if they are wrong.
   • If the replica has a fragment that the primary does not have, then it marks that fragment as deleted (it likely existed on the primary at some point in the past but has been merged out of existence).
   • If the replica does not have a fragment, then the replica adds that fragment to a list of missing fragments to be returned by the primary.

3. The primary iterates over the list of missing fragments returned from the replica and sends each of them, along with their nascent or deleted timestamps, to the replica, where they are inserted.

For more information on fragments, see Fragments in Administrate MarkLogic Server.

Bootstrap Hosts

Each cluster in a database replication scheme contains one or more bootstrap hosts that are used to establish an initial connection to foreign clusters that it replicates to or from and to retrieve more complete configuration information once a connection has been established. When a host initially starts up and needs to communicate with a foreign cluster, it bootstraps communications by establishing a connection to one or more of the bootstrap hosts on the foreign cluster. Once a connection to the foreign cluster is established, cluster configuration information is exchanged between all of the local and foreign hosts.

For details on selecting the bootstrap hosts for your cluster, see Coupling Clusters in Administrate MarkLogic Server.

Inter-cluster Communication

Communication between clusters is done using the intra-cluster XDQP protocol on the foreign bind port. A host listens on the foreign bind port only if it is a bootstrap host or if it hosts a forest that is involved in inter-cluster replication. By default, the foreign bind port is port 7998, but it can be configured for each host as described in Changing the Foreign Bind Port. When secure XDQP is desired, a single certificate or private-key pair is shared by all hosts in the cluster when communicating with foreign hosts.

XDQP connections to foreign hosts are opened when needed and closed when no longer in use. While the connections are open, foreign heartbeat packets are sent once every second. The foreign heartbeat contains information used to determine when the foreign cluster's configuration has changed so that the local bootstrap host can retrieve updated information from the foreign bootstrap host.

Replication Lag

Queries on a replica database must run at a timestamp that lags the current cluster commit timestamp due to replication lag. Each forest in a replica database maintains a special timestamp, called a non-blocking timestamp, that indicates the most current time at which it has complete state to answer a query. As the replica forest receives journal frames from its primary, it acknowledges receipt of each frame and advances its non-blocking timestamp to ensure that queries on the local replica run at an appropriate timestamp. Replication lag is the difference between the current time on the primary and the time at which the oldest unacknowledged journal frame was queued to be sent to the replica.

You can set a lag limit in your configuration that specifies how long the primary should wait for an acknowledgement from the replica before stalling its transactions to let the replica catch up. The default lag limit is 15 seconds, which works well for most installations. To set the lag limit, see Configuring Database Replication.

Primary and Replica Database Index Settings

Starting with MarkLogic 9.0-7, the primary database replicates indexing information to the replica system. This replication ensures that the index data on the replica is always in sync with the primary database. If you want the option to switch over to the replica database after a disaster, then you still need to ensure that the index settings are identical on the primary and replica clusters.

If you need to update index settings after configuring database replication, then make sure that they are updated on both the primary and replica databases. Changes to the index settings on the primary database trigger reindexing, after which the reindexed documents are replicated to the replica. When a database replication configuration is removed for the replica database (such as after a disaster), the replica database reindexes, if necessary.

Example Database Replication Configurations

This section describes some of the possible database replication configurations.

The most basic form of database replication consists of two clusters, each containing a single database with a single forest, as in the following scenario:

Primary and replica forests can be distributed differently across hosts in each cluster. In the following scenario, all of the primary forests reside on a single host and the replica forests on two hosts:

A primary cluster can replicate forests to multiple replica clusters. In the following scenario, the primary maintains a copy of each forest on two clusters:

In the following scenario, the primary maintains a single copy of Database 1 on Replica Cluster 1 and a single copy of Database 2 on Replica Cluster 2:

A cluster can contain both primary and replica databases. On Cluster A in the following scenario, Database 1 is the primary database and Database 2 is the replica database:

Database Replication for Disaster Recovery

Database replication makes data continuously available to mission-critical applications with minimal impact to application performance. In a typical disaster recovery scheme, clusters are located in data centers at different geographical locations. If a disaster happens at the primary cluster location, the replica cluster at a different location can make the data available.

There are two basic approaches to a disaster recovery operation, each of which involves redirecting the applications to Database B. Before executing your applications on Database B, the database must be rolled back, as described in Rolling Back to the Non-Blocking Timestamp.

• Simply disable database replication on Database B so that it is no longer a replica and can accept updates:

  

• Reconfigure Database B as the primary database and replicate updates to a third foreign database (Database C):

  

When Database A is returned to service, you can resynchronize it with Database B in one of two ways:

• Configuring Database A as a replica of Database B. In this case, Database B will automatically bulk replicate to Database A:

  

• Running a backup operation on Database B and restoring Database A from the backup file, as described in Backing Up and Restoring a Database in Administrate MarkLogic Server:

  

Once the data has been restored on Database A, you can redirect the applications from Database B back to Database A.

Database replication can be used in combination with the local-disk failover feature described in High Availability of Data Nodes With Failover and Configuring Local-Disk Failover for a Forest in Scalability, Availability, and Failover.

In the following scenario, the primary and replica clusters are configured for both local-disk failover and database replication. If, for example, the F1 forest in the primary cluster becomes unavailable, then the R1 forest resumes replication to the F1 forest in the replica cluster:

Rolling Back to the Non-Blocking Timestamp

After you fail over your applications to a replica database, each replica forest will likely have committed its last transaction at a different timestamp. If the replica database has multiple forests, and relationships exist between those forests, then this inconsistency may cause problems. To return the database to a transactionally consistent state, you must roll back all forests to the minimum non-blocking timestamp.

The following timeline shows four forests and their committed transactions before failure. 'T#-u#' represents transaction updates. 'C' represents commits:

When the primary forest failure occurred, each forest had completed its last commit at a different timestamp: Forest A has committed transactions only up to Timestamp 3 while Forest B has committed transactions up to Timestamp 6. So, to return the database to a transactionally consistent state, you must roll back all forests to at least Timestamp 3.

The following timeline shows the minimum non-blocking timestamp identified as Timestamp 3 and the four forests rolled back to that timestamp:

Follow these steps to restore to the minimum non-blocking timestamp using JavaScript or XQuery functions:

1. Use xdmp.forestStatus() or xdmp:forest-status() to locate the non-blocking timestamp value for each forest.

2. Use admin.databaseGetMergeTimestamp() or admin:database-get-merge-timestamp() to get the current merge timestamp. Save this value so that it can be reset after you have completed the rollback operation.

3. Use admin.databaseSetMergeTimestamp() or admin:database-set-merge-timestamp() to set the merge timestamp to any time before the failure of the primary database. Doing this preserves fragments in merge after this timestamp until you have rolled back your forest data.

4. Use xdmp.forestRollback() or xdmp:forest-rollback() to roll back the forests to the minimum non-blocking timestamp returned in Step 1 for each forest that is in the open or open replica state.

5. Use admin.databaseSetMergeTimestamp() or admin:database-set-merge-timestamp() to set the merge timestamp back to the value that you saved in Step 2.

Upgrading Clusters Configured with Database Replication

If the Security database is not replicated, then there should not be anything special that you need to do other than upgrade the two clusters.

If the Security database is replicated, then follow these steps:

1. Upgrade the replica cluster and run the upgrade scripts. Doing this updates the replica's Security database to indicate that it is current. It also does any necessary configuration upgrades.

2. Upgrade the primary cluster and run the upgrade scripts. Doing this updates the primary's Security database to indicate that it is current. It also does any necessary configuration upgrades.

Comparison of Database Replication with Flexible Replication

MarkLogic Server provides two types of replication: database replication, as described in this guide, and flexible replication, as described in Use Flexible Replication. Choose a replication type based on your application and high-availability requirements.

Depending on your requirements, one type of replication might have an advantage over the other. These are the key differences between the two types of replication:

• How much of the primary database is replicated:
  • Database replication copies the entire primary database to the replica database.
  • Flexible replication copies only select content from the primary database by replicating Content Processing Framework (CPF) domains and allowing you to write XQuery filters that enable you to modify the replicated documents in some way, determine whether to replicate a change, or select which parts of a document to replicate.

• Performance:
  • Database replication has less overhead because it simply copies and replays journal frames.
  • Flexible replication has more overhead because it uses CPF and generates information in the properties of each replicated document.

• Transactional completeness:
  • Database replication replicates as soon as a document is inserted or updated and ensures that a single multi-document transaction in the primary database replicates as a single multi-document transaction to the replica database. So, if a transaction commits 10 documents to the primary database and failure occurs, then the replica will have either all 10 documents or none of them.
  • Flexible replication replicates documents after each transaction without consideration of transactional grouping in the primary database. So, if a transaction commits 10 documents to the primary database and failure occurs, then the replica could wind up with, say, 5 documents.