Configuring Database Replication

Before you configure clusters for database replication, you must couple the clusters as described in Coupling Clusters in Administrate MarkLogic Server.

This section describes how to configure database replication on the cluster that hosts the primary database. Before starting this procedure, you must have identified all of the foreign clusters that you plan to replicate to by following the procedure described in Coupling Clusters in Administrate MarkLogic Server.

Warning:

When upgrading the replica cluster, do not disable database replication on the Security database. Disabling database replication on the Security database causes the Security forest to stay in the syncing replica state after the cluster is restarted for the upgrade. That state prevents you from accessing the Admin Interface of the replica cluster to complete the upgrade, because the Security database must be in open replica state to access the Admin Interface.

This section describes how to configure database replication from the primary cluster. You can also configure database replication on a replica cluster. But, if you are replicating to multiple replica clusters, then it is more convenient to do all of the configuration from the primary cluster.

Warning:

Any documents with URIs in a replica forest that are not in its respective primary forest will be automatically cleared when database replication is configured.

v12.1.0 and up

Follow these steps for each replicated database and for each replica cluster:

On the bootstrap host in the primary cluster, navigate to the database to be replicated (Databases > (database name)) in the left menu tree, and select Database Replication. The database replication Summary page appears.
Select the Configure tab. The Configure Database Replication wizard, Step 1 page appears.
Select the foreign cluster from the Foreign Cluster dropdown, and click Next. The Configure Database Replication wizard, Step 2 page appears.
Configure the replication settings as needed:
- Local Database as: Select Master (primary).
- Foreign Database as dropdown: Select the replica database for this primary.
- Lag Limit
- Queue Size
- Replication Enabled
Click Next to accept the configuration values. The Configure Database Replication wizard, Step 3 page appears.
Configure the connection settings:
Click Connect. A table appears on the page to show each forest in the primary (master) database and the forest in the replica database to which it will replicate.
- Connect Forests by Name = true: The Replica Forest column contains Connect by Name, and the Forest Matching column contains (matching forest available).
- Connect Forests by Name = false: The Replica Forest column has dropdowns for manually selecting replica forests, and the Forest Matching column is blank.
Make sure that each Master Forest (primary forest) is matched to the proper Replica Forest:
- Connect Forests by Name = true: Each Forest Matching column entry contains (matching forest available). If it contains anything else, then see Automatic Forest Matching States.
- Connect Forests by Name = false: Make sure that you have chosen a unique Replica Forest from the dropdown for each Master Forest (primary forest).
Click Next. The Configure Database Replication wizard, Step 4 page appears. (You may have to log in to the foreign bootstrap host before this page appears.)
Review the configuration summary.

Note:

If the Forest Matching column contains an unexpected state, then see Forest Matching Codes Reference to resolve the issue if necessary.
Click Confirm. The database replication Summary page appears with your database replication configuration.

v12.0.x

Follow these steps for each replicated database and for each replica cluster:

On the bootstrap host in the primary cluster, navigate to the database to be replicated in the left menu tree, and select Database Replication. The database replication Summary page appears.
Select the Configure tab. Configure Database Replication, Page 1 appears with your chosen database unchangeable in the Local Database field.
Select a replica cluster from the Foreign Cluster dropdown, and click OK. Configure Database Replication, Page 2 appears.
For Local Database As, select master (primary).
From the Foreign Database dropdown, select the replica database for this primary.
Set the rest of the values according to your needs. See Database Replication Field Reference.
When you have finished setting these values, click OK. The final Configure Database Replication page appears, summarizing the data that you have entered so far. This summary includes a table to show each forest in the primary (master) database and the forest in the replica database to which it will replicate.
- Connect Forests by Name = true: The Replica Forest column contains Connect by Name, and the Forest Matching column contains (matching forest available).
- Connect Forests by Name = false: The Replica Forest column has dropdowns for manually selecting replica forests, and the Forest Matching column is blank.
Make sure that each Master Forest (primary forest) is matched to the proper Replica Forest:
- Connect Forests by Name = true: Each Forest Matching column entry contains (matching forest available). If it contains anything else, then see Automatic Forest Matching States.
- Connect Forests by Name = false: Make sure that you have chosen a unique Replica Forest from the dropdown for each Master Forest (primary forest).
Click OK to confirm the database replication configuration. The Database Replication - Validated (Local) confirmation page appears.
Click OK. The Sign in page for the foreign cluster appears.
Sign in to the foreign cluster. The Confirm to Add Foreign Replica confirmation page appears.
Review the configuration summary.

Note:

If the Forest Matching column contains an unexpected state, then see Forest Matching Codes Reference to resolve the issue if necessary.
Click OK. The database replication Summary page appears with your database replication configuration.

Database Replication Field Reference

This table describes the database replication configuration fields (field names and cases may vary slightly by version):

Field Name	Description	Location [v12.1.0 and up]
Connect Forests by Name	Set to True to automatically match current primary forests to replica forests of the same name, to automatically match any future primary forests to replicas of the same name, and to automatically configure these future forests for database replication. Set to False to manually match current and future primary forests to their replicas and to manually configure future forests for database replication. This setting is required if your primary forests have different names than their replicas. See Connecting Primary and Replica Forests with Different Names. Default: True	Step 3 of Configure Database Replication wizard
Foreign Admin Interface Port	Set the foreign cluster Admin App Server bootstrap host port number. Default: `8001`	Step 3 of Configure Database Replication wizard
Foreign Admin Interface Protocol	Select from the dropdown the communication protocol for connecting to the foreign Admin App Server. Select `https` if SSL is enabled on the foreign cluster Admin App Server bootstrap host. Select `http` otherwise. Default: `http`	Step 3 of Configure Database Replication wizard
Foreign Cluster	Select the foreign cluster that contains the replica database.	Step 1 of Configure Database Replication wizard
Foreign Database as	Select the replica database for this primary database.	Step 2 of Configure Database Replication wizard
Host in Foreign Cluster	Select from the dropdown the host on the foreign cluster to be the bootstrap host.	Step 3 of Configure Database Replication wizard
Lag Limit	Set how long the primary cluster should wait for its replica cluster to acknowledge receipt of a replicated journal frame before the primary stalls replication transactions. Range: `0`-`1,000,000,000` Default:`15` seconds See Replication Lag.	Step 2 of Configure Database Replication wizard
Local Database as	Set to Master (primary) when configuring replication from the primary cluster.	Step 2 of Configure Database Replication wizard
Queue Size	Set the maximum number of journal frames that can be buffered for replication. Range: `1`-`1000` Default: `10`	Step 2 of Configure Database Replication wizard
Replication Enabled	Set to True to keep database replication enabled after restart or failover. Set to False to keep database replication disabled after restart or failover. Default: True	Step 2 of Configure Database Replication wizard

Forest Matching Code Reference

The tables in these sections explain the values that you may see in the Forest Matching column before you confirm to add the foreign replica and what to do if you see them.

Automatic Forest-Matching States

When Connect Forests by Name is set to true, these states may occur:

Forest Matching State	Description	Action
(matching forest available)	A replica forest with the same name as this primary forest exists and is available for replication.	Expected state: Continue.
(no matching forest available)	No replica forest with the same name as this primary forest exists or is available for replication.	Take one of these actions: Create a forest with the same name in the replica database. Change Connect Forests by Name to false and manually select forests.
Will Connect by Name when there is a matching forest	The forest is configured to connect by name but is waiting for a matching replica forest to become available.	Take one of these actions: Create the matching replica forest. Wait for the matching forest to become available before continuing.
Forest is in Error state	The forest cannot participate in replication due to an error condition. No information about foreign forests is available until the error is resolved.	Resolve the error condition with the forest before continuing.
no mapping defined	This forest has no replication configuration defined in the summary view.	If replication is desired for this forest, configure database replication to include this forest.

Manual Forest-Matching States

When Connect Forests by Name is set to false, these states may occur:

Forest Matching State	Description	Action
(manual forest override)	A replica forest whose name does not match the primary forest's name has been explicitly chosen.	Expected state: Continue.
No Forest Mapping selected	No replica forest has been selected for this primary forest.	Go back and select a replica forest for this primary forest.
Forest is in Error state	The forest cannot participate in replication due to an error condition. No information about foreign forests is available until the error is resolved.	Resolve the error condition with the forest before continuing.
no mapping defined	This forest has no replication configuration defined in the summary view.	If replication is desired for this forest, configure database replication to include this forest.

Connecting Primary and Replica Forests with Different Names

If you are replicating between databases that contain forests with different names, then you can manually connect primary to replica forests.

Note:

The replica databases must have at least as many forests as the primary database. Otherwise, not all of the data on the primary database will be replicated.
You must match each primary forest to a unique replica forest.

Follow the steps in Configuring Database Replication, setting Connect Forests by Name to false and setting any other values as desired. At the appropriate point in the process, the forest matching table appears with dropdowns in the Replica Forest column.
Use the dropdowns to choose a unique Replica Forest for each Master Forest (primary forest).
When each Master Forest (primary forest) has a unique Replica Forest, click OK, then continue the rest of the steps in Configuring Database Replication until the Summary tab appears.

Enabling, Disabling, Suspending, and Resuming Database Replication

At the top of the database replication Summary page are buttons to disable or enable database replication, suspend or resume database replication, and repair the replica database indexing information.

More details about the button functions are in this table ([v12.0.x] the button labels are lowercase):

Button	Description
Disable	Disable database replication. This button changes the database replication configuration so that database replication remains disabled after restart or failover. Toggles to Enable when clicked.
Enable	Enable database replication. This button changes the database replication configuration so that database replication remains enabled after restart or failover. Toggles to Disable when clicked.
Suspend	Suspend database replication. This button does not change the database replication configuration. After a forest failover or node restart, replication of a suspended database resumes for that forest only. The other forests are still suspended until Resume is clicked or until the other remaining forests go through failover or node restart. Toggles to Resume when clicked. Note: Database replication may be suspended internally for a short period of time when rebalancing of documents occurs between two forests.
Resume	Resume database replication. This button does not change the database replication configuration. Toggles to Suspend when clicked.
Repair	Repair the indexing information on the replica database when the replica index configuration is different from the primary database. Index configurations involving either the Security database or the Schemas databases, such as TDE and ELS, are excluded from the repair. You can also use `xdmp.forestValidateReplicaIndex()` or `xdmp:forest-validate-replica-index()`.

The indexing information on the replica database can become different from that on the primary database because the replica is read-only and so does not reindex like the primary. This condition needs to be repaired so that queries return the same results from both databases.

Note:

The index settings on the primary database are used on the replica for as long as replication is enabled. When replication is disabled, the replica's original index settings are reinstated. When replication is enabled, queries that rely on the original replica index settings fail.

Deleting a Database Replication Configuration

You can delete a database replication configuration from a bootstrap host on either the primary or the replica cluster. Follow these steps to delete a database replication configuration on the primary cluster:

On the bootstrap host in the primary cluster, navigate to the primary database in the left menu tree, and select Database Replication. The database replication Summary page appears.
In the Foreign Replicas for Database (database) section of the Summary page, click Delete in the Delete column for the Foreign Cluster to delete. The Delete Database Replication page appears.
If SSL is enabled on the replica cluster's bootstrap host's Admin App Server, select https from the Foreign Admin Interface Protocol dropdown.
Click OK. The Sign in page for the foreign cluster appears.
Sign in to the foreign cluster. The Confirm to Delete Database Replication page appears.
Click OK. The Summary tab appears.

Decoupling the Local and Foreign Clusters

Note:

Before decoupling clusters, you must delete any database replication configurations between the local cluster and the cluster to be decoupled.

To decouple clusters, see Decoupling Clusters in Administrate MarkLogic Server.

Configuring App Servers on the Replica Cluster

As described in Reducing Blocking with Multi-Version Concurrency Control in Develop Server-Side Applications, setting multiversion concurrency control to nonblocking on an app server minimizes transaction blocking, which is useful if the app server uses a replica database that significantly lags its primary database.

We recommend that, at a minimum, you set multiversion concurrency control to nonblocking on the Admin App Server because the Security database is typically not updated frequently. The nonblocking multiversion concurrency control option minimizes transaction blocking, but queries potentially see a less timely view of the database. So, you must weigh these two factors when determining whether to set this option on other app servers on your replica cluster.

Changing the Foreign Bind Port

As described in Inter-cluster Communication, communication between clusters is done using the intra-cluster XDQP protocol on the foreign bind port. By default, the foreign bind port is port 7998. This section describes how to change the foreign bind port.

Note:

Changing the foreign bind port of a host causes it to automatically restart.

Follow these steps for each host in the cluster that is involved in inter-cluster replication:

Select a host in the local cluster under Hosts in the left menu tree. The Host Configuration page appears.
Change the Foreign Bind Port to your desired value, and click OK. The host restarts.

TCP Tuning For High-Latency Environments

On Linux systems, if you have configured database replication where there is high-latency between the primary and the replica environments (for example, if your primary is in San Francisco and your replica is in Tokyo), then you might need to tune the Linux TCP settings to increase throughput. These are examples of the tuned Linux TCP setting (these settings are tuned with values greater than the typical defaults):

# sysctl -w net.core.rmem_max=8388608
net.core.rmem_max = 8388608
# sysctl -w net.core.wmem_max=8388608
net.core.wmem_max = 8388608
# sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'
net.ipv4.tcp_mem = 8388608 8388608 8388608
# sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'
net.ipv4.tcp_rmem = 4096 87380 8388608
# sysctl -w net.ipv4.tcp_wmem='4096 87380 8388608'
net.ipv4.tcp_wmem = 4096 87380 8388608
# sysctl -w net.ipv4.route.flush=1
net.ipv4.route.flush = 1

To see your current TCP settings, run this Unix command:

# sysctl -a | grep mem

The preceding setting changes the running system, but it does not survive a reboot. So, once you have tuned your system to your satisfaction, you need to add these settings to your startup environment to persist them through system reboots.

For details on your TCP settings, see your operating system documentation. If you have questions about how these operating system parameters might behave with your MarkLogic environment and you have an active maintenance contract, then you can contact MarkLogic Technical Support for help.

Configure Database Replication