Scripting Flexible Replication Configuration

This section describes how to use the XQuery and REST APIs to configure flexible replication.

• Configuring Push Replication using the XQuery API
• Configuring Pull Replication using the XQuery API
• Configuring Push Replication using the REST API
• Configuring Query-based Replication using the REST API

Configuring Push Replication using the XQuery API

This section describes how to use the XQuery API to configure a Master database to push replication updates to the Replica database.

• Preliminary Configuration Procedures
• Configuring the Master Database
• Creating a Replication Configuration Element
• Creating a Replication Target
• Creating a Push Replication Scheduled Task
• Configuring Pull Replication using the XQuery API

Preliminary Configuration Procedures

The procedures described in this section assume you have done the following configuration on MarkLogic Server:

1. Create three new forests:

   • MasterForest
   • MyTriggersForest

   • ReplicaForest

     The MasterForest and MyTriggersForest must be on the same server. For details on how to create forests, see Creating Forests and Databases.

2. Create three new databases:

   • Master
   • MyTriggers

   • Replica

     The Master and MyTriggers databases must be on the same server. For details on how to create databases, see Creating Forests and Databases.

3. Attach the forests to the databases:

   • MasterForest to Master
   • MyTriggersForest to MyTriggers

   • ReplicaForest to Replica

     For details on how to attach forests to databases, see Attaching Forests to Databases.

4. Configure the Master database to use the MyTriggers database as its Triggers Database:

   Copy

   xquery version "1.0-ml";
   import module namespace admin = "http://marklogic.com/xdmp/admin" 
     at "/MarkLogic/admin.xqy";
   let $config := admin:get-configuration()
   let $config := admin:database-set-triggers-database(
     $config, xdmp:database("Master"),
     xdmp:database("MyTriggers"))
   return admin:save-configuration($config)
   

5. For the MyTriggers database:

   • Insert a Flexible Replication pipeline and a Status Change Handling pipeline, as described in Inserting Existing CPF Pipelines.
   • Create a CPF domain, named Replicated Content, that uses the Flexible Replication and Status Change Handling pipelines, as described in Creating a CPF Domain.

6. Create two HTTP App Servers with the following settings

   Server Name	Root	Port	Database
   Master-flexrep	FlexRep	8010	Master
   Replica-flexrep	FlexRep	8011	Replica

   For details on how to use the Admin API to create App Servers, see Creating an App Server.

Configuring the Master Database

The flexrep:configure-database function creates the indexes needed by the Master database for CPF based replication.

xquery version "1.0-ml"; 
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
  at "/MarkLogic/flexrep.xqy";
import module namespace admin = "http://marklogic.com/xdmp/admin"
  at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration() 
let $config := flexrep:configure-database(
  $config,
  xdmp:database("Master")) 
return admin:save-configuration($config)

Creating a Replication Configuration Element

Most of the Flexible Replication API functions require a replication configuration element for each replicated domain. You create a replication configuration element by calling the flexrep:configuration-create function and insert it into the database by calling the flexrep:configuration-insert function.

The following query creates a new replication configuration element for the Replication Content domain and inserts it into the database. This query is executed against the Master database, so an xdmp:eval function is used to obtain the domain ID from the MyTriggers database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Obtain the id of the replicated CPF domain from the 
   Triggers database. :)
let $domain:= xdmp:eval(
    'xquery version "1.0-ml";
    import module namespace dom = "http://marklogic.com/cpf/domains" 
      at "/MarkLogic/cpf/domains.xqy";
    fn:data(dom:get( "Replicated Content" )//dom:domain-id)',
    (),
    <options xmlns="xdmp:eval">
      <database>{xdmp:database("MyTriggers")}</database>
    </options>)
(: Create a replication configuration for the Replicated 
   Content domain. :)
let $cfg := flexrep:configuration-create($domain)
(: Insert the replication configuration element into the database. :)
return flexrep:configuration-insert($cfg) 

Creating a Replication Target

This section describes how to use the flexrep:target-create function to create a replication target. The following query is executed against the Master database, so an xdmp:eval function is used to obtain the domain id from the MyTriggers database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Obtain the id of the replicated CPF domain from the 
   Triggers database. :)
let $domain:= xdmp:eval(
    'xquery version "1.0-ml";
    import module namespace dom = "http://marklogic.com/cpf/domains" 
      at "/MarkLogic/cpf/domains.xqy";
    fn:data(dom:get( "Replicated Content" )//dom:domain-id)',
    (),
    <options xmlns="xdmp:eval">
      <database>{xdmp:database("MyTriggers")}</database>
    </options>)
(: Obtain the replication configuration. :)
let $cfg := flexrep:configuration-get($domain, fn:true()) 
(: Specify the HTTP options for the replication target. :)
let $http-options := 
  <flexrep:http-options     xmlns:flexrep="http://marklogic.com/xdmp/flexible-replication">
    <http:authentication xmlns:http="xdmp:http">
      <http:username>admin</http:username>
      <http:password>admin</http:password>
    </http:authentication>
    <http:client-cert xmlns:http="xdmp:http"/>
    <http:client-key xmlns:http="xdmp:http"/>
    <http:pass-phrase xmlns:http="xdmp:http"/>
  </flexrep:http-options>
(: Create the replication target. :)
let $cfg := flexrep:target-create(
  $cfg,
  "Replica",
  "http://localhost:8011/",
  60,
  300,
  10,
  fn:true(),
  $http-options,
  fn:false(),
  (),
  () ) 
(: Insert the changes to the replication configuration. :)
return flexrep:configuration-insert($cfg) 

Creating a Push Replication Scheduled Task

This section describes how to use the scheduler functions to create a scheduled replication push task. The following query is executed against the Master database.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration()
(: Define a minutely scheduled task to push replication 
   each minute. :)
let $task := admin:group-minutely-scheduled-task(
  "/MarkLogic/flexrep/tasks/push.xqy",
  "Modules",
  1,
  xdmp:database("Master"),
  0,
  xdmp:user("admin"), 
  admin:host-get-id($config, xdmp:host-name())
)
(: Add the scheduled task to the Default group. :)
let $config:= admin:group-add-scheduled-task(
  $config, 
  admin:group-get-id($config, "Default"), 
  $task)
return admin:save-configuration($config)

Configuring Pull Replication using the XQuery API

This section describes how to use the XQuery API to configure a Replica database to retrieve replication updates from the Master database.

• Disabling Push Replication on the Master Database
• Creating a Pull Replication Configuration
• Creating a Pull Replication Scheduled Task

Disabling Push Replication on the Master Database

Before configuring Pull Replication on the Replica database, you must disable Push Replication on the Master database. The following query is executed against the Master database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Obtain the id of the replicated CPF domain from the 
   Triggers database. :)
let $domain:= xdmp:eval(
    'xquery version "1.0-ml";
    import module namespace dom = "http://marklogic.com/cpf/domains" 
      at "/MarkLogic/cpf/domains.xqy";
    fn:data(dom:get( "Replicated Content" )//dom:domain-id)',
    (),
    <options xmlns="xdmp:eval">
      <database>{xdmp:database("MyTriggers")}</database>
    </options>)
(: Obtain the replication configuration. :)
let $cfg := flexrep:configuration-get($domain, fn:true()) 
(: Obtain the replication target id. :)
let $target-id := flexrep:configuration-target-get-id(
  $cfg,
  "Replica")
(: Disable Push Replication on the replication target. :)
let $cfg := flexrep:configuration-target-set-enabled(
  $cfg,
  $target-id,
  fn:false())
(: Insert the replication configuration element into the database. :)
return flexrep:configuration-insert($cfg) 

Creating a Pull Replication Configuration

Pull Replication requires a pull replication configuration element for each replicated domain. You create a pull replication configuration element by calling the flexrep:pull-create function and insert it into the database by calling the flexrep:pull-insert function.

The following query is executed against the Replica database, which is most likely running on a different server than the Master database. As a consequence, you will need to first obtain the domain ID from the master database and the target ID from the master's triggers database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Specify the id of the replicated CPF domain obtained from the 
   Master's Triggers database. :)
let $domain:= 9535475951259984368
(: Specify the id of the replication target obtained from the 
   Master database. :)
let $target-id := 18130470845627037840
(: Specify the HTTP options for the replication target. :)
let $http-options := 
  <flexrep:http-options     xmlns:flexrep="http://marklogic.com/xdmp/flexible-replication">
    <http:authentication xmlns:http="xdmp:http">
      <http:username>admin</http:username>
      <http:password>admin</http:password>
    </http:authentication>
    <http:client-cert xmlns:http="xdmp:http"/>
    <http:client-key xmlns:http="xdmp:http"/>
    <http:pass-phrase xmlns:http="xdmp:http"/>
  </flexrep:http-options>
let $pullconfig := flexrep:pull-create(
  "Master",
  $domain,
  $target-id,
  "http://localhost:8010/",
  $http-options)
(: Insert the pull configuration into the Replica database. :)
return flexrep:pull-insert($pullconfig) 

Creating a Pull Replication Scheduled Task

This section describes how to use the scheduler functions to create a scheduled replication pull task. The following query is executed against the Replica database.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration()
(: Define a minutely scheduled task to pull updates from the
   Master database each minute. :)
let $task := admin:group-minutely-scheduled-task(
  "/MarkLogic/flexrep/tasks/pull.xqy",
  "Modules",
  1,
  xdmp:database("Replica"),
  0,
  xdmp:user("admin"), 
  admin:host-get-id($config, xdmp:host-name())
)
(: Add the scheduled task to the Default group. :)
let $config:= admin:group-add-scheduled-task(
  $config, 
  admin:group-get-id($config, "Default"), 
  $task)
return admin:save-configuration($config)

Configuring Push Replication using the REST API

This section describes how to use the REST API to configure a Master database to push replication updates to the Replica database.

• Preliminary Configuration Procedures
• Installing and Configuring CPF
• Creating a Replication Configuration Element
• Creating a Replication Target
• Creating a Push Replication Scheduled Task

Preliminary Configuration Procedures

This section describes how to create the databases, forests and App Servers to be used for Flexible Replication.

1. Use POST /manage/v2/databases to create a Master, Replica, and Triggers database:

   Copy

   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"database-name":"Master"}' \
   http://localhost:8002/manage/v2/databases
   
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"database-name":"MyTriggers"}' \
   http://localhost:8002/manage/v2/databases
   
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"database-name":"Replica"}' \
   http://localhost:8002/manage/v2/databases
   

2. Use POST /manage/v2/forests to create forests for the databases and attach them to the databases.

   The host property must specify the name of the host, not localhost. In this example, the name of the host is myhost.

   Copy

   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name": "MasterForest", 
        "host": "myhost.marklogic.com", 
        "database": "Master"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"MyTriggersForest",
        "host":"myhost.marklogic.com",
        "database":"MyTriggers"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"ReplicaForest",
        "host":"myhost.marklogic.com",
        "database":"Replica"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   

3. Use PUT /manage/v2/databases/{id|name}/properties to configure the Master database to use the MyTriggers database:

   Copy

   curl -v -X PUT --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"triggers-database" : "MyTriggers"}' \
   http://localhost:8002/manage/v2/databases/Master/properties
   

4. Use POST /manage/v2/servers to create the Master and Replica App Servers, configured with their respective databases.

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"Master-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8010,
      "content-database":"Master"
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default 
   
   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"Replica-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8011,
      "content-database":"Replica"
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default
   

Installing and Configuring CPF

This section describes how to install CPF and create a domain with the pipelines required for Flexible Replication.

1. Use POST /manage/v2/databases/{id|name}/pipelines to load the Flexible Replication Pipleines

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d'{"operation": "load-default-cpf-pipelines"}' \
   http://localhost:8002/manage/v2/databases/MyTriggers/pipelines
   

2. Use POST /manage/v2/databases/{id|name}/domains to create a domain

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "domain-name": "Replicated Content",
      "description": "Flexible Replication Domain",
      "scope": "directory",
      "uri": "/",
      "depth": "infinity",
      "eval-module": "Modules",
      "eval-root": "/",
      "pipeline": ["Flexible Replication","Status Change Handling"]
      }' \
   http://localhost:8002/manage/v2/databases/MyTriggers/domains
   

3. Use POST /manage/v2/databases/{id|name}/cpf-configs to install CPF:

   Copy

   curl -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
   -d '{
      "domain-name": "Replicated Content",
      "restart-user-name": "admin",
      "eval-module": "Modules",
      "eval-root": "/",
      "conversion-enabled": true,
      "permission": [{
          "role-name": "app-user",
          "capability": "read"
          }]
      }' \
   http://localhost:8002/manage/v2/databases/MyTriggers/cpf-configs?format=json
   

Creating a Replication Configuration Element

Use POST /manage/v2/databases/{id|name}/flexrep/configs to configure the Replicated Content domain to be used for Flexible Replication:

curl -v -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{"domain-name": "Replicated Content"}' \
http://localhost:8002/manage/v2/databases/Master/flexrep/configs

Creating a Replication Target

Use POST /manage/v2/databases/{id|name}/flexrep/configs/{id|name}/targets to create a flexible replication target on Master-flexrep which defines url to Replica-flexrep

curl -v -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{
   "target-name": "replica",
   "url": [ "http://localhost:8011/" ],
   "retry-seconds-min": 60,
   "immediate-push": true,
   "retry-seconds-max": 300,
   "documents-per-batch": 10,
   "enabled": true,
   "replicate-cpf": false,
   "http-options": {
       "username": "admin",
       "password": "admin",
       "client-cert": "",
       "client-key": "",
       "client-pass-phrase": ""},
   "filter-module": "",
   "filter-option": [""]
   }' \
http://localhost:8002/manage/v2/databases/Master/flexrep/configs/Replicated%20Content/targets?format=json

Creating a Push Replication Scheduled Task

Use POST /manage/v2/tasks to create a flexrep scheduled task:

curl -v -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{
   "task-enabled":true, 
   "task-path":"/MarkLogic/flexrep/tasks/push-local-forests.xqy", 
   "task-root":"Modules/", 
   "task-type":"minutely", 
   "task-period":1, 
   "task-timestamp":"2014-11-24T13:44:53.143178-08:00", 
   "task-database":"Master", 
   "task-modules":"", 
   "task-user":"admin", 
   "task-priority":"higher"}' \
http://localhost:8002/manage/v2/tasks?group-id=Default

Configuring Query-based Replication using the REST API

This section describes how to use the REST API to configure a Master database for Query-based Flexible Replication. In this example, the master database is configured to push replication updates to a database named push and a database, named pull, to pull replicated updates from the master database. Alerting is used to control what documents are are pushed and pulled, based on their content and which users have read permission on the documents.

Query-based Flexible Replication is described in Configuring Alerting With Flexible Replication in Use Flexible Replication.

These are the procedures:

• Preliminary Configuration Procedures
• Installing and Configuring CPF
• Configuring the Master Database and Creating App Servers
• Creating Users
• Configuring Alerting
• Creating a Replication Configuration Element
• Creating Replication Targets
• Creating Alerting Rules
• Configuring Pull Replication
• Creating a Push and Pull Replication Scheduled Task
• Inserting Some Documents to Test

Preliminary Configuration Procedures

1. Use POST /manage/v2/databases to create the following databases:

   master - the flexrep master database

   master-triggers - the triggers database for 'master'

   master-modules - the modules database for master's triggers

   push - the target database for push replication

   pull - the target database for pull replication

   Copy

   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"master"}' http://localhost:8002/manage/v2/databases
   
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"master-triggers"}' http://localhost:8002/manage/v2/databases
   
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"master-modules"}' http://localhost:8002/manage/v2/databases
   
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"push"}' http://localhost:8002/manage/v2/databases
   
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"pull"}' http://localhost:8002/manage/v2/databases
   

2. Use POST /manage/v2/forests to create forests for the databases and attach them to the databases.

   The host property must specify the name of the host, not localhost. In this example, the name of the host is myhost.

   Copy

   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name": "master", 
        "host": "myhost.marklogic.com", 
        "database": "master"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"master-triggers",
        "host":"myhost.marklogic.com",
        "database":"master-triggers"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"master-modules",
        "host":"myhost.marklogic.com",
        "database":"master-modules"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"push",
        "host":"myhost.marklogic.com",
        "database":"push"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"pull",
        "host":"myhost.marklogic.com",
        "database":"pull"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   

3. Use PUT /manage/v2/databases/{id|name}/properties to set master-triggers as the triggers database.

   Copy

   curl -v -X PUT --anyauth -u admin:admin \
   --header "Content-Type:application/json" \
   -d '{"triggers-database" : "master-triggers"}' \
   http://localhost:8002/manage/v2/databases/master/properties
   

Installing and Configuring CPF

This section describes how to install CPF and create a domain with the pipelines required for Flexible Replication.

1. Use POST /manage/v2/databases/{id|name}/pipelines to load the Flexible Replication Pipleines into the master-triggers database.

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d'{"operation": "load-default-cpf-pipelines"}' \
   http://localhost:8002/manage/v2/databases/master-triggers/pipelines
   

2. Use POST /manage/v2/databases/{id|name}/domains to create a domain with the Flexible Replication and Status Change Handling pipelines.

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
        "domain-name": "my-cpf-domain",
        "description": "My domain of documents",
        "scope": "directory",
        "uri": "/",
        "depth": "infinity",
        "eval-module": "master-modules",
        "eval-root": "/",
        "pipeline": ["Flexible Replication","Status Change Handling"]
      }' \
   http://localhost:8002/manage/v2/databases/master-triggers/domains
   

3. Use POST /manage/v2/databases/{id|name}/cpf-configs to install CPF:

   Copy

   curl -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
   -d '{
       "domain-name": "my-cpf-domain",
       "restart-user-name": "admin",
       "eval-module": "master-modules",
       "eval-root": "/",
       "conversion-enabled": true,
       "permission": [{
           "role-name": "admin",
           "capability": "execute"
          }]
      }' \
   http://localhost:8002/manage/v2/databases/master-triggers/cpf-configs?format=json
   

Configuring the Master Database and Creating App Servers

1. Use PUT /manage/v2/databases/{id|name}/properties to configure the range indexes needed for flexible replication.

   The range indexes will replace any existing range indexes, so if the application needs additional range indexes you'll need to include them here as well.

   Copy

   curl -X PUT  --anyauth --user admin:admin --header "Content-Type:application/json" \
   -d '{"word-positions": true,
        "element-word-positions": true,
        "range-element-index":
       [ { "scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/cpf",
           "localname": "last-updated",
           "collation": "",
           "range-value-positions": false,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "last-success",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "next-try",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "unsignedLong",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "target-id",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "unsignedLong",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "domain-id",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         {"scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "received-at",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }]}' \
   http://localhost:8002/manage/v2/databases/master/properties
   

2. Use POST /manage/v2/servers to create a master-flexrep App Server to service the master database. The master-flexrep App Server is also used to service the alerting rules, as described in Creating Alerting Rules.

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"master-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8010,
      "content-database":"master",
      "log-errors": true,
      "default-error-format": "compatible",
      "error-handler": "/error-handler.xqy",
      "url-rewriter": "/rewriter.xml",
      "rewrite-resolves-globally": false
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default 
   

3. Create a push-flexrep App Server for the replica database to receive push replicated content. Each target cluster would need to have one of these configured for inbound replication.

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"push-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8011,
      "content-database":"push",
      "log-errors": true,
      "default-error-format": "compatible",
      "error-handler": "/error-handler.xqy",
      "url-rewriter": "/rewriter.xml",
      "rewrite-resolves-globally": false
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default
   

4. Create a pull-flexrep App Server for the replica database to receive pulled replicated content. This would be used for status information on the replica database.

   Copy

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"pull-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8012,
      "content-database":"pull",
      "log-errors": true,
      "default-error-format": "compatible",
      "error-handler": "/error-handler.xqy",
      "url-rewriter": "/rewriter.xml",
      "rewrite-resolves-globally": false
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default
   

Creating Users

Use POST /manage/v2/users to create user1 for the query-based push target and user2 for the query-based pull target. Create flexrep-admin-user for admin access to the master-flexrep App Server.

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"user-name": "user1",
     "password": "password1",
     "description": "first user",
     "role": [ "flexrep-user", "alert-user" ]
    }' \
http://localhost:8002/manage/v2/users

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"user-name": "user2",
     "password": "password2",
     "description": "second user",
     "role": [ "flexrep-user", "alert-user" ]
    }' \
http://localhost:8002/manage/v2/users

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"user-name": "flexrep-admin-user",
     "password": "flexrep-admin-password",
     "description": "FlexRep administrative user",
     "role": [ "flexrep-admin", "flexrep-user" ]
    }' \
http://localhost:8002/manage/v2/users

Configuring Alerting

Use POST /manage/v2/databases/{id|name}/alert/configs to create the alerting configuration for the master database.

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{
  "uri": "http://acme.com/alerting",
  "name": "qbfr",
  "description": "alerting rules for query-based flexrep",
  "trigger": [],
  "domain": [],
  "action": [],
  "option": []
   }' \
http://localhost:8002/manage/v2/databases/master/alert/configs

Use POST /manage/v2/databases/{id|name}/alert/actions to create the alert action and apply it to the alert configuration.

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{
  "name": "log",
  "description": "QBFR log action",
  "module": "/log.xqy",
  "module-db": "master-modules",
  "module-root": "/",
  "option": []
   }' \
http://localhost:8002/manage/v2/databases/master/alert/actions?uri=http://acme.com/alerting

Creating a Replication Configuration Element

Use POST /manage/v2/databases/{id|name}/flexrep/configs to configure the my-cpf-domain domain to be used for Flexible Replication:

curl -v -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"domain-name": "my-cpf-domain",
     "alerting-uri": "http://acme.com/alerting"}' \
http://localhost:8002/manage/v2/databases/master/flexrep/configs 

Creating Replication Targets

Use POST /manage/v2/databases/{id|name}/flexrep/configs/{id|name}/targets to create flexible replication targets for the push and pull databases on master database.

The user1 user is associated with user1-target to ensure only documents intended for user1 are replicated to that target. The same is done for user2 and user1-target. The alerting rules described in Creating Alerting Rules control which documents are replicated to which target.

1. Create the push target, user1-target. The user setting is required for a query-based target.

   Copy

   curl -X POST  --anyauth --user admin:admin \
   -H "Content-Type: application/json" \
   -d '{"target-name": "user1-target",
        "url": [ "http://localhost:8011/" ],
        "documents-per-batch": 10,
        "http-options": {
           "username": "admin",
           "password": "admin"},
        "user": "user1"}' \
   http://localhost:8002/manage/v2/databases/master/flexrep/configs/my-cpf-domain/targets
   

2. Create the pull target, user2-target. The user setting is required for a query-based target

   Copy

   curl -X POST  --anyauth --user admin:admin \
   -H "Content-Type: application/json" \
   -d '{"target-name": "user2-target",
        "url": [],
        "documents-per-batch": 10,
        "user": "user2"}' \
   http://localhost:8002/manage/v2/databases/master/flexrep/configs/my-cpf-domain/targets
   

Creating Alerting Rules

Use POST /v1/domains/{domain-id-or-default-domain-name}/targets/{id|name}/rules to create the alerting rules for the two users, via the master-flexrep App Server (port 8010) on the master database.

1. The first two rules match documents readable by user1 that also contain either the word apple or orange. In this way, only documents with these attributes will be pushed to the push replica database.

   Copy

   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{
         "action-name": "log",
         "name": "apple",
         "description": "apple rule",
         "query": {
           "wordQuery": {
             "options": [ "lang=en" ],
             "text": [ "apple" ]
           }
         },
         "user-name": "user1"
       }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting
   
   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{
         "action-name": "log",
         "name": "orange",
         "description": "orange rule",
         "query": {
           "wordQuery": {
             "options": [ "lang=en" ],
             "text": [ "orange" ]
           }
         },
         "user-name": "user1"
       }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting
   

2. The second two rules match documents readable by user2 that also contain either the word apple or banana. In this way, only documents with these attributes will be pulled into the pull replica database.

   Copy

   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{
         "action-name": "log",
         "name": "apple",
         "description": "apple rule",
         "query": {
           "wordQuery": {
             "options": [ "lang=en" ],
             "text": [ "apple" ]
           }
         },
         "user-name": "user2"
       }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting
   
   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{
         "action-name": "log",
         "name": "banana",
         "description": "banana rule",
         "query": {
           "wordQuery": {
             "options": [ "lang=en" ],
             "text": [ "banana" ]
           }
         },
         "user-name": "user2"
       }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting
   

Configuring Pull Replication

To configure pull replication, you need the domain id and target id of the user2-target.

1. You can return the domain id and target id by doing a GET on the following endpoints (note that these are executed on port 8010, which is the master-flexrep App Server):

   Copy

   http://localhost:8010/v1/domains/my-cpf-domain/targets?format=json
   http://localhost:8010/v1/domains/my-cpf-domain/targets/user2-target?format=json
   

2. In this example, the domain id returned is 7860365094706821763 and the target id of user2-target is 15425733952606449897. Use POST:/manage/v2/databases/{id|name}/flexrep/pulls to configure pull replication on the pull database:

   Copy

   curl -X POST  --anyauth --user admin:admin \
   -H "Content-type: application/json" \
   -d '{"pull-name": "from-master",
        "domain-id": "7860365094706821763",
        "target-id": "15425733952606449897",
        "url": [ "http://localhost:8010/" ],
        "http-options": {
             "username": "user2",
             "password": "password2"}
        }' \
   http://localhost:8002/manage/v2/databases/pull/flexrep/pulls
   

Creating a Push and Pull Replication Scheduled Task

The master database needs a scheduled task to handle retries and zero-day replication. The pull target's database also needs a scheduled task to pull content from the master database.

1. Use POST /manage/v2/tasks to create a flexrep scheduled task to push replicated content to the push database:

   Copy

   curl -X POST --anyauth --user admin:admin \
   -H "Content-type: application/json" \
   -d '{"task-priority": "higher",
        "task-user": "admin",
        "task-database": "master",
        "task-path": "/MarkLogic/flexrep/tasks/push-local-forests.xqy",
        "task-root": "Modules/",
        "task-type": "minutely",
        "task-period": "1"}' \
   http://localhost:8002/manage/v2/tasks?group-id=Default
   

2. Use POST /manage/v2/tasks to create a flexrep scheduled task to pull replicated content from the master database into the pull database:

   Copy

   curl -X POST --anyauth --user admin:admin \
   -H "Content-type: application/json" \
   -d '{"task-priority": "higher",
        "task-user": "admin",
        "task-database": "pull",
        "task-path": "/MarkLogic/flexrep/tasks/pull.xqy",
        "task-root": "Modules/",
        "task-type": "minutely",
        "task-period": "1"}' \
   http://localhost:8002/manage/v2/tasks?group-id=Default
   

Inserting Some Documents to Test

Use PUT /v1/documents to insert some JSON and XML documents into the master database. (Note that these are executed on port 8000 and that single quotes are used in the URI because of the multiple parameters.)

These calls require that you have the eval-in privilege, as described in Security Requirements in Develop Using the REST API. Also, you must insert each document with the alert-user:read permission.

1. Insert some documents that include the word 'apple.' For example:

   Copy

   curl -X PUT --anyauth  --user admin:admin -H "Content-type: application/json" \
   -d '{"produce":{"fruit": "apple"}}' \
   'http://localhost:8000/LATEST/documents?uri=/json/apple.json&database=master&perm:alert-user=read'
   

   Copy

   curl --anyauth  --user admin:admin -X PUT -H "Content-type: application/xml" \
   -d '<produce>
           <fruit>apple</fruit>
       </produce>' \
   'http://localhost:8000/LATEST/documents?uri=/xml/apple.xml&database=master&perm:alert-user=read'
   

2. Insert some documents that include the word 'orange.' For example:

   Copy

   curl -X PUT --anyauth  --user admin:admin -H "Content-type: application/json" \
   -d '{"produce":{"fruit": "orange"}}' \
   'http://localhost:8000/LATEST/documents?uri=/json/orange.json&database=master&perm:alert-user=read'
   

   Copy

   curl --anyauth  --user admin:admin -X PUT -H "Content-type: application/xml" \
   -d '<produce>
           <fruit>orange</fruit>
       </produce>' \
   'http://localhost:8000/LATEST/documents?uri=/xml/orange.xml&database=master&perm:alert-user=read'
   

3. Insert some documents that include the word 'banana.' For example:

   Copy

   curl -X PUT --anyauth  --user admin:admin -H "Content-type: application/json" \
   -d '{"produce":{"fruit": "banana"}}' \
   'http://localhost:8000/LATEST/documents?uri=/json/banana.json&database=master&perm:alert-user=read'
   

   Copy

   curl --anyauth  --user admin:admin -X PUT -H "Content-type: application/xml" \
   -d '<produce>
           <fruit>banana</fruit>
       </produce>' \
   'http://localhost:8000/LATEST/documents?uri=/xml/banana.xml&database=master&perm:alert-user=read'
   

Based on the rules created in Creating Alerting Rules, the documents containing apple and orange should be replicated to the push database and the documents containing apple and banana should be replicated to the pull database.