Example—Rolling Upgrade

The following procedure is a simplified, step-by-step process for a rolling upgrade on a small, three-host RHEL8 cluster with no converter packages installed. Here is the general outline of the procedure:

1. Back up all of your databases.
2. Make any changes to software applications.
3. Proceed with the rolling upgrade, failing over and upgrading each node.
4. Verify that you can commit the upgrade.
5. Change the cluster effective version to the new version.
6. Do any necessary cleanup.

In addition, prior to starting the upgrade, you may need to modify some of your existing software to run in a mixed version cluster. See Interaction with Other MarkLogic Features for details.

To perform the rolling upgrade, follow these steps:

1. Back up all databases in your existing cluster. See Backing Up and Restoring a Database for details on backing up your databases.

2. Modify any code that needs to be modified. See Interaction with Other MarkLogic Features for a list of potential software issues.

3. Start the rolling upgrade on the host that contains your primary security and schema forests.

4. Configure your load balancer to avoid sending further transactions towards the host to be upgraded.

5. Wait for all existing and queued transactions towards the target host to complete before proceeding with the next steps.

6. [FOR THE FIRST NODE ONLY] Trigger the forest failover for your security, schema, and other auxiliary forests.

   You can use this API:

   Copy

   curl -X POST --anyauth --user admin:admin -d "state=restart" "http://{host}:8002/manage/v2/forests/Security"
   curl -X POST --anyauth --user admin:admin -d "state=restart" "http://{host}:8002/manage/v2/forests/Schemas"
   

   Using this API prioritizes the failover of your security and schema forests over the failover of your content forests. This priority order minimizes the impact of the time needed to remount the replica auxiliary forests.

   Note:
   
   

   • While your replica security forest is remounting, no one can authenticate.

   • While your replica schema forest is remounting, no TDEs, redaction, etc. are available.

   • While your replica trigger forest is remounting, no pre- or post-commit triggers can occur.

   • While your replica module forest is remounting, no data services, REST API extensions, etc. are available.

7. Stop MarkLogic Server. Use this cURL command so that you can also take advantage of the fast failover feature:

   Copy

   curl -X POST --anyauth --user admin:admin -d "state=shutdown&failover=true" "http://{host}:8002/manage/v2/hosts/{host}"
   

   Note:
   
   

   "Fast" does not mean "instantaneous." It will still take some time to remount the replica forests as primary.

8. Uninstall the existing MarkLogic package:

   Copy

   rpm -e MarkLogic
   

9. Install the new RPM:

   Copy

   rpm install [rpm filename]
   

10. Start MarkLogic Server:

    Copy

    sudo /sbin/service MarkLogic start
    

11. Wait for the forests on this node to catch up with replication. Local forests' mount state should be in sync replicating:

    This command

    Copy

    curl --anyauth --user admin:admin "http://{host}:8002/manage/LATEST/forests/{id|name}?view=status&format=json"
    

    should return a result including this property:

    Copy

    {
     ...
      "status-properties": {
       "state": {
         "units": "enum",
         "value": "sync replicating"
       }
     }
     ...
    }
    

12. [FOR THE FIRST NODE ONLY] Trigger the forest failover for your replica security, schema, and other auxiliary forests that are acting as primary using this REST API:

    Copy

    curl -X POST --anyauth --user admin:admin -d "state=restart" "http://{host}:8002/manage/v2/forests/Security-replica"
    curl -X POST --anyauth --user admin:admin -d "state=restart" "http://{host}:8002/manage/v2/forests/Schemas-replica"
    

    Using this API prioritizes the failover of your security and schema forests over the failover of your content forests. This priority order minimizes the impact of the time needed to remount the primary auxiliary forests.

    Note:
    
    

    • While your primary security forest is remounting, no one can authenticate.

    • While your primary schema forest is remounting, no TDEs, redaction, etc. are available.

    • While your primary trigger forest is remounting, no pre- or post-commit triggers can occur.

    • While your primary module forest is remounting, no data services, REST API extensions, etc. are available.

13. Trigger a failover for the replica forests that are acting as primary:

    Copy

    curl -X POST --anyauth --user admin:admin -d "state=restart" "http://{nodeX}:8002/manage/v2/forests/{content-replica-X}"
    

14. Repeat Step 4 - Step 13 for node2 and node3.

15. When you have completed all of the host upgrades, check the software version and the effective version for the cluster, and then commit the upgrade:

    1. Use one of these queries to check if the cluster is ready to commit the upgrade. It will return true when the cluster is ready:

       JavaScript

       Copy

       const admin = require('/MarkLogic/admin.xqy');
       admin.canCommitUpgrade();
       

       XQuery

       Copy

       xquery version "1.0-ml"; 
       import module namespace admin = "http://marklogic.com/xdmp/admin" 
       at "/MarkLogic/admin.xqy";
       admin:can-commit-upgrade()
       

    2. Upgrade the security database on the host cluster:
       Copy

       curl -X POST --anyauth --user admin:admin \
       --header "Content-Type:application/json" \
       -d '{"operation": "security-database-upgrade-local-cluster"}'\
       "http://localhost:8002/manage/v2"
       

    3. After committing the upgrade, verify it by retrieving the effective software version of the cluster. Use one of these queries:

       JavaScript

       Copy

       const admin = require('/MarkLogic/admin.xqy');
       var config = admin.getConfiguration();
       admin.clusterGetEffectiveVersion(config);
       

       XQuery

       Copy

       xquery version "1.0-ml"; 
       import module namespace admin = "http://marklogic.com/xdmp/admin" 
       at "/MarkLogic/admin.xqy";
       let $config := admin:get-configuration()
       return 
       admin:cluster-get-effective-version($config)
       

       Or use this cURL command:

       Copy

       curl -X POST --anyauth --user admin:admin -d "state=shutdown&failover=true" 
       "http://{host}:8002/manage/v2/clusters/{cluster-name}?format=json"
       

       The cluster version, returned in the property effective-version, should match the intended version. For example, if your target version is 12.0.0, make sure that the effective-version is 12000000.