Overview of a MarkLogic Cluster on AWS

A Managed Cluster is automatically initialized and pre-configured with recommended topology, such as the one illustrated below. This diagram shows a typical architecture of MarkLogic Cluster on AWS. Elastic Network Interfaces are attached to each MarkLogic node and CloudWatch is used to monitor the EC2 instance health. AWS Lambda is used to managed Elastic Network Interface in VPCs.

The sample CloudFormation templates implement a simple example of this reference architecture and makes use of the Managed Cluster feature. Regardless of how the cluster is created, the necessary components need to be created, configured and deployed in a controlled fashion.

Cloud Formation is an AWS Technology that allows you to specify the set of components necessary for creating a Stack. You can use one of the provided Amazon Cloud Formation templates to create a Managed Cluster. The Managed Cluster templates create:

• IAM Roles necessary for running AWS services without needing to pass in security credentials

• Security groups to control the incoming network traffic delivered to the instances.

• AutoScaling groups one per node

• Launch Configuration for the AutoScaling Groups

• Load balancer fronting all of the nodes

• EBS Volumes for each node

When using the Cloud Formation templates there are parameters that must be filled in (either via the AWS Console or any 3rd party command line tool that can launch a cloud formation stack). These parameters include:

• What Zone each node will run in

• The admin user and password for initially creating the security database

• The SSL Key name (Used to login to the instances once they are started)

• The size and EBS type of the volumes (in GB) to create for the initial data volume /var/opt/MarkLogic.

• The EC2 instance type of the created instance.

• Optional: The Simple Notification Service (SNS) topic to be used to capture messages from the AutoScaling Groups and Managed Cluster Support startup procedure.

When launched, the Cloud Formation creates all the necessary resources. On startup, the Amazon EC2 nodes recognize that they are part of a Managed Cluster and perform the following actions without user intervention:

• Attach any volumes associated with this node

• Create a filesystem, if needed

• Mount the filesystem

• Start MarkLogic

• Apply and accept the EC2 license

• Either create the initial node (master) and set the admin username and password or attach to the cluster

• Associate the node with the Load Balancer

The Load Balancer detects proper running of MarkLogic via the HealthCheck App Server on port 7997 and will only direct traffic to that node if it has verified that the MarkLogic instance is up and running.

Each AutoScaling Group (ASG) detects system stability and will terminate and restart the node if the operating system is having problems. At any time you can hibernate the cluster by setting the ASG NodesPerZone value to 0. You can then restart the node by resetting the NodesPerZone to the previously set value. On restart, either by resuming from hibernate or restarting from the ASG detecting faults and restarting the server, the system will automatically do the following:

• Detect any previously attached volumes and re-attach them

• Detect if the hostname has changed since the previous start and, if so, rename the host to the new hostname in the MarkLogic cluster

• Re-attach to the cluster