This guide provides instructions for running Hybrid Data Pipeline as a Docker container on a trial basis. A trial Docker deployment uses an internal database which allows you to deploy a working instance of Hybrid Data Pipeline on a single node. A trial deployment may be used for a 30-day evaluation period. It is not intended to be used in a production environment, and it may not be migrated to a load balancer or cluster environment.

Attention: For information on deploying to production and for complete deployment configuration information, refer to the Hybrid Data Pipeline Deployment Guide.

Performing a trial Docker deployment

Prerequisite: Before you proceed with deploying the Docker image, Docker must be installed on the host machine. You will need to load and run the image to deploy. For download and installation, refer to "Install Docker Engine" in Docker Docs: https://docs.docker.com/engine/install/.

Take the following steps to deploy Hybrid Data Pipeline on a trial basis.

  1. Download the Docker image from the DataDirect Connectors Download page.
    Note: The deployment package is available as a tar.gz file. For example: PROGRESS_DATADIRECT_HDP_SERVER_version_DOCKER.tar.gz, where version is a three-part version nummber denoting major, minor, and service pack numbers.
  2. Unzip the Hybrid Data Pipeline Docker Deployment Package to a package directory.

    The hdp-docker-deploy directory includes the following folders and files.

    • demo: The demo folder includes example batch files to demonstrate how deployment may be automated for a number of deployment scenarios.
    • hdp: The hdp folder includes hdpdeploy.properties that is used to set deployment configuration options.
    • hdp_docker_version.build.tar.gz: This is the Hybrid Data Pipeline Docker image file, where version is the three-part version number of the server and build is the server build number.
    • hdp-docker-deploy-readme.txt: The readme includes general information and a detailed directory structure of the deployment package.
  3. Run the docker load command to load the Docker image. For example: 
    docker load -i hdp_docker_version.build.tar.gz

    where version is the three-part version number of the server, and build is the server build number.

  4. Create a shared file location on the local file system. This shared file location will be mounted to the container and will be used in the deployment and operation of the server.

    Windows

    For Docker on Windows, you may simply create the shared file location on the host. For example:

    C:\hdpshare

    Linux

    For Docker on Linux, you must create the shared file location, create a non-root user, and then change the ownership of the shared file location directory to the non-root user. For step-by-step instructions, see Creating the shared file location on Linux.

    Note:
    • The shared file location must be mapped to the local file system when you execute the docker run command to deploy the image.
    • If running Docker on Windows in Hyper-V mode, the drive that has the directory to be mounted must be marked as a shared drive in the Docker settings.
  5. Follow the steps provided for your preferred configuration.
    • Standard configuration. Choose this option if you want to deploy a trial instance with a self-signed SSL certificate, and you do not require connectivity to an on-premises data source. Continue to Step 6.
    • SSL configuration. Choose this option if you need to use an SSL certificate issued by a certificate authority. Continue to Step 7.
      Note: If you are using Hybrid Data Pipeline to access external data with Salesforce Connect, a TLS certificate from a well-known certificate authority must be provided.
    • On-premises configuration. Choose this option if you need to connect to on-premises data. Continue to Step 8.
      Note: Hybrid Data Pipeline supports connectivity to on-premises data stores with the On-Premises Connector. For details on accessing on-premises data sources, refer to On-Premises Connector deployment configuration.
    Important:
    • The HDP_EVAL environment variable must be set to true in the docker run command. For example: -e "HDP_EVAL=true".
    • By default, the system database is located in the Docker container for a trial deployment. However, the deployment may be configured such that the system database is persisted in the shared file location by setting HDP_HSQL_INTERNAL to no as an environment variable in the docker run command. For example: -e "HDP_HSQL_INTERNAL=no".
  6. For a standard configuration, execute the following docker run command. Then, continue to Step 9.

    Standard configuration sample run command

    docker run -dt -p 8443:8443 -e "HDP_EVAL=true" -e "ACCEPT_EULA=true" -e "HDP_ADMIN_PASSWORD=AdminSecret" -e "HDP_USER_PASSWORD=UserSecret" -v /home/users/username/hdpshare:/hdpshare --hostname DockerHost --name ContainerName hdp-docker-version:tag
    Note: See Docker arguments for descriptions of these arguments.
  7. For SSL configuration, take the following steps.
    1. Create a PEM file with the full certificate chain. Refer to The PEM file for details.
    2. Copy the PEM file to the shared file location.
    3. Execute the following docker run command. Then, continue to Step 9.
      Note: You must specify the PEM file with the HDP_NODE_CERT_FILE environment variable in the docker run command. For example: -e "HDP_NODE_CERT_FILE=cacert.pem".

    SSL configuration sample run command

    docker run -dt -p 8443:8443 -e "HDP_EVAL=true" -e "ACCEPT_EULA=true" -e "HDP_ADMIN_PASSWORD=AdminSecret" -e "HDP_USER_PASSWORD=UserSecret" -e "HDP_NODE_CERT_FILE=CertName.pem" -v /home/users/username/hdpshare:/hdpshare --hostname DockerHost --name ContainerName hdp-docker-version:tag
    Note: See Docker arguments for descriptions of these arguments.
  8. For on-premises configuration, execute the following docker run command. Then, continue to Step 9.
    Note: To connect to on-premises data, you must enable the On-Premises Connector and map On-Premises Connector ports in the docker run command.

    On-premises configuration sample run command

    docker run -dt -p 8443:8443 -p 40501:40501 -p 11280:11280 -p 11443:11443 -e "HDP_EVAL=true" -e "ACCEPT_EULA=true" -e "HDP_ADMIN_PASSWORD=AdminSecret" -e "HDP_USER_PASSWORD=UserSecret" -e "HDP_ENABLE_ONPREMISE_CONNECTOR=yes" -v /home/users/username/hdpshare:/hdpshare --hostname DockerHost --name ContainerName hdp-docker-version:tag
    Note: See Docker arguments for descriptions of these arguments.
  9. Verify the deployment by accessing the Hybrid Data Pipeline Web UI from a browser. The server will be accessible via the Docker container hostname. For example:

    https://dockerhost.example.com:8443/hdpui

Deploying additional components

After you deploy Hybrid Data Pipeline as a Docker container, you might need to deploy one or more components to extend the service, depending on the requirements of your application environment. Refer to the following resources for details.

Note: A separate component is not required for OData connectivity. OData connectivity is enabled during the process of creating a Hybrid Data Pipeline data source. For details, refer to Create an OData Data Source.

When you deploy the Docker image, four configuration and certificate files are generated. These files are written to the redist directory in the shared file location. These files must be copied to the directory from which the component installation program will be run.

The four configuration and certificate files are:

  • config.properties
  • OnPremise.properties
  • ddcloud.pem
  • ddcloudTrustStore.jks
Note: If the Docker host port numbers are different than the default port numbers used in the Docker container, you must update these port numbers in the OnPremise.properties file when installing the On-Premises Connector, the ODBC driver, or the JDBC driver.

Using Hybrid Data Pipeline

After the deployment of Hybrid Data Pipeline and any additional components, you may begin to tailor your Hybrid Data Pipeline environment, provision users, and create data sources. Refer to the following resources for details.

For additional documentation resources, refer to the Hybrid Data Pipeline documentation page.