Hybrid Data Pipeline supports whitelisting IP addresses at runtime through the Web UI and API, as described in Implementing IP address whitelists in the User's Guide. Depending on your deployment scenario, load balancer and deployment configurations may be required to implement IP address whitelists. See the following sections for details on applicable scenarios.

Note:
  • When using either an AWS or Azure load balancer without an external proxy, additional configurations are not required.
  • While configuring IP address whitelists at runtime offers a measure of security, networks may still be vulnerable to IP spoofing. Therefore, you should consider adopting best practices with respect to managing and monitoring network traffic. For details, see Network best practices for mitigating IP spoofing.

HAProxy load balancer without an external proxy

In a deployment using an HAProxy load balancer without an external proxy1, you must add the following http-request directive to the frontend section of the HAProxy configuration file.

http-request set-header X-Forwarded-For %[src]

When an HTTP request is received, this directive sets the X-Forwarded-For header to the IP address of the client that made the request. This IP address is checked against the Hybrid Data Pipeline IP address whitelist.

After modifying the HAProxy configuration file, you must restart the HAProxy load balancer.

For more information about deploying Hybrid Data Pipeline with HAProxy, see Configuring a network load balancer with the On-Premises Connector.

HAProxy load balancer with an external proxy

In a deployment using an HAProxy load balancer with an external proxy1, you must run a silent installation to whitelist the IP addresses of the HAProxy load balancer and the external proxy. In turn, HAProxy must be configured and restarted. Take the following steps to configure your environment.

  1. Whitelist the IP addresses of the HAProxy load balancer and the external proxy IP addresses by running a silent installation or upgrade.

    In the silent installation response file, you must provide the IP addresses of the HAProxy load balancer and the external proxy with the D2C_REMOTE_IP_INTERNAL_PROXIES property. For example, if the IP addresses of the load balancer and proxy are 10.71.4.217 and 172.144.5.6, then you would include the following setting.

    D2C_REMOTE_IP_INTERNAL_PROXIES=10.71.4.217|172.144.5.6
  2. Add the following acl entry and http-request directive to the frontend section of the HAProxy configuration file.
    acl from_trusted_proxy src 172.144.5.6 
http-request set-header X-Forwarded-For %[src] if !from_trusted_proxy

    The acl entry specifies the IP address of the trusted proxy. The http-request directive sets the X-Forwarded-For header to the IP address of the client that made the request, if the request is from a source other than the trusted proxy.

  3. Restart the HAProxy load balancer.

For more information about silent installation and upgrade, see Silent installation process and Silent upgrade process.

For more information about deploying Hybrid Data Pipeline with HAProxy, see Configuring a network load balancer with the On-Premises Connector.

AWS deployment with an AWS load balancer and an external proxy

In an AWS deployment that uses an AWS load balancer with an external proxy1, you must perform a silent installation to whitelist the IP addresses of the external proxy, the AWS load balancer, and any internal AWS proxies.

In the silent installation response file, you must specify the IP addresses with the D2C_REMOTE_IP_INTERNAL_PROXIES property. For example, take the following configuration:

  • External proxy IP address: 20.71.4.217
  • AWS load balancer CIDR block: 203.0.113.0/24 (regex: 203\.0\.113\.\d{1,3})
  • AWS virtual network CIDR block: 25.174.56.0/24 (regex: 25\.174\.56\.\d{1,3})

In this configuration, the external proxy IP address is static. However, the load balancer and the virtual network use dynamic IP addresses which are associated with a range of IP addresses as indicated by the CIDR block. To ensure that all IP addresses in the given range are whitelisted, you must use a regular expression to specify the range in the D2C_REMOTE_IP_INTERNAL_PROXIES property. Based on the example configuration, you would provide the following setting in the silent installation response file:

D2C_REMOTE_IP_INTERNAL_PROXIES=20.71.4.217|203\.0\.113\.\d{1,3}|25\.174\.56\.\d{1,3}

For more information about silent installation and upgrade, see Silent installation process and Silent upgrade process.

For more information about cloud load balancer configurations for Hybrid Data Pipeline deployments, see Configuring a network load balancer with the On-Premises Connector.

Azure deployment with an Azure load balancer and an external proxy

In an Azure deployment that uses an Azure load balancer with an external proxy1, you must perform a silent installation to whitelist the IP addresses of the external proxy, the Azure load balancer, and any internal Azure proxies.

In the silent installation response file, you must provide the IP addresses with the D2C_REMOTE_IP_INTERNAL_PROXIES property. For example, take the following configuration:

  • External proxy IP address: 30.71.4.217
  • Azure load balancer CIDR block: 303.0.113.0/24 (regex: 303\.0\.113\.\d{1,3})
  • Azure virtual network CIDR block: 35.174.56.0/24 (regex: 35\.174\.56\.\d{1,3})

In this configuration, the external proxy IP address is static. However, the load balancer and the virtual network use dynamic IP addresses which are associated with a range of IP addresses as indicated by the CIDR block. To ensure that all IP addresses in the given range are whitelisted, you must use a regular expression to specify the range in the D2C_REMOTE_IP_INTERNAL_PROXIES property. Based on the example configuration, you would provide the following setting in the silent installation response file:

D2C_REMOTE_IP_INTERNAL_PROXIES=30.71.4.217:\d{1,5}|303\.0\.113\.\d{1,3}|35\.174\.56\.\d{1,3}
Note: The regex \d{1,5} indicates the allowable range of the external proxy port number. This port number may not be required in all scenarios.

For more information about silent installation and upgrade, see Silent installation process and Silent upgrade process.

For more information about cloud load balancer configurations for Hybrid Data Pipeline deployments, see Configuring a network load balancer with the On-Premises Connector.

Docker deployment with a load balancer and an external proxy

When deploying Hybrid Data Pipeline as a Docker container with a load balancer and an external proxy1, you must whitelist the load balancer and proxy IP addresses with the HDP_REMOTE_IP_INTERNAL_PROXIES property in the hdpdeploy.properties file. For example, if the load balancer IP is 40.71.4.217 and the external proxy IP is 472.144.5.6, then you would include the following setting.

HDP_REMOTE_IP_INTERNAL_PROXIES=40.71.4.217|472.144.5.6

For more information about Docker deployments, see Deploying Hybrid Data Pipeline using Docker.

1 In this context, the term external proxy refers to a proxy that sits between the client and the load balancer that serves the Hybrid Data Pipeline deployment.