Hybrid Data Pipeline provides logging capabilities to help administrators monitor Hybrid Data Pipeline services and troubleshoot issues. Administrators can specify the location of a logs directory and configure log levels for several services using the hdplogging.properties file.

The logs directory is a central location for logs across a Hybrid Data Pipeline deployment. You may specify the location of the logs directory with the hdpLoggingDir property in the hdplogging.properties file. When deploying or restarting a Hybrid Data Pipeline server, Hybrid Data Pipeline creates a server-specific directory in the logs directory if one is not already present. The name of this directory is based on the hostname of the server. For example:

hdpLoggingDir/hybrid-data-pipeline.example.com

The hostname is the value of the MUSTANG_SERVER property in the update.properties file. The update.properties file is written to the hdp_install_dir/ddcloud directory during the installation of Hybrid Data Pipeline on a node.

Note:
  • The Hybrid Data Pipeline user must have write permissions for the logs directory.
  • For a cluster deployment, a server directory is created for each server node.
  • For versions 4.6.2 and later, the logs are generated to a centralized logs directory by default, even when no location is specified. If you upgraded from a previous version to 4.6.2, then you can view the previous log locations for reference.
  • The location of data source logs is determined by the hdpLoggingDir property in the update.properties file. However, data source logging levels are configured separately. For information on configuring data source logging levels, refer to Data source logging in the Hybrid Data Pipeline User's Guide.

Hybrid Data Pipeline also creates the following subfolders in the server-specific logs directory.

  • das - Service logs, including administrative and data source logs
  • database - Logs that record calls to the system database, including schema creation
  • hdpui - Logs that relate to the Web UI
  • heapdumps - Logs that capture system failures
  • install - Logs that relate to installation and deployment
  • notification - Logs that relate to notification and messaging services, including communications with On-Premises Connectors
  • tomcat - Catalina and other Apache Tomcat logs

Here is an example of the logs directory structure for a two-node cluster where hdpLoggingDir is the location of the logs directory and the value of the hdpLoggingDir property.

hdpLoggingDir
   host1.example.com
      das
      database
      hdpui
      heapdumps
      install
      notification
      tomcat
   host2.example.com
      das
      database
      hdpui
      heapdumps
      install
      notification
      tomcat