Hybrid Data Pipeline provides logging capabilities for each component to help administrators monitor the system and troubleshoot issues. The following sections summarize the logs for each of the different components and refer to additional resources for details.
Note: Hybrid Data Pipeline also supports correlated logging. Correlated logging or log correlation enables you to track and relate events across components and services in your application environment. See Correlated logging for details.
Important: Administrators can assign custom locations for generating the logs and set logging levels using the hdplogging.properties file. Hybrid Data Pipeline creates a hostname directory in the location you have specified with the hdpLoggingDir property. The default locations for the logs are given in the following sections. For details, refer to Log management in the Progress DataDirect Hybrid Data Pipeline Deployment Guide.

Hybrid Data Pipeline server

The logs associated with the Hybrid Data Pipeline server include deployment logs, data source logs, and system logs.

Deployment logs

Deployment logs record information about Hybrid Data Pipeline installation and deployment. Deployment log files are bundled with system log files, and they may be obtained with system log files through the Web UI, the API, or the getlogs.sh script.

When an installation directory is successfully created, the following logs are generated:

  • hdp_install_dir/_ProgressHDPServer_installation/Logs/Progress_DataDirect_Hybrid_Data_Pipeline_Server_Install_datetimestamp.log

  • hdp_install_dir/ddcloud/logs/hostname/install/deploy.log
  • hdp_install_dir/ddcloud/logs/hostname/install/error.log
  • hdp_install_dir/ddcloud/logs/hostname/install/final.log
Note: The log locations for the deploy.log, error.log, and final.log files can be configured as described in Log management.

Additionally, for Docker deployments, the stdout and stderr logs provide deployment information. The docker logs command may be used to view this output.

For a failed Linux installation, the following log is written to the machine's default temporary directory:

Progress_DataDirect_Hybrid_Data_Pipeline_Server_InstallFailed.txt

When a Docker deployment fails, any logs present at the time of failure are bundled into a compressed archive file named hdp_logs.datetimestamp.tar.gz and written to the shared file location.

For a Linux silent installation, the following logs are written to the user's home directory:

  • SilentInstallError.log
  • SilentInstallInfo.log (only generated when hostname and port validation are skipped)

SilentInstallInfo.log (only generated when hostname and port validation are skipped)

For information about deployment as it relates to the system database, the following log is written to the utility logs directory by default. The location for the log file can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/install/hdpCloudSchema.datestamp.log

For more information, refer to Deployment logs in the Deployment Guide.

Data source logs

Data source logs record user activity, including actual SQL statements, against data sources. These logs often help to uncover issues with queries. They may be obtained through the Web UI or the API. These log files are written to the server installation directory at the following location by default. The location for these log files can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/das

See Data source logging for more information.

Note: Hybrid Data Pipeline provides a separate SQL statement auditing feature. When SQL statement auditing is enabled, SQL statements and metrics are recorded in the system database. See SQL statement auditing for details.

System logs

System logs record information about administrative operations, system database activity, and on-premises connections. They encompass runtime logs, such as Tomcat logs, Web UI logs, and service logs. Service logs themselves include a number of log files that relate to the operation of the Hybrid Data Pipeline service. These logs may be obtained through the Web UI or the API. They are written to the following server installation directory by default. The location for these log files can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/das

See System logs for more information.

On-Premises Connector

Since the On-Premises Connector runs a local version of the Hybrid Data Pipeline service, On-Premises Connector logs also include deployment logs, data source logs, and system logs.

Important: On-Premises Connector logs may not be obtained using the Web UI or the API. These log files must be retrieved manually from the On-Premises Connector installation directory.

Deployment logs

When installed successfully, the following log is written to the On-Premises Connector installation directory:

opc_install_dir\install\logs\Progress_DataDirect_Hybrid_Data_Pipeline_On-Premise_Connector_-_64_bit.log

For a failed installation, the following log is written to the machine's default temporary directory:

On-Premise_Connector_InstallFailed.txt

See also On-Premises Connector installation log files in the Deployment Guide

Data source logs

Data source logs record user activity, including actual SQL statements, against data sources. These logs often help to uncover issues with queries. The data source logs written to the On-Premises Connector installation directory include only the logs for data sources that use the connector. These are written to the following location:

opc_install_dir\OPDAS\server\logs\das

Note: The Hybrid Data Pipeline server writes corresponding On-Premises Connector data source logs to the server installation directory. The logs the server generates convey nearly the same information as the logs generated by the On-Premises Connector. So, depending on the issue, it may not be necessary to retrieve data source logs from the server and the On-Premises Connector.

See Data source logging for more information.

System logs

System logs record information about the runtime operations of the On-Premises Connector. They include the On-Premises Connector Tomcat logs and service logs. Service logs themselves include a number of log files that relate to the operation of the connector. These log files are written to the server installation directory:

opc_install_dir\OPDAS\server\logs

See System logs for more information.

ODBC driver

Deployment logs

When installed successfully, the following log is written to the ODBC driver installation directory:

odbc_install_dir\install\logs\Progress_DataDirect_Hybrid_Data_Pipeline_for_ODBC_-_64_bit.log

For a failed installation, the following log is written to the machine's default temporary directory:

Progress_DataDirect_Hybrid_Data_Pipeline_for_ODBC_InstallFailed.txt

See also ODBC driver installation log files in the Deployment Guide.

Driver logs

The ODBC driver supports ODBC tracing to log ODBC function calls. For more information, see ODBC Trace and ODBC driver troubleshooting.

JDBC driver

Deployment logs

When installed successfully, the following log is written to the JDBC driver installation directory:

jdbc_install_dir\install\logs\Progress_DataDirect_Hybrid_Data_Pipeline_for_JDBC.log

For a failed installation, the following log is written to the machine's default temporary directory:

Progress_DataDirect_Hybrid_Data_Pipeline_for_JDBC_InstallFailed.txt

See also JDBC driver installation log files in the Deployment Guide.

Driver logs

The driver provides flexible and comprehensive logging through Java logging. For more information, see Java logging.