System logs record information about administrative operations, system database activity, and on-premises connections. They encompass Tomcat logs, Web UI logs, system database logs, and service logs. Service logs themselves include a variety of separate logs which record everything from background threads to exceptions to authentication activity. The On-Premises Connector generates its own set of system logs apart from the Hybrid Data Pipeline server. However, On-Premises Connector system logs do not include Web UI or system database logs. See the following sections for details.

Important: The default locations for logs are given in the following sections. However, administrators may specify a centralized logs directory. For details, refer to Log management in the Progress DataDirect Hybrid Data Pipeline Deployment Guide.
Note: Data source logs are technically a type of service log. However, given their utility in diagnosing end-user issues, they are documented separately in Data source logging.

Tomcat logs

Hybrid Data Pipeline runs on an Apache Tomcat server. Therefore, a number of Tomcat logs are produced during the operation of Hybrid Data Pipeline. With a few differences, the Tomcat log files are also generated for the On-Premises Connector. The Tomcat log files are written to the following directories by default.

  • Hybrid Data Pipeline server:

    hdp_install_dir/ddcloud/logs/hostname/tomcat

    Note: For the Hybrid Data Pipeline server, the log location can be configured as described in Log management.
  • On-Premises Connector:

    opc_install_dir\OPDAS\server\logs

Tomcat logs on server and connector

  • catalina.datestamp.log

    The Catalina log file is a global log file which records server admin operations, such as server stop, server start, restart, application deployment, and subsystem failure. Messages written to the Tomcat system.err and system.out stream are recorded in this log file.

  • localhost_access_log..datestamp.txt

    The localhost access log records the HTTP operations that are being executed against the Hybrid Data Pipeline service. This log includes the actual API calls coming into the Tomcat server.

  • localhost.datestamp.log

    The localhost log tracks web application activity. It monitors HTTP transactions between the Tomcat application server and client applications.

  • host-manager.datestamp.log and manager.datestamp.log

    The host-manager and manager logs record output from the Tomcat Manager App.

Tomcat logs for server only

  • catalina.out

    The Catalina out file is the system output log file which persists during the operation of the server. Servlet logs and standard error messages are written to this log file. This file may also contain thread dumps and uncaught exceptions.

Tomcat logs for connector only

  • DDCLoud_Connector.datestamp.log

    The DDCloud connector log records the initialization of the connectivity service on the On-Premises Connector.

  • hdpcloudopcon-stderr.datestamp.log

    This standard err log records standard error messages

  • hdpcloudopcon-stdout.datestamp.log

    This standard out log file includes servlet logs.

  • opacessor.datestamp.log

    The on-premises accessor log records details about communications between the On-Premises Connector and the Hybrid Data Pipeline server. If a problem occurs where the On-Premises Connector is unable to communicate with the Hybrid Data Pipeline server, this log may help identify the issue.

Web UI logs

The Web UI logs record operations handled by the Web UI. These may include establishing an SSL connection, invoking an IP address whitelist, authenticating users, and handling queries. The name of the Web UI log is hdpui.log. The Hybrid Data Pipeline server writes Web UI log files to the following directory by default. The log location can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/hdpui

This directory includes the current hdpui.log file and past logs in a compressed file with a datestamp.

Notification server logs

The notification server handles communication between Hybrid Data Pipeline and the On-Premises Connector. Notification server logs are written to the following folder in the Hybrid Data Pipeline server installation directory by default. The log location can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/notification/palette

Notification server logs include the following files:

  • [com.ddtek.palette.notification.concurrent.socketreadtask].datestamp.log

    This log records the connection requests made from the Hybrid Data Pipeline server to an On-Premises Connector to confirm the availability of the On-Premises Connector.

  • [messagequeue].datestamp.log

    The message queue log records notification server statistics, such as the number of messages received by the notification server and the number of messages fetched by the notification server.

  • [SYSTEM].datestamp.log

    This log records registration key threads for secure communication between the Hybrid Data Pipeline server and its components.

  • [tenant][user_account].datestamp.log

    where tenant is the is the Hybrid Data Pipeline tenant, and user_account is the Hybrid Data Pipeline user account.

    This log records registration key threads for a particular user in a particular tenant.

Service log files

Service logs include several log files that record activity related directly to the operation of the Hybrid Data Pipeline service, whether the service is running on a Hybrid Data Pipeline server or an On-Premises Connector. These files are written to the following directories by default.

  • Hybrid Data Pipeline server:

    hdp_install_dir/ddcloud/logs/hostname/das

    Note: The log location can be configured as described in Log management.
  • On-Premises Connector:

    opc_install_dir\OPDAS\server\logs\das

Unless otherwise noted, the service logs are written to the Hybrid Data Pipeline server and the On-Premises Connector.

  • [background].datestamp.log

    The background log records events from the background threads in Hybrid Data Pipeline.

  • clouddb.datestamp.log

    The clouddb log records exceptions from non-relational data sources, such as Google BigQuery and Salesforce.

  • das-monitor.datestamp.log

    Server only. The das-monitor log records system statistics every 60 seconds.

  • ddcloud.datestamp.log

    The ddcloud log records initialization and shutdown of the servlet.

  • extauth.datestamp.log

    Server only. The external authentication log records the activity of external authentication services.

  • [filter].datestamp.log

    This log records activity related to filters, such as the authentication filter, the IP address whitelist filter, and the CORS filter.

  • [messaging].datestamp.log

    The messaging log records activity related to the internal message queue.

  • meter.datestamp.log

    The meter log captures metering information if the Hybrid Data Pipeline service has issues persisting meter information.

  • [tenant][user_account][data_store][data_source].datestamp.log

    This is a data source log. For details, see Data source logging.

  • onpremise.datestamp.log

    Server only. The on-premises log records activity related to making on-premises connections using the On-Premises Connector. Log entries include the connector ID, the connector IP address, and connection requests.

  • [system].datestamp.log

    This log file captures any runtime logging events that cannot be associated with a user.

Shutdown log file

The shutdown.log file records shutdown requests and processes associated with shutting down the Hybrid Data Pipeline server. The shutdown.log file is included with the log file bundle when obtaining system logs through the Web UI, the API, or the getlogs.sh script. The shutdown.log file is written to the following directory by default. The log location can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/tomcat