Data source logs record user activity against data sources. This information may include SQL queries or operations related to data source configuration. The Hybrid Data Pipeline server generates data source logs for all data sources across the system. In contrast, the On-Premises Connector generates data source log files only for the data sources it connects to. The same general description for data source logs applies to the logs whether generated by the server or the On-Premises Connector.

Important: The default locations for logs are given in the following sections. However, administrators may specify a centralized logs directory. For details, refer to Log management in the Progress DataDirect Hybrid Data Pipeline Deployment Guide.

Hybrid Data Pipeline Server Data Source Logs

The Hybrid Data Pipeline server writes data source logs to the Hybrid Data Pipeline server installation directory. When running the server on multiple nodes behind a load balancer, data source logs are written to each node in the deployment. It should be noted that the logs for a specific data source may be written to multiple nodes. Therefore, the data source logs from each node should be retrieved to review all the activity against the data source.

Note: You may retrieve the data source logs from individual nodes or across the system with the Web UI. Alternatively, you can obtain the logs from individual nodes using the Data Sources API or the getdslogs.sh script. See Obtaining data source logs for details.

The logs are written to the following location by default. Note that the log location can be configured as described in Log management.

hdp_install_dir/ddcloud/logs/hostname/das

Note: There are additional logs written to the this directory. For information about these and other logs, see System logs.

The data source log file names have the following format:

[tenant][user_account][data_store][data_source].datestamp.log

where:

tenant is the Hybrid Data Pipeline tenant. This could be the system tenant or some other tenant created by an administrator.

user_account is the Hybrid Data Pipeline user account which is connecting to the data source.

data_store is the data store, such as Db2, Oracle, or Salesforce, on which the data source was created.

data_source is the name of the data source that was created to connect to the data store.

datestamp is the date on which the activity was logged.

On-Premises Connector Data Source Logs

The On-Premises Connector generates data source log files for the data sources it connects to. On-Premises Connector data source logs are written to the On-Premises Connector installation directory. Unlike the server data source logs, these logs must be retrieved manually. However, the logs the server generates for data sources using an On-Premises Connector convey nearly the same information as the logs generated by the On-Premises Connector. So, depending on the issue, it may not be necessary to retrieve data source logs from the server and the On-Premises Connector.

The On-Premises Connector data source logs are written to the following directory:

opc_install_dir\OPDAS\server\logs\das

Note: There are additional logs written to the this directory. For information about these and other logs, see System logs.

The On-Premises Connector logs follow similar naming conventions to those of the server logs. However, they do not include the tenant name prefix and begin with the user account:

[user_account][data_store][data_source].datestamp.log

Data source log description

Data source logs record the data source activity of specific users. They capture the queries and operations sent by users that are coming into Hybrid Data Pipeline. You may configure Hybrid Data Pipeline to record more or less information, as described in Setting data source logging levels. However, regardless of the level of detail, every log entry will begin with the same elements as shown in this example:

08-Sep-2017 07:11:54.493 CONFIG [http-bio-8080-exec-12][steve@abctestmail.com]
   [salesforce][d2c_salesforce_odatav4][aYDHNkfB6Fd4mCk3].[execute][success=true][ms=82]
   [stmtId=1][bytesIn=2][bytesOut=1861][worker=Worker-1][rowsFetched=0]

The following table can be used to interpret the information contained in data source logs.

Table 1. Data source log elements
Element Example Description
Date and time stamp 08-Sep-2017 07:11:54.493 UTC date time value for when the logging event was written.
Log Level CONFIG The Java logging level associated with the event.
Thread Name http-bio-8080-exec-12 The name of the thread that logged the event.
User Name steve@abctestmail.com The name of the user.
Data Source Name salesforce The name of the data source.
Session Token aYDHNkfB6Fd4mCk3 The session identifier assigned to the user.
Operation Context execute The operational context in which the event occurred. If a Tomcat servlet thread, this will identify the command. Other types of operations include: login, logout, upload, clear, and continue. A worker value indicates the operation is being done asynchronously in a worker thread. This is only done as part of an execute request.
Message success=true ms=82 The rest of the line (or lines) will be the actual log message. Most messages are just key value pairs. Most messages include a success flag. When the flag is false, an error event message will usually proceed the message. The ms key gives the duration of the operation in milliseconds.