The Hybrid Data Pipeline server writes data source logs to the Hybrid Data Pipeline server installation directory. When running the server on multiple nodes behind a load balancer, data source logs are written to each node in the deployment. It should be noted that the logs for a specific data source may be written to multiple nodes. Therefore, the data source logs from each node should be retrieved to review all the activity against the data source. You may retrieve the data source logs from individual nodes or across the system with the Web UI. Alternatively, you may obtain the logs node-by-node using the Data Sources API or the getdslogs.sh script. Each of these methods for obtaining logs are explained in the sections below.

Data source log files are also generated by the On-Premises Connector for the data sources it connects to. On-Premises Connector data source logs are written to the On-Premises Connector installation directory. Unlike the server data source logs, these logs must be retrieved manually. However, the logs the server generates for data sources using an On-Premises Connector convey nearly the same information as the logs generated by the On-Premises Connector. So, depending on the issue, it may not be necessary to retrieve data source logs from the server and the On-Premises Connector. On-Premises Connector data source logs may be retrieved manually from the following On-Premises Connector installation directory:

opc_install_dir\OPDAS\server\logs\das

The following sections show the ways you can obtain data source logs on the Hybrid Data Pipeline server.

Obtain the logs with the Web UI

Permissions

To obtain the logs for a data source using the Web UI, the user must be the data source owner. This means that the user either created the data source, or an administrator created the data source on behalf of the user. In addition, the user must have either set of the following permissions.

  • Administrator (12) permission
  • MgmtAPI (11), ViewDataSource (2), WebUI (8), and at least one of the following query permissions:
    • UseDataSourceWithJDBC (5)
    • UseDataSourceWithODBC (6)
    • UseDataSourceWithOData (7)

Web UI steps

Take the following steps to download the logs for a data source.

  1. Navigate to the Data Sources view by clicking the data sources icon .
  2. Click the download icon for the data source for which you are downloading logs.

Result: A zip file containing the data source logs will be downloaded to your browser's download directory.

Obtain the logs with the Data Sources API

Permissions

With the required permissions, a user may obtain the logs for a data source that they own using the Data Sources API. In addition, an administrator with the appropriate permissions may obtain the logs for a data source on behalf of the data source owner.

Owner permissions

The owner of a data source must have either set of the following permissions to obtain the logs of a data source.

  • Administrator (12) permission
  • MgmtAPI (11), ViewDataSource (2), WebUI (8), and at least one of the following query permissions:
    • UseDataSourceWithJDBC (5)
    • UseDataSourceWithODBC (6)
    • UseDataSourceWithOData (7)

Administrator permissions

An administrator must have either set of the following permissions to obtain the logs for another user.

  • Administrator (12) permission
  • MgmtAPI (11), ViewDataSource (2), WebUI (8), OnBehalfOf (21) and at least one of the following query permissions:
    • UseDataSourceWithJDBC (5)
    • UseDataSourceWithODBC (6)
    • UseDataSourceWithOData (7)

API request by data source owner

If you are the owner of the data source, you may use the Data Sources API to obtain logs as shown in the following example. {DatasourceId} is the ID of the data source for which you are retrieving logs.

GET https://hdp.example.com/api/mgmt/datasources/{DatasourceId}/logs

Result: A zip file containing the data source logs will be downloaded to your browser's download directory.

Note: If allowed by your browser, you may enter the URL in the address field. You will be prompted for credentials. After you enter your credentials, the logs will be downloaded. If you are using an API tool, you may need to select a download option to ensure that the logs are downloaded. For example, if you are using the Postman GUI client, you must select Send and Download from the Send dropdown.

API request by administrator on behalf of the data source owner

If you are an administrator but not the owner of the data source, you may use the Data Sources API in combination with the on-behalf-of functionality to obtain the logs for the data source. In the following example, {DatasourceId} is the ID of the data source for which you are retrieving logs, and userName is the account ID of the data source owner.

GET https://hdp.example.com/api/mgmt/datasources/{DatasourceId}/logs?user=userName

Result: A zip file containing the data source logs will be downloaded to your browser's download directory.

Note: If allowed by your browser, you may enter the URL in the address field. You will be prompted for credentials. After you enter your credentials, the logs will be downloaded. If you are using an API tool, you may need to select a download option to ensure that the logs are downloaded. For example, if you are using the Postman GUI client, you must select Send and Download from the Send dropdown.

Use the getdslogs.sh script

Take the following steps to retrieve data source logs using the getdslogs.sh script.

  1. From the file manager of the host machine, navigate to install_dir/ddcloud.
  2. Run the getdslogs.sh script. The user_name and data_source_name arguments are required, where the user_name is the name of the user who owns the data source and data_source_name is the name of the data source. For example:
    getdslogs.sh HdpUser12 OracleDs3

    Result

    The script will generate a tar.gz file. The name of the file will have the format user_name-datasource_name-datetimestamp.tar.gz.

  3. In a cluster deployment, repeat Steps 1 and 2 for each node.