Powered by Zoomin Software. For more details please contactZoomin

Progress Data Platform Help

Import data with pipelines

  • Last Updated: November 19, 2026
  • 5 minute read
    • Documentation

Use a pipeline to import content from an external data source into a dataset. Pipelines are useful when your content lives in another system, or when you want to bring in new content on a regular schedule instead of uploading files manually.

To import data with a pipeline, you need to:

  1. Configure a data source
  2. Create a pipeline
  3. Run the pipeline
  4. Review executions

Before you begin

  • Create the project and the destination dataset that should receive the content. See Create projects and Create datasets for more information.
  • Have the connection details or credentials for your source system available. The required fields depend on the data source type.

1. Configure a data source

A data source is a saved connection to an external system, such as a storage bucket or database. Data sources are configured at the account level, so pipelines in different projects can use the same data source.

If the data source that you need already exists, skip to the next step. For more information about data source types, testing, editing, and sharing, see Manage data sources.

To configure a data source:

  1. Select Data Sources from the sidebar menu. The Data Sources view opens.
  2. Click New Data Source. The Connect Source window opens.
  3. Select the data source type to connect to.
  4. Provide values for the following fields:
    • Title: A name for the data source. For example, Production S3 Bucket.
    • (Optional) Description: A description of the data source.
    • The connection fields for the selected type, such as the location of the content and the credentials to access it. Required fields are marked with an asterisk. See Data source options for details.
    • For file-based sources, also set the file type and read options. See File types and read options. Required fields are marked with an asterisk.
  5. Click Connect.

To confirm that Progress Data Platform can reach the source, open the data source and click Test Connection.

Note: You can also create a data source while you create a pipeline. In the pipeline's Data Source field, select Create New Data Source.

2. Create a pipeline

Pipelines connect one data source to one dataset in a project. A pipeline can run manually or on a schedule. You create and manage a pipeline from its destination dataset.

To create a pipeline:

  1. Open your project, and then select Datasets from the sidebar menu.

  2. Select the dataset that should receive the content.

  3. Select the Pipelines tab.

  4. Click New Pipeline. The New Pipeline window opens.

  5. Provide values for the following fields:

    • Title: A name for the pipeline. For example, Daily GCS Sync.
    • (Optional) Description: A description of the pipeline.
    • Data Source: The data source to import data from. To add a new one, select Create New Data Source.

    Note: The Dataset field displays dataset that receives the imported content. This field is set to the current dataset and cannot be changed.

  6. Choose when the pipeline runs:

    • To run the pipeline only when you start it, clear Run on a schedule.
    • To run the pipeline automatically, select Run on a schedule, and then select a frequency in the Schedule field. For daily and weekly schedules, also set the Time of day. For weekly schedules, set the Day of week.
  7. Click Create Pipeline.

The view for your new pipeline opens.

Note: Imported resources are processed using the enrichment settings of the destination dataset. Review the dataset's Enrichments tab before you run the pipeline. See Enrichments.

3. Run the pipeline

To run pipeline, open your the dataset for your pipeline and select the select the Pipelines tab. Then, click the card for your pipeline to open your pipeline view.

You can run the pipeline using the following methods:

  • Manual run: Open your pipeline view and click Run Pipeline. You can also click Run now on the pipeline's card in the dataset's Pipelines tab.
  • Scheduled run: In your pipeline view, click Edit. Then, enable Run on a schedule and specify the Schedule and Time of day.

Each pipeline run adds new resources and updates existing resources. It does not delete resources that are no longer present in the data source.

Notes:

  • A scheduled pipeline runs automatically at its scheduled time; however, you can also run it manually at any time.
  • Wait for a run to finish before you start another one.

4. Review executions

An execution is single run of a pipeline. Each execution records its status and how many resources were imported, skipped, or failed.

For more information about execution statuses and errors, see Review ingestion executions.

Each run appears in the pipeline's Execution History, which shows when the run started and finished, how long it took, and its status. Select Only show failures to list only failed runs.

Each execution has one of the following statuses:

Status Meaning
In Progress The execution is still running.
Completed The execution finished. Review its counts to confirm what was imported.
Completed With Errors The execution finished, but one or more resources failed.
Failed The execution did not complete.

To see details, click an execution. The details show how many resources succeeded, were skipped, or had errors.

  • For a Failed execution, review the Failure Message.
  • For an execution with resource errors, review the Resource Errors pane. Resource Errors lists the URI and error message for each failed item.

After an execution finishes, open the destination dataset's Explorer tab to search and inspect the imported resources.

Manage a pipeline

From the view for your pipeline, you can also perform the following actions:

  • Edit pipeline setttings: Click Edit to change the pipeline's settings, such as its title, data source, or run schedule. You cannot change a pipeline's dataset. To import into a different dataset, create a pipeline in that dataset.
  • Delete your pipeline: Click Delete to remove the pipeline.

Troubleshooting

The following table describes common issues you might encounter when importing data with a pipeline and potential solutions.

Issue What to do
A data source is missing from the list Confirm that the data source was saved. You can also create one from the Data Source field.
I can't find a pipeline Pipelines are listed in their destination dataset. Open the dataset and select the Pipelines tab.
The execution failed Review the Failure Message. If it points to the connection, open the data source, check its settings and credentials, and click Test Connection.
The execution completed with errors Review Resource Errors to find the affected items, correct the reported issue, and then run the pipeline again.
Imported content does not appear in the dataset Confirm that the execution finished and review its counts. Then, check the dataset's Explorer tab. Resources might still be processing.
A scheduled pipeline did not run Confirm that Run on a schedule is selected and review the schedule in the pipeline's settings. You can run the pipeline manually in the meantime.
Alert