Data Hub Gradle Tasks

Data Hub Gradle Tasks

A complete list of all of the Gradle tasks available in Data Hub Gradle Plugin (ml-data-hub).

Using Gradle in Data Hub

To use Data Hub Gradle Plugin in the Data Hub flows, see Data Hub Gradle Plugin.

To pass parameters to Gradle tasks, use the -P option.

Unix systems

./gradlew taskname ... -PparameterName=parameterValue ... -i

Windows

gradlew.bat taskname ... -PparameterName=parameterValue ... -i

You can use Gradle's -i option to enable info-level logging.

This page provides the list of Gradle tasks available in Data Hub Gradle Plugin (ml-data-hub).

• Tasks with names starting with ml are customized for Data Hub from the ml-gradle implementation.
• Tasks with names starting with hub are created specifically for Data Hub.

• MarkLogic Data Hub Setup Tasks
• MarkLogic Data Hub Scaffolding Tasks
• MarkLogic Data Hub Flow Management Tasks
• MarkLogic Data Hub Uninstall Tasks
• Legacy (DHF 4.x) Tasks

MarkLogic Data Hub Setup Tasks

These tasks are used to configure MarkLogic Data Hub and manage the data hub.

mlDeploy

Uses hubPreinstallCheck to deploy your Data Hub project.

Unix systems

./gradlew mlDeploy -i

Windows

gradlew.bat mlDeploy -i

mlWatch

Extends ml-gradle's WatchTask by ensuring that modules in Data Hub-specific folders (plugins and entity-config) are monitored.

Unix systems

./gradlew mlWatch -i

Windows

gradlew.bat mlWatch -i

mlUpdateIndexes

Updates the properties of every database without creating or updating forests. Many properties of a database are related to indexing.

Unix systems

./gradlew mlUpdateIndexes -i

Windows

gradlew.bat mlUpdateIndexes -i

hubUpdate

Updates your Data Hub instance to a newer version.

Unix systems

./gradlew hubUpdate -i

Windows

gradlew.bat hubUpdate -i

Before you run the hubUpdate task, edit the build.gradle file. Under plugins, change the value of 'com.marklogic.ml-data-hub' version to the new Data Hub version.

For example, if you are updating to Data Hub 5.0.4,

 plugins {
  id 'com.marklogic.ml-data-hub' version '5.0.4'
}

For complete instructions on upgrading to a newer Data Hub version, see Upgrading Data Hub.

Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

hubInfo

Prints out basic info about the Data Hub configuration.

Unix systems

./gradlew hubInfo -i

Windows

gradlew.bat hubInfo -i

hubDeployUserArtifacts

Installs user artifacts, such as entities and mappings, to the MarkLogic server. (Data Hub 4.2 or later)

Unix systems

./gradlew hubDeployUserArtifacts -i

Windows

gradlew.bat hubDeployUserArtifacts -i

MarkLogic Data Hub Scaffolding Tasks

These tasks allow you to scaffold projects, entities, flows, and steps.

hubInit

Initializes the current directory as a Data Hub project.

Unix systems

./gradlew hubInit -i

Windows

gradlew.bat hubInit -i

hubCreateEntity

Creates a boilerplate entity.

Unix systems

./gradlew hubCreateEntity -PentityName=YourEntityName -i

Windows

gradlew.bat hubCreateEntity -PentityName=YourEntityName -i

Parameters

entityName

   (Required) The name of the entity to create.

hubCreateFlow

Creates a boilerplate flow definition file.

Unix systems

./gradlew hubCreateFlow -PflowName=YourFlowName -i

Windows

gradlew.bat hubCreateFlow -PflowName=YourFlowName -i

Parameters

flowName

   (Required) The name of the flow to create.

hubCreateStepDefinition

Creates a custom step definition that can be added to a flow as a step.

Unix systems

./gradlew hubCreateStepDefinition -PstepDefName=yourstepname -PstepDefType=yoursteptype -Pformat=[sjs|xqy] -i

Windows

gradlew.bat hubCreateStepDefinition -PstepDefName=yourstepname -PstepDefType=yoursteptype -Pformat=[sjs|xqy] -i

Parameters

stepDefName

   (Required) The name of the custom step definition to create.

stepDefType

   The type of the step definition to create: ingestion, mapping, mastering, or custom. Default is custom.

format

   The format of the module to associate with the new step definition: xqy for XQuery or sjs for JavaScript. Default is sjs.

A JavaScript module (main.sjs) is created and associated with the step definition to perform the processes required for the step.

• If -Pformat=sjs or if the option is not specified, only the main.sjs file is created, and it contains the processes required for the step.
• If -Pformat=xqy, two files are created:
  • lib.xqy, which is the XQuery module that you must customize. It contains the processes required for the step; for example, custom code to create an envelope.
  • main.sjs, which acts as a wrapper around lib.xqy.

These modules can be found under your-project-root/src/main/ml-modules.

hubGeneratePii

Generates security configuration files for protecting entity properties designated as Personally Identifiable Information (PII). For details, see Managing Personally Identifiable Information.

Unix systems

./gradlew hubGeneratePii -i

Windows

gradlew.bat hubGeneratePii -i

MarkLogic Data Hub Flow Management Tasks

These tasks allow you to run flows and clean up.

hubRunFlow

Runs a flow.

Unix systems

./gradlew hubRunFlow -PflowName=YourFlowName -PentityName=YourEntityName -PbatchSize=100 -PthreadCount=4 -PshowOptions=[true|false] -PfailHard=[true|false] -Psteps="1,2" -PjobId="abc123" [ -Poptions="{ customkey: customvalue, ... }" | -PoptionsFile=/path/to.json ] -i

Windows

gradlew.bat hubRunFlow -PflowName=YourFlowName -PentityName=YourEntityName -PbatchSize=100 -PthreadCount=4 -PshowOptions=[true|false] -PfailHard=[true|false] -Psteps="1,2" -PjobId="abc123" [ -Poptions="{ customkey: customvalue, ... }" | -PoptionsFile=/path/to.json ] -i

Parameters

flowName

   (Required) The name of the harmonize flow to run.

entityName

   (Required if the flow includes a mapping step) The name of the entity used with the mapping step.

batchSize

   The number of items to include in a batch. Default is 100.

threadCount

   The number of threads to run. Default is 4.

showOptions

   If true, options that were passed to the command are printed out. Default is false.

failHard

   If true, the flow's execution is ended immediately if a step fails. Default is false.

steps

   The comma-separated numbers of the steps to run. If not provided, the entire flow is run.

jobId

A unique job ID to associate with the flow run. This option can be used if the flow run is part of a larger process (e.g., a process orchestrated by NiFi with its own job/process ID). Must not be the same as an existing Data Hub job ID. If not provided, a unique Data Hub job ID will be assigned.

options

   A JSON structure containing key-value pairs to be passed as custom parameters to your step modules.

optionsFile

   The path to a JSON file containing key-value pairs to be passed as custom parameters to your step modules.

The custom key-value parameters passed to your step module are available through the $options (xqy) or options (sjs) variables inside your step module.

hubExportJobs

Exports job records. This task does not affect the contents of the staging or final databases.

Unix systems

./gradlew hubExportJobs -PjobIds=list-of-ids -Pfilename=export.zip -i

Windows

gradlew.bat hubExportJobs -PjobIds=list-of-ids -Pfilename=export.zip -i

Parameters

jobIds

   A comma-separated list of job IDs to export.

filename

   The name of the zip file to generated, including the file extension. Default is jobexport.zip.

hubDeleteJobs

Deletes job records. This task does not affect the contents of the staging or final databases.

Unix systems

./gradlew hubDeleteJobs -PjobIds=list-of-ids -i

Windows

gradlew.bat hubDeleteJobs -PjobIds=list-of-ids -i

Parameters

jobIds

   (Required) A comma-separated list of job IDs to delete.

MarkLogic Data Hub Uninstall Tasks

mlUndeploy

Removes all components of your data hub from the MarkLogic server, including databases, application servers, forests, and users.

Unix systems

./gradlew mlUndeploy -Pconfirm=true -i

Windows

gradlew.bat mlUndeploy -Pconfirm=true -i

Legacy (DHF 4.x) Tasks

hubCreateInputFlow

Creates a legacy (DHF 4.x) input flow. The resulting DHF 4.x flow must be executed using hubRunLegacyFlow.

Unix systems

./gradlew hubCreateInputFlow -PentityName=YourEntityName -PflowName=YourFlowName -PdataFormat=[xml|json] -PpluginFormat=[xqy|sjs] -i

Windows

gradlew.bat hubCreateInputFlow -PentityName=YourEntityName -PflowName=YourFlowName -PdataFormat=[xml|json] -PpluginFormat=[xqy|sjs] -i

Parameters

entityName

   (Required) The name of the entity that owns the flow.

flowName

   (Required) The name of the input flow to create.

dataFormat

xml or json. Default is json.

pluginFormat

xqy or sjs. The plugin programming language.

hubCreateHarmonizeFlow

Creates a legacy (DHF 4.x) harmonization flow. The resulting DHF 4.x flow must be executed using hubRunLegacyFlow.

Unix systems

./gradlew hubCreateHarmonizeFlow -PentityName=YourEntityName -PflowName=YourFlowName -PdataFormat=[xml|json] -PpluginFormat=[xqy|sjs] -PmappingName=yourmappingname -i

Windows

gradlew.bat hubCreateHarmonizeFlow -PentityName=YourEntityName -PflowName=YourFlowName -PdataFormat=[xml|json] -PpluginFormat=[xqy|sjs] -PmappingName=yourmappingname -i

Parameters

entityName

   (Required) The name of the entity that owns the flow.

flowName

   (Required) The name of the harmonize flow to create.

dataFormat

xml or json. Default is json.

pluginFormat

xqy or sjs. The plugin programming language.

mappingName

   The name of a model-to-model mapping to use during code generation.

hubRunLegacyFlow

Runs a (legacy) DHF 4.x harmonization flow.

Unix systems

./gradlew hubRunLegacyFlow -PentityName=YourEntityName -PflowName=YourFlowName -PbatchSize=100 -PthreadCount=4 -PsourceDB=data-hub-STAGING -PdestDB=data-hub-FINAL -PshowOptions=[true|false] -Pdhf.YourKey=YourValue -i

Windows

gradlew.bat hubRunLegacyFlow -PentityName=YourEntityName -PflowName=YourFlowName -PbatchSize=100 -PthreadCount=4 -PsourceDB=data-hub-STAGING -PdestDB=data-hub-FINAL -PshowOptions=[true|false] -Pdhf.YourKey=YourValue -i

Parameters

entityName

   (Required) The name of the entity containing the harmonize flow.

flowName

   (Required) The name of the harmonize flow to run.

batchSize

   The number of items to include in a batch. Default is 100.

threadCount

   The number of threads to run. Default is 4.

sourceDB

   The name of the database to run against. Default is the name of your staging database.

destDB

   The name of the database to put harmonized results into. Default is the name of your final database.

showOptions

   Whether to print out options that were passed in to the command. Default is false.

dhf.YourKey

   The value to associate with your key. These key-value pairs are passed as custom parameters to your flow. You can pass additional key-value pairs as separate options:

hubrunlegacyflow ... -Pdhf.YourKeyA=YourValueA -Pdhf.YourKeyB=YourValueB ...

The custom key-value parameters passed to your step module are available through the $options (xqy) or options (sjs) variables inside your step module.

See Also

Related concepts

• Data Hub Gradle Plugin
• Gradle Properties

Related tasks

• Set Up Gradle for Different Environments

Related information

• ml-gradle Common Tasks
• ml-gradle Task Reference