Using Gradle

Create and Add a Step to a Flow Using Gradle

Before you begin

You need:

• Java SE JDK 8
• MarkLogic Server (See Version Compatibility.)
• Gradle 4.6 or later

About this task

A flow must have at least one step. You can create steps in two ways:

• By customizing the example steps. If you created your flow using Gradle, the resulting flow definition file includes an example step for each predefined type of step (ingestion, mapping, mastering). Simply customize these example steps.
• By creating and customizing a step definition using Gradle. Use the Gradle task hubCreateStepDefinition to initialize a step definition, copy the step definition to the flow definition, and customize.

This task describes how to create and customize a step definition using Gradle.

Ingestion

1. Using Gradle, create the step definition.

   1. Open a command-line window, and go to your project root directory.

   2. At your project root, run the Gradle task hubCreateStepDefinition with the stepType as ingestion.

      Unix systems

      Copy

      ./gradlew hubCreateStepDefinition -PstepDefName=your-ingestion-step-name -PstepDefType=ingestion -i
      

      Windows

      Copy

      gradlew.bat hubCreateStepDefinition -PstepDefName=your-ingestion-step-name -PstepDefType=ingestion -i
      

      To associate an XQuery module (instead of the default JavaScript module) with the step definition, add the option -Pformat=xqy. This creates an XQuery sample module, which you can customize, and a JavaScript wrapper.

      Note: The default stepType is custom.

      Result:

      The following files are created:

      • The step definition file:
      Copy

      your-project-root/step-definitions/ingestion/your-ingestion-step-name/your-ingestion-step-name.step.json
      

      • The custom module file:
      Copy

      your-project-root/src/main/ml-modules/root/custom-modules/ingestion/your-ingestion-step-def-name/main.sjs
      

      The modulePath setting in the step definition points to the custom module file.

2. Manually add the step to the flow.

   1. In a text editor, open the step definition file.

      The custom ingestion step definition file contains the following:

      Copy

         {
          "language" : "zxx",
          "name" : "your-ingestion-step-def-name",
          "description" : null,
          "type" : "INGESTION",
          "version" : 1,
          "options" : {
            "collections" : [ "your-ingestion-step-def-name" ],
            "outputFormat" : "json",
            "targetDatabase" : "data-hub-STAGING"
          },
          "customHook" : { },
          "modulePath" : "/custom-modules/ingestion/your-ingestion-step-def-name/main.sjs",
          "retryLimit" : 0,
          "batchSize" : 100,
          "threadCount" : 4,
          "fileLocations" : {
            "inputFilePath" : "",
            "outputURIReplacement" : "",
            "inputFileType" : ""
          }
        }
      
      

   2. Likewise, open the flow definition file.

      You can find your flow definition file in your-project-root/flows.

      The default flow definition file without any steps contains the following:

      Copy

         {
          "name": "your-flow-name",
          "description": "",
          "batchSize": 100,
          "threadCount": 4,
          "options": {
            "sourceQuery": null
          },
          "steps": {}
        }
      
      

   3. In the steps node, add the step as a key-value pair.

      • For the key, enter a string containing a number which represents the order of the step in the sequence.

        Note: The steps can be listed in any order, as long as the keys are unique within the steps node of the flow. Duplicate keys can produced unexpected results. The key number must be greater than 0.

      • For the value, copy and paste the entire content of the step definition file.

   4. Edit the step in the flow definition file.

      • Rename the name setting to stepDefinitionName.

      • Rename the type setting to stepDefinitionType.

      • Create a new name setting and assign it a name for the step.

      • (Optional) Delete the following settings:

        • language
        • version
        • modulePath

        The values for these settings are retrieved from the step definition.

   After adding a default ingestion step to the default flow as the first step ("1"), the flow definition file looks as follows:

   Copy

   {
       "name": "your-flow-name",
       "description": "",
       "batchSize": 100,
       "threadCount": 4,
       "options": {
         "sourceQuery": null
       },
       "steps": {
         "1": {
           "name" : "your-step-name",
           "stepDefinitionName" : "your-ingestion-step-def-name",
           "description" : null,
           "stepDefinitionType" : "INGESTION",
           "options" : {
             "collections" : [ "your-ingestion-step-name" ],
             "outputFormat" : "json",
             "targetDatabase" : "data-hub-STAGING"
           },
           "customHook" : { },
           "retryLimit" : 0,
           "batchSize" : 100,
           "threadCount" : 4,
           "fileLocations" : {
             "inputFilePath" : "",
             "outputURIReplacement" : "",
             "inputFileType" : ""
           }
         }
       }
   }
   

Mapping

1. Using Gradle, create the step definition.

   1. Open a command-line window, and go to your project root directory.
   2. At your project root, run the Gradle task hubCreateStepDefinition with the stepType as mapping.

   Unix systems

   Copy

     ./gradlew hubCreateStepDefinition -PstepDefName=your-mapping-step-name -PstepDefType=mapping -i
   

   Windows

   Copy

     gradlew.bat hubCreateStepDefinition -PstepDefName=your-mapping-step-name -PstepDefType=mapping -i
   

   To associate an XQuery module (instead of the default JavaScript module) with the step definition, add the option -Pformat=xqy. This creates an XQuery sample module, which you can customize, and a JavaScript wrapper.

   Note: The default stepType is custom.

   Result:

   The following files are created:

   • The step definition file:
     Copy

       your-project-root/step-definitions/mapping/your-mapping-step-name/your-mapping-step-name.step.json
     

   • The custom module file:
     Copy

     your-project-root/src/main/ml-modules/root/custom-modules/mapping/your-mapping-step-def-name/main.sjs
     

     The modulePath setting in the step definition points to the custom module file.

2. Manually add the step to the flow.

   1. In a text editor, open the step definition file.

      The custom ingestion step definition file contains the following:

      Copy

         {
          "language" : "zxx",
          "name" : "your-ingestion-step-def-name",
          "description" : null,
          "type" : "INGESTION",
          "version" : 1,
          "options" : {
            "collections" : [ "your-ingestion-step-def-name" ],
            "outputFormat" : "json",
            "targetDatabase" : "data-hub-STAGING"
          },
          "customHook" : { },
          "modulePath" : "/custom-modules/ingestion/your-ingestion-step-def-name/main.sjs",
          "retryLimit" : 0,
          "batchSize" : 100,
          "threadCount" : 4,
          "fileLocations" : {
            "inputFilePath" : "",
            "outputURIReplacement" : "",
            "inputFileType" : ""
          }
        }
      

   2. Likewise, open the flow definition file.

      You can find your flow definition file in your-project-root/flows.

      The default flow definition file without any steps contains the following:

      Copy

         {
          "name": "your-flow-name",
          "description": "",
          "batchSize": 100,
          "threadCount": 4,
          "options": {
            "sourceQuery": null
          },
          "steps": {}
        }
      
      

   3. In the steps node, add the step as a key-value pair.

      • For the key, enter a string containing a number which represents the order of the step in the sequence.

        Note: The steps can be listed in any order, as long as the keys are unique within the steps node of the flow. Duplicate keys can produced unexpected results. The key number must be greater than 0.

      • For the value, copy and paste the entire content of the step definition file.

   4. Edit the step in the flow definition file.

      • Rename the name setting to stepDefinitionName.

      • Rename the type setting to stepDefinitionType.

      • Create a new name setting and assign it a name for the step.

      • (Optional) Delete the following settings:

        • language
        • version
        • modulePath

        The values for these settings are retrieved from the step definition.

   After adding a default ingestion step to the default flow as the first step ("1"), the flow definition file looks as follows:

   Copy

   {
       "name": "your-flow-name",
       "description": "",
       "batchSize": 100,
       "threadCount": 4,
       "options": {
         "sourceQuery": null
       },
       "steps": {
         "1": {
           "name" : "your-step-name",
           "stepDefinitionName" : "your-ingestion-step-def-name",
           "description" : null,
           "stepDefinitionType" : "INGESTION",
           "options" : {
             "collections" : [ "your-ingestion-step-name" ],
             "outputFormat" : "json",
             "targetDatabase" : "data-hub-STAGING"
           },
           "customHook" : { },
           "retryLimit" : 0,
           "batchSize" : 100,
           "threadCount" : 4,
           "fileLocations" : {
             "inputFilePath" : "",
             "outputURIReplacement" : "",
             "inputFileType" : ""
           }
         }
       }
   }
   

Mastering

1. Using Gradle, create the step definition.

   1. Open a command-line window, and go to your project root directory.

   2. At your project root, run the Gradle task hubCreateStepDefinition with the stepType as mastering.

      Unix systems

      Copy

      ./gradlew hubCreateStepDefinition -PstepDefName=your-mastering-step-name -PstepDefType=mastering -i
      

      Windows

      Copy

      gradlew.bat hubCreateStepDefinition -PstepDefName=your-mastering-step-name -PstepDefType=mastering -i
      

      To associate an XQuery module (instead of the default JavaScript module) with the step definition, add the option -Pformat=xqy. This creates an XQuery sample module, which you can customize, and a JavaScript wrapper.

      Note: The default stepType is custom.

      Result:

      The following files are created:

      • The step definition file:
        Copy

        your-project-root/step-definitions/mastering/your-mastering-step-name/your-mastering-step-name.step.json
        

      • The custom module file
        Copy

        your-project-root/src/main/ml-modules/root/custom-modules/mastering/your-mastering-step-def-name/main.sjs
        

      The modulePath setting in the step definition points to the custom module file.

2. Manually add the step to the flow.

   1. In a text editor, open the step definition file.

      The custom ingestion step definition file contains the following:

      Copy

         {
          "language" : "zxx",
          "name" : "your-ingestion-step-def-name",
          "description" : null,
          "type" : "INGESTION",
          "version" : 1,
          "options" : {
            "collections" : [ "your-ingestion-step-def-name" ],
            "outputFormat" : "json",
            "targetDatabase" : "data-hub-STAGING"
          },
          "customHook" : { },
          "modulePath" : "/custom-modules/ingestion/your-ingestion-step-def-name/main.sjs",
          "retryLimit" : 0,
          "batchSize" : 100,
          "threadCount" : 4,
          "fileLocations" : {
            "inputFilePath" : "",
            "outputURIReplacement" : "",
            "inputFileType" : ""
          }
        }
      
      

   2. Likewise, open the flow definition file.

      You can find your flow definition file in your-project-root/flows.

      The default flow definition file without any steps contains the following:

      Copy

         {
          "name": "your-flow-name",
          "description": "",
          "batchSize": 100,
          "threadCount": 4,
          "options": {
            "sourceQuery": null
          },
          "steps": {}
        }
      
      

   3. In the steps node, add the step as a key-value pair.

      • For the key, enter a string containing a number which represents the order of the step in the sequence.

        Note: The steps can be listed in any order, as long as the keys are unique within the steps node of the flow. Duplicate keys can produced unexpected results. The key number must be greater than 0.

      • For the value, copy and paste the entire content of the step definition file.

   4. Edit the step in the flow definition file.

      • Rename the name setting to stepDefinitionName.

      • Rename the type setting to stepDefinitionType.

      • Create a new name setting and assign it a name for the step.

      • (Optional) Delete the following settings:

        • language
        • version
        • modulePath

        The values for these settings are retrieved from the step definition.

   After adding a default ingestion step to the default flow as the first step ("1"), the flow definition file looks as follows:

   Copy

   {
       "name": "your-flow-name",
       "description": "",
       "batchSize": 100,
       "threadCount": 4,
       "options": {
         "sourceQuery": null
       },
       "steps": {
         "1": {
           "name" : "your-step-name",
           "stepDefinitionName" : "your-ingestion-step-def-name",
           "description" : null,
           "stepDefinitionType" : "INGESTION",
           "options" : {
             "collections" : [ "your-ingestion-step-name" ],
             "outputFormat" : "json",
             "targetDatabase" : "data-hub-STAGING"
           },
           "customHook" : { },
           "retryLimit" : 0,
           "batchSize" : 100,
           "threadCount" : 4,
           "fileLocations" : {
             "inputFilePath" : "",
             "outputURIReplacement" : "",
             "inputFileType" : ""
           }
         }
       }
   }
   

Custom

1. Using Gradle, create the step definition.

   1. Open a command-line window, and go to your project root directory.

   2. At your project root, run the Gradle task hubCreateStepDefinition with the stepType as custom.

      Unix systems

      Copy

      ./gradlew hubCreateStepDefinition -PstepDefName=your-custom-step-name -PstepDefType=custom -i
      

      Windows

      Copy

      gradlew.bat hubCreateStepDefinition -PstepDefName=your-custom-step-name -PstepDefType=custom -i
      

      To associate an XQuery module (instead of the default JavaScript module) with the step definition, add the option -Pformat=xqy. This creates an XQuery sample module, which you can customize, and a JavaScript wrapper.

      Note: The default stepType is custom.

      Result:

      The following files are created:

      • The step definition file

        Copy

        your-project-root/step-definitions/custom/your-custom-step-name/your-custom-step-name.step.json
        

      • The custom module file

        Copy

        your-project-root/src/main/ml-modules/root/custom-modules/custom/your-custom-step-def-name/main.sjs
        

      The modulePath setting in the step definition points to the custom module file.

2. Manually add the step to the flow.

   1. In a text editor, open the step definition file.

      The custom ingestion step definition file contains the following:

      Copy

         {
          "language" : "zxx",
          "name" : "your-ingestion-step-def-name",
          "description" : null,
          "type" : "INGESTION",
          "version" : 1,
          "options" : {
            "collections" : [ "your-ingestion-step-def-name" ],
            "outputFormat" : "json",
            "targetDatabase" : "data-hub-STAGING"
          },
          "customHook" : { },
          "modulePath" : "/custom-modules/ingestion/your-ingestion-step-def-name/main.sjs",
          "retryLimit" : 0,
          "batchSize" : 100,
          "threadCount" : 4,
          "fileLocations" : {
            "inputFilePath" : "",
            "outputURIReplacement" : "",
            "inputFileType" : ""
          }
        }
      
      

   2. Likewise, open the flow definition file.

      You can find your flow definition file in your-project-root/flows.

      The default flow definition file without any steps contains the following:

      Copy

         {
          "name": "your-flow-name",
          "description": "",
          "batchSize": 100,
          "threadCount": 4,
          "options": {
            "sourceQuery": null
          },
          "steps": {}
        }
      
      

   3. In the steps node, add the step as a key-value pair.

      • For the key, enter a string containing a number which represents the order of the step in the sequence.

        Note: The steps can be listed in any order, as long as the keys are unique within the steps node of the flow. Duplicate keys can produced unexpected results. The key number must be greater than 0.

      • For the value, copy and paste the entire content of the step definition file.

   4. Edit the step in the flow definition file.

      • Rename the name setting to stepDefinitionName.

      • Rename the type setting to stepDefinitionType.

      • Create a new name setting and assign it a name for the step.

      • (Optional) Delete the following settings:

        • language
        • version
        • modulePath

        The values for these settings are retrieved from the step definition.

   After adding a default ingestion step to the default flow as the first step ("1"), the flow definition file looks as follows:

   Copy

   {
       "name": "your-flow-name",
       "description": "",
       "batchSize": 100,
       "threadCount": 4,
       "options": {
         "sourceQuery": null
       },
       "steps": {
         "1": {
           "name" : "your-step-name",
           "stepDefinitionName" : "your-ingestion-step-def-name",
           "description" : null,
           "stepDefinitionType" : "INGESTION",
           "options" : {
             "collections" : [ "your-ingestion-step-name" ],
             "outputFormat" : "json",
             "targetDatabase" : "data-hub-STAGING"
           },
           "customHook" : { },
           "retryLimit" : 0,
           "batchSize" : 100,
           "threadCount" : 4,
           "fileLocations" : {
             "inputFilePath" : "",
             "outputURIReplacement" : "",
             "inputFileType" : ""
           }
         }
       }
   }
   

What to do next

• Modify your custom module.
• (Optional) To perform other tasks outside the Data Hub space, you can create a custom hook module and add it to a custom hook in the step.

See Also

Previous topic: Create a Flow Using Gradle

Next topic: Run a Flow Using Gradle

Related concepts

• Steps
• Custom Modules
• Editing a Custom Step Module

Related tasks

• Create a Step Using QuickStart
• Add a Custom Hook to a Step Manually
• Manage Steps in a Flow Manually

Related reference

• Flow Definition File