Reducing Memory Consumption with Streaming

The streaming protocol allows you to insert a large document into the database without holding the entire document in memory. Streaming uploads documents to MarkLogic Server in 128k chunks.

Streaming content into the database usually requires less memory on the host running mlcp, but ingestion can be slower because it introduces additional network overhead. Streaming also does not take advantage of the mlcp built-in retry mechanism. If an error occurs that is normally retriable, the job will fail.

To use streaming, enable the -streaming option. For example:

# Windows users, see Modifying the Example Commands for Windows
$ mlcp.sh import -username user -password password -host localhost \
    -port 8000 -input_file_path /my/dir -streaming