Augment endpoint

The /v1/retrieve/augment endpoint retrieves full document content from the MarkLogic database by URI. This endpoint is designed to work in conjunction with the /v1/retrieve endpoint, which returns URIs of matching documents that can then be augmented with their full content.

Overview

The augment endpoint provides:

• Document Retrieval: Fetch complete document content by URI
• Batch Operations: Retrieve multiple documents in a single request
• Content Type Support: Returns documents in their native format (JSON, XML, text, binary)
• Opt-in Response Metadata: Optionally include MarkLogic database metadata and document-level processing metadata alongside each document
• Efficient Loading: Optimized for retrieving search results

Basic Usage

Retrieve documents by supplying URIs as part of the payload:

curl -X POST http://localhost:8003/v1/retrieve/augment \
  -H "Content-Type: application/json" \
  --digest -u admin:admin \
  -d '{
    "URIs": ["..."]
  }'

GET requests are supported with URL-encoded parameters:

curl -X GET "http://localhost:8003/v1/retrieve/augment?URIs=%5B%22%2Fmedical%2Fdoc001.json%22%5D" \
  --digest -u admin:admin

Note: POST is recommended for readability and when retrieving multiple documents.

Request Parameters

URIs (array, required)

Array of document URIs to retrieve. Each URI must be a valid document identifier in the MarkLogic database.

Structure:

{
  "URIs": ["/path/to/document1.json", "/path/to/document2.xml"]
}

Requirements:

• Must be an array of strings (even for single documents)
• URIs are case-sensitive and must match exactly as stored in MarkLogic

metadata (array, optional)

Controls which augment-level metadata fields appear in each document entry of the response. By default (when metadata is absent or empty), no metadata fields are added and the response is identical to the pre-metadata behaviour.

Valid values:

• "metadata" — MarkLogic database metadata: collections, contentType, lastModified
• "processingMetadata" — Customer-defined processing metadata extracted from the document body using the processingMetadataPath field in retrieveConfig.json
• "provenance" — Chunk provenance when XPath-based chunk retrieval is used (see Chunk-Level Retrieval)

Structure:

{
  "URIs": ["/medical/doc001.json"],
  "metadata": ["metadata", "processingMetadata", "provenance"]
}

Response Format

Success Response

HTTP Status: 200 OK

Body Structure:

{
  "documents": [
    {
      "uri": "/medical/doc001.json",
      "document": { ... }
    },
    {
      "uri": "/medical/doc002.json",
      "document": { ... }
    }
  ]
}

Response Fields

Top-level fields:

• documents (array): Array of document entry objects, one per URI requested

Per-document entry fields:

• uri — string; always present. The document URI.
• document — object | string | null; always present. Document content in its native format; null if the URI was not found.
• metadata — object | null; only when requested. MarkLogic database metadata (see below).
• processingMetadata — object | null; only when requested. Customer-defined processing metadata (see below).
• provenance — object | null; only when requested. Chunk provenance for XPath-addressed chunks (see below).

metadata object fields (present when "metadata" is in the request array):

• collections — string[]. MarkLogic collections the document belongs to.
• contentType — string. MIME type of the document.
• lastModified — string. ISO 8601 timestamp of the last modification.

processingMetadata (present when "processingMetadata" is in the request array): Extracted from the document body using the processingMetadataPath XPath configuration in retrieveConfig.json. Returns an object, or null when: the config is absent; the XPath matches no nodes; or (in string form) the XPath selects a text/scalar node rather than an object or array. See the Configuration Guide for string vs. map form details.

provenance (present when "provenance" is in the request array): null for whole-document retrievals. Contains chunk address information for XPath chunk retrievals — see Chunk-Level Retrieval.

Example Response — With Opt-in Metadata

{
  "documents": [
    {
      "uri": "/medical/doc001.json",
      "document": {
        "id": "doc001",
        "title": "Diabetes Management Guidelines",
        "category": "endocrinology"
      },
      "metadata": {
        "collections": ["medical-data"],
        "contentType": "application/json",
        "lastModified": "2025-03-15T12:34:56Z"
      },
      "processingMetadata": null,
      "provenance": null
    }
  ]
}

In this example processingMetadata is null because processingMetadataPath is not configured in retrieveConfig.json, and provenance is null because the full document was returned (no XPath chunk address was supplied). See Chunk-level retrieval

Example Response - XML Documents

{
  "documents": [
    {
      "uri": "/medical/doc004.xml",
      "document": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<medical-record>\n  <patient-id>12345</patient-id>\n  <diagnosis>Hypertension</diagnosis>\n  <treatment>Beta blockers</treatment>\n</medical-record>"
    }
  ]
}

Working with Search Results

Typical Workflow

The augment endpoint is designed to work with the retrieve endpoint:

1. Search - Use /v1/retrieve to find relevant documents
2. Get URIs - Extract URIs from search results
3. Augment - Use /v1/retrieve/augment to fetch related content
4. Display - Present content to users

Complete Example

# Step 1: Search for documents
curl -X POST http://localhost:8003/v1/retrieve \
  -H "Content-Type: application/json" \
  --digest -u admin:admin \
  -d '{
    "text": "diabetes treatment",
    "topk": 5
  }' > search_results.json

# Step 2: Extract URIs (using jq)
URIS=$(cat search_results.json | jq -c '[.matches[].id]')

# Step 3: Augment with full content
curl -X POST http://localhost:8003/v1/retrieve/augment \
  -H "Content-Type: application/json" \
  --digest -u admin:admin \
  -d "{\"URIs\": $URIS}"

Document Handling

Content Types

The endpoint handles various document formats:

• JSON Documents — Returned as parsed JSON objects; nested structures preserved; native JSON data types maintained.
• XML Documents — Returned as XML strings; preserves XML structure and attributes; includes XML declaration if present.
• Text Documents — Returned as plain text strings; line breaks and formatting preserved.
• Binary Documents — May be base64-encoded depending on configuration; check content type for proper handling.

Empty Results

If no URIs are provided or the URIs array is empty:

{
  "documents": []
}

Partial Results

If some URIs exist and others don't, all requested URIs are still returned. Non-existent documents appear in the response with "document": null.

{
  "documents": [
    {
      "uri": "/medical/doc001.json",
      "document": {
        /* content */
      }
    },
    {
      "uri": "/medical/nonexistent.json",
      "document": null
    },
    ...
  ]
}

Chunk-Level Retrieval

In addition to whole-document retrieval, each entry in the URIs array can be a {uri, xpath} object to retrieve a specific fragment of a document using an XPath expression.

{
  "URIs": [
    { "uri": "/medical/doc001.json", "xpath": "/root/content" }
  ]
}

Plain strings (whole-document) and {uri, xpath} objects can be mixed in the same request.

Fallback behaviour: if xpath is absent, empty, or matches zero nodes in the document, the full document is returned silently — no error is raised.

Provenance: when a chunk is returned, the provenance field (if requested via the metadata parameter) contains the source document URI and the XPath that was evaluated:

{
  "uri": "/medical/doc001.json",
  "document": "Comprehensive guidelines for managing type 2 diabetes...",
  "provenance": {
    "sourceDocument": "/medical/doc001.json",
    "chunkAddress": {
      "xpath": "/root/content"
    }
  }
}

When the full document is returned (no XPath or fallback), provenance is null.

End-to-End Grounded Citation Workflow

1. POST /v1/retrieve — search returns fulltextExtractedText with { text, xpathSource } entries
2. Select a passage; pass its xpathSource as the xpath in a {uri, xpath} augment request
3. POST /v1/retrieve/augment with metadata: ["provenance"] — response includes the chunk text and provenance.chunkAddress.xpath
4. Use the provenance to build a grounded citation (document URI + exact XPath location)