Retrieve config

Overview

retrieveConfig.json is the central configuration file for the MarkLogic Retrieval API. It defines the schemas, text extraction paths, vector embeddings, filters, and other search and retrieval behavior exposed by the API.

For a deployment, maintain your own copy of this file rather than editing the one bundled with the distribution, then deploy it with ./gradlew mlLoadSchemas. See Maintaining Your Own retrieveConfig.json. The deployed document is available at /marklogic-retrieval-api/retrieveConfig.json and is the source of truth for the /v1/retrieve/definition endpoint.

Purpose

This configuration file:

1. Defines available document schemas - References JSON schemas that describe the structure of documents in the system
2. Configures text extraction - Specifies XPath expressions for extracting searchable text from documents
3. Declares vector embeddings - Lists all available vector embeddings with their metadata for vector search
4. Provides label and filter descriptions - Supplies optional guidance metadata exposed through /v1/retrieve/definition
5. Identifies pre-processed RAG chunks - Marks chunk documents so the API can return parent document provenance alongside search results

The configuration is read by the API at runtime to:

• Generate the response for /v1/retrieve/definition
• Validate search requests to /v1/retrieve
• Enable vector similarity search
• Support TDE (Template Driven Extraction) view-based searches

Maintaining your own retrieveConfig.json

All paths and commands in this section are relative to the mra-deployer/ directory of the distribution. Run the commands from that directory.

The distribution ships a default retrieveConfig.json inside the MRA artifact, which is extracted to build/mlBundle/mra/ml-schemas/. Do not edit that copy. The build/ directory is regenerated from the MRA artifact whenever the deployer runs, so any changes you make there are overwritten without warning and are not captured in version control.

Instead, maintain your own retrieveConfig.json alongside the deployer and let it override the default.

Why this works

The deployer's gradle.properties declares two schema paths:

mlSchemaPaths=build/mlBundle/mra/ml-schemas,src/main/ml-schemas

Both paths are loaded into the schemas database in the order listed, and both map to the same target URI, /marklogic-retrieval-api/retrieveConfig.json. Because src/main/ml-schemas is loaded last, a file placed there replaces the bundled default. The src/main/ml-schemas directory is not present in the distribution; you create it.

Set up your own configuration

1. Create the directory structure:

   Copy

   mkdir -p src/main/ml-schemas/marklogic-retrieval-api
   

2. Copy the shipped default to use as your starting point:

   Copy

   cp build/mlBundle/mra/ml-schemas/marklogic-retrieval-api/retrieveConfig.json \
      src/main/ml-schemas/marklogic-retrieval-api/retrieveConfig.json
   

   Note:
   
   If build/mlBundle/ does not exist yet, run ./gradlew installMra (or any deployer task) once to extract the MRA artifact, then copy the file.

3. Edit src/main/ml-schemas/marklogic-retrieval-api/retrieveConfig.json to describe your own schemas, vector embeddings, extraction paths, filters, and labels.

4. Commit src/main/ml-schemas/ to your own version control. Exclude build/ from version control.

Deploy your configuration

Deploy configuration changes without reinstalling MRA:

./gradlew mlLoadSchemas

Because MRA reads retrieveConfig.json from the schemas database on each request, the new configuration takes effect immediately. A module reload is not required for a configuration-only change.

Additional schema documents

The same directory holds any other documents you want in the schemas database, such as the JSON Schema files referenced by documentSchemas. Each file is loaded to a URI matching its path relative to src/main/ml-schemas. For example:

src/main/ml-schemas/
└── marklogic-retrieval-api/
    ├── retrieveConfig.json      →  /marklogic-retrieval-api/retrieveConfig.json
    └── yourSchema.json          →  /marklogic-retrieval-api/yourSchema.json

Upgrading MRA

Because your configuration lives outside build/, upgrading the MRA artifact does not disturb it. After an upgrade, compare your file against the newly extracted default to pick up any newly supported properties:

diff src/main/ml-schemas/marklogic-retrieval-api/retrieveConfig.json \
     build/mlBundle/mra/ml-schemas/marklogic-retrieval-api/retrieveConfig.json

Structure

Top-Level Properties

{
  "databaseDescription": "A short human-readable description...",
  "descriptions": {...},
  "documentSchemas": [...],
  "extractionXPaths": [...],
  "chunks": [...],
  "vectorMetadata": [...],
  "metadataFieldXPaths": {...},
  "labelDiscovery": {...},
  "baseFilter": {...},
  "scoring": {...}
}

databaseDescription (string)

Recommended. A human-readable description of what this database contains. Returned as the description field in /v1/retrieve/definition responses.

This is the first thing an agent or LLM reads when it calls /v1/retrieve/definition. A good description tells the agent what domain the data covers, what kinds of documents it contains, and what questions it can reasonably answer. If this field is omitted, the response contains a generic fallback message that provides no useful context.

Example:

{
  "databaseDescription": "PubMed biomedical literature corpus containing peer-reviewed medical..."
}

Guidelines for writing a good description:

• State the domain (medical, legal, financial, general knowledge, etc.)
• Describe the document type (journal articles, SEC filings, news articles, etc.)
• Mention the approximate scope or coverage where helpful
• Keep it to 1–2 sentences

If this field is absent, the definition response will return: "No description configured for this deployment. Set databaseDescription in retrieveConfig.json."

descriptions (object)

Provides optional metadata used to enrich /v1/retrieve/definition output.

Supported sections:

• labels - Metadata for collection labels
• filters - Custom descriptions for filters/views

labels metadata format

Each key under descriptions.labels should be a collection name. Value can be either:

• A string description (backward-compatible)
• An object with:
  • description (string)
  • requireWhen (array of strings)
  • avoidWhen (array of strings)

Example:

{
  "descriptions": {
    "labels": {
      "medical-data": {
        "description": "Data related to medical documents.",
        "requireWhen": [
          "clinical studies",
          ...
        ],
        "avoidWhen": [
          "user distrusts academic sources"
        ]
      }
    }
  }
}

This information appears in /v1/retrieve/definition as metadata on each label object.

filters metadata format

Each key under descriptions.filters should be a constraint name. The shape of the value depends on whether the constraint is a Value/Word/Range constraint or a View constraint.

Value, Word, and Range constraints take an object with any of the following properties, which are merged into that filter's entry in the definition response:

• description (string) — replaces the generated description.
• operators (array of strings) — advertised comparison operators.
• exampleValues (array) — sample values illustrating valid input.
• caseSensitive (boolean)
• wildcardSupport (boolean)

View constraints are keyed by {schemaName}.{viewName} and use a different set of properties:

• viewDescription (string) — replaces the generated view description. If the value of the key is a plain string rather than an object, it is treated as the view description.
• Any other key is matched against a column name in the view. Its string value becomes the description of that column in the definition response. Column names that do not exist in the view are ignored.

Example: Assuming a TDE exists defining the Keywords view within the Medical schema, that view would be included in the filters list as shown here.

{
  "descriptions": {
    "filters": {
      "categoryValueConstraint": {
        "description": "Ensures documents belong to the specified category.",
        "operators": ["eq", "ne"],
        "exampleValues": ["abstract", "full-text"],
        "caseSensitive": false,
        "wildcardSupport": false
      },
      "Medical.Keywords": {
        "viewDescription": "Rows in the Medical.Keywords view.",
        "keyword": "The indexed keyword term for the document."
      }
    }
  }
}

documentSchemas (array)

Defines the JSON schemas for documents in the system. Each schema describes the structure and properties of a document type.

Properties:

• schemaUri (string) - URI of the JSON schema document in the database
• description (string) - Human-readable description of the schema

Example:

{
  "documentSchemas": [
    {
      "schemaUri": "/marklogic-retrieval-api/medicalSchema.json",
      "description": "Schema for medical documents."
    }
  ]
}

Referenced Schemas: The schemaUri points to a JSON Schema (draft-07) document stored in MarkLogic. For example, /marklogic-retrieval-api/medicalSchema.json defines:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "medicalSchema",
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "title": { "type": "string" },
    "category": { "type": "string" },
    "content": { "type": "string" },
    "symptoms": { "type": "array", "items": { "type": "string" } },
    "treatments": { "type": "array", "items": { "type": "string" } },
    "keywords": { "type": "array", "items": { "type": "string" } },
    "lastUpdated": { "type": "string" }
  }
}

extractionXPaths (array)

Defines XPath expressions used to extract text from XML or JSON documents. These paths are used during search operations to extract text to return in the response.

Format: Array of XPath strings

Example:

{
  "extractionXPaths": ["/root/content"]
}

Usage: The API uses these paths to locate and extract text content from documents for full-text search indexing.

Response format: When extractionXPaths is configured and a document matches a full-text search query, each expression is evaluated against the document. Results appear in the extractedText.fulltextExtractedText field of the /v1/retrieve response as an array of objects:

{
  "fulltextExtractedText": [
    {
      "text": "Comprehensive guidelines for managing diabetes...",
      "xpathSource": "/root/content" 
    }
  ]
}

Each object pairs the extracted text (text) with the source XPath expression (xpathSource) that produced it. Pass the xpathSource value as the xpath property in a {uri, xpath} entry in a /v1/retrieve/augment request to retrieve that exact passage from the document. For vector-only searches (no text query), fulltextExtractedText is null.

chunks (array)

Declares collections of pre-processed RAG chunk documents. When a match returned by /v1/retrieve belongs to a configured chunkCollection and its URI matches the chunkIdRegex, the API populates a chunkData object on that match in the response. Matches that are not identified as chunks are returned without any chunkData field.

Use this when your corpus is stored as individual chunk documents (e.g., produced by a chunking pipeline) rather than as full source documents.

Fields per entry

• chunkCollection — string (required): The MarkLogic collection that identifies documents as chunks. The API checks whether a matching document's URI belongs to this collection before applying the regex.

• chunkIdRegex — string (required): A regular expression matched against the document URI. Capture group 1 derives parentUri; capture group 2 (must be numeric) derives chunkNumber.

• extractionXPaths — array of strings (optional): XPath expressions evaluated against the chunk document to extract text content. Results appear as chunkData.extractedText in the /v1/retrieve response.

• textChunkUriTemplate — string (optional): A URI template used to locate the text-bearing chunk document when the matched document is not itself the document holding the chunk text (for example, when vectors and text are stored in separate documents). The template may reference the values captured by chunkIdRegex using {parentUri} and {chunkNumber}; zero-padding in the matched chunk number is preserved. The resolved URI is used both for extractionXPaths evaluation and for deriving previousChunkUri, nextChunkUri, and totalChunks. When the template is omitted, or resolves to a document that does not exist, the API falls back to the matched document itself. For example:

  Copy

  {
    "chunkCollection": "studioResourceCollection-chunks-embeddings",
    "chunkIdRegex": "(.*)/chunks-embeddings/(\\d+)/\\d+\\.json
  quot;,
    "extractionXPaths": ["/text"],
    "textChunkUriTemplate": "{parentUri}/chunks/{chunkNumber}.json"
  }
  

Capture group mapping

• Group 1 — Omitted from response if the regex contains no capture group 1.
  • Response field: chunkData.parentUri
• Group 2 — Omitted from response if absent. If capture group 2 is present but the captured value is not numeric, a warning is logged and no chunkData is attached to the match at all — the document is returned as a non-chunk result.
  • Response field: chunkData.chunkNumber (integer)

Validation

At read time each chunks entry is validated. Entries with a missing chunkCollection or chunkIdRegex, or with an invalid regex pattern, are skipped (a warning is logged). Valid entries are applied at search time.

chunkData in the /v1/retrieve response

When a match is identified as a chunk, the match object gains a chunkData field:

{
  "id": "/reports/annual-chunk-3.json",
  "chunkData": {
    ...
  },
  ...

• chunkCollection — string: The collection that identified this document as a chunk.
• parentUri — string, optional: URI of the original source document that was chunked. Derived from capture group 1. Omitted when the regex lacks capture group 1.
• chunkNumber — integer, optional: Zero-based ordinal position of this chunk within the parent document. Derived from capture group 2. Omitted when the regex lacks capture group 2.
• totalChunks — integer, optional: Total number of chunks for the same parent document. Omitted when the total cannot be derived.
• previousChunkUri — string, optional: URI of the immediately previous chunk. Omitted for first chunks or when the neighbor cannot be derived/found.
• nextChunkUri — string, optional: URI of the immediately next chunk. Omitted for last chunks or when the neighbor cannot be derived/found.
• extractedText — array of strings: Text extracted from the chunk via each extractionXPaths expression. Empty array when no extractionXPaths is configured.

See chunkData in the Retrieve Guide for full response field documentation.

Example

Configuration:

{
  "chunks": [
    {
      "chunkCollection": "SmallPdf-chunks",
      "chunkIdRegex": "(.*)-chunk-(\\d+)\\.json
quot;,
      "extractionXPaths": ["/text"]
    }
  ]
}

With this configuration, a document /reports/annual-chunk-3.json that belongs to the SmallPdf-chunks collection produces the following chunkData on its match in the /v1/retrieve response:

{
  "id": "/reports/annual-chunk-3.json",
  "score": { "fulltext": 5120, "vectors": {} },
  "extractedText": { "vectorExtractedText": null, "fulltextExtractedText": [] },
  "chunkData": {
    "chunkCollection": "SmallPdf-chunks",
    "parentUri": "/reports/annual",
    ...
  }
}

vectorMetadata (array)

Declares all available vector embeddings in the system. Each entry describes a vector column in a TDE (Template Driven Extraction) view that can be used for vector-based similarity search.

• schemaName — string (required): Name of the TDE schema containing the view.
• viewName — string (required): Name of the TDE view containing the vector column.
• vectorColumn — string (required): Name of the column containing vector embeddings. This value is the key used in the vectors request parameter when calling /v1/retrieve: {"vectors": {"<vectorColumn>": [...]}}.
• embeddingModel — string (required): Name or identifier of the embedding model used.
• dimensions — integer >= 1 (required): Dimensionality of the vector embeddings. Submitted vectors must match this length exactly.
• modelVersion — string (optional): Version of the embedding model.
• description — string (optional): Human-readable description of what domain this embedding covers and what queries it is suited for. Agents use this to choose between multiple available embeddings.
• requireWhen — array of strings (optional): Advisory guidance for scenarios where using this embedding is recommended. Same semantics as descriptions.labels[].requireWhen.
• avoidWhen — array of strings (optional): Advisory guidance for scenarios where this embedding should not be used. Same semantics as descriptions.labels[].avoidWhen.

Validation: At read time, each entry is validated. Entries with missing or wrongly-typed required fields are skipped (a warning is logged). Optional fields present with an incorrect type are stripped from that entry (a warning is logged) but the entry is still returned.

Example:

{
  "vectorMetadata": [
    {
      "schemaName": "Medical",
      "viewName": "MedicalDataView",
      "vectorColumn": "clinicalEmbedding",
      "embeddingModel": "text-embedding-3-large",
      "modelVersion": "1.0",
      "dimensions": 3072,
      "description": "Dense embedding for clinical...",
      "requireWhen": [
        "The query contains clinical..."
      ],
      "avoidWhen": [
        "The query is purely administrative or structural"
      ]
    },
    {
      "schemaName": "Medical",
      "viewName": "SmallMedicalView",
      "vectorColumn": "smallMedicalEmbedding",
      "embeddingModel": "text-embedding-3-small",
      "modelVersion": "1.0",
      "dimensions": 16
    }
  ]
}

Vector Search Integration: The API uses this metadata to:

• Validate vector dimensions in search requests
• Map vectorColumn names to TDE view columns for Optic queries
• Execute vector similarity searches using MarkLogic's vector operations

Relationship to /v1/retrieve/definition

The /v1/retrieve/definition endpoint reads retrieveConfig.json and combines it with other MarkLogic configuration to generate its response.

Mapping: retrieveConfig.json → /v1/retrieve/definition Response

• databaseDescription → description — Set this to give agents context about the database. Falls back to a "not configured" message if absent.
• documentSchemas → documentSchemas — Directly included in response.
• vectorMetadata → vectorMetadata — Directly included in response.
• descriptions.labels → labels[].description, labels[].requireWhen, labels[].avoidWhen — Metadata enriches discovered collections.
• descriptions.filters → filters[*].description — Custom description override when provided.
• extractionXPaths → (internal use) — Not exposed in API response.
• chunks → (internal use) — Not exposed in /v1/retrieve/definition; used at search time to identify chunk documents and populate chunkData in /v1/retrieve responses.
• (derived from MarkLogic) → labels — Collections discovered from database.
• (derived from MarkLogic) → filters — Constraints from search options + TDE views.
• (always present) → recommendedWorkflow — Machine-readable 3-step agent workflow; not configurable.

Why Label Metadata Matters for LLM Applications

Label metadata in descriptions.labels is especially useful when an LLM is selecting retrieval constraints.

• description gives meaning to a collection so the model can choose relevant sources.
• requireWhen provides positive retrieval hints for intent routing.
• avoidWhen provides guardrails to reduce poor label choices for specific user intents.

Recommended LLM Reasoning Model

For each user question:

1. Read the question and infer intent.
2. For each label:

• Use description to understand what the label represents.
• Check whether the question clearly matches any requireWhen hints.
• Check whether the question clearly matches any avoidWhen hints.

3. Decide per label whether to:

• require the label,
• exclude the label, or
• ignore the label.

Important nuance:

• requireWhen does not mean "always require".
• avoidWhen does not mean "always exclude".

These are strong hints to support context-sensitive retrieval decisions.

Practical impact for LLM systems:

1. Better retrieval planning: the model can choose labels aligned to user intent.
2. Improved transparency: applications can show users why a label was suggested.
3. Safer defaults: avoid high-risk or low-trust sources in inappropriate contexts.
4. Lower hallucination risk: retrieval is grounded in the most appropriate collections.

Configuration Workflow

1. Define Document Schemas

Create JSON Schema files that describe your document structure:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "yourSchema",
  "type": "object",
  "properties": {
    "field1": { "type": "string" },
    "field2": { "type": "number" }
  }
}

Store in MarkLogic Server at /marklogic-retrieval-api/yourSchema.json by placing the file in the deployer at src/main/ml-schemas/marklogic-retrieval-api/yourSchema.json. See Maintaining Your Own retrieveConfig.json.

2. Configure Vector Embeddings

If using vector search:

1. Create TDE templates that define views with vector columns
2. Deploy TDE templates to MarkLogic
3. Add vector metadata entries to vectorMetadata array.

3. Update retrieveConfig.json

Add references to your document schemas and vector configurations in your own copy of the file in the deployer at src/main/ml-schemas/marklogic-retrieval-api/retrieveConfig.json. See Maintaining Your Own retrieveConfig.json.

4. Configure Chunk Identification (if using RAG chunks)

If your corpus consists of pre-processed RAG chunk documents (e.g., produced by a document-chunking pipeline), add a chunks entry so the API can annotate matching chunks with provenance data.

5. Deploy Configuration

Deploy your configuration and schema documents:

./gradlew mlLoadSchemas

Run ./gradlew mlReloadModules as well only if you have also changed MRA modules; it is not needed for a configuration-only change.

6. Verify Configuration

Check the /v1/retrieve/definition endpoint.

Testing Configuration

After deploying your configuration, verify each component works correctly using the tests below. Each test includes expected success output and common failure scenarios.

Verify Schema References

Ensure all schemaUri references point to valid documents in the database:

curl --digest -u admin:admin "http://localhost:8003/v1/documents?uri=/marklogic-retrieval-api/medicalSchema.json&database={{your schema database name}}"

If you get RESTAPI-NODOCUMENT, verify the URI spelling and redeploy schemas with ./gradlew mlLoadSchemas.

Validate Vector Metadata

Confirm TDE views exist for all vector metadata entries:

This example uses the Optic API's fromView function to read a TDE view in Query Console.

// Query in Query Console
const op = require('/MarkLogic/optic');
const view = op.fromView("Medical", "SmallMedicalView");
const results = view.result();
results;

Expected output: Query Console returns an array with at least one result (if data exists in that view). The result object contains columns matching your TDE view definition.

If TDE view not found: Query Console displays an error like "Unknown table: Table 'Medical.SmallMedicalView' not found". Verify:

• Schema name matches TDE template schema (case-sensitive)
• View name matches TDE view definition (case-sensitive)
• TDE templates are deployed: ./gradlew mlLoadSchemas

Test Search Functionality

Verify vectors work in search requests using the retrieve endpoint with vectors payload.

Expected output: HTTP 200 with JSON response containing:

• "matches" array as part of the search results
• Each match has "id", "score", and "extractedText" fields
• Vector dimensions in your request must match the dimensions value in vectorMetadata

Common failures:

• HTTP 400 "Vector dimensions mismatch" — Your vector array length (16) doesn't match configured dimensions. Verify dimensions in vectorMetadata.
• HTTP 200 with empty matches — Search ran successfully but no results matched your query. Try adjusting the query text or vector values, or verify data is loaded in the database.

metadataFieldXPaths (object)

Object map of metadata field names to XPath expressions used by /v1/retrieve when the request includes a metadata array.

When a request includes "metadata": ["author", "date"], each field name is looked up in metadataFieldXPaths. Matching configured XPaths are evaluated against each matching document, and results are returned in metadataProperties.

Example:

{
  "metadataFieldXPaths": {
    "author": "/root/metadata/author",
    "date": "/article/header/date"
  }
}

If a request includes unknown metadata fields, /v1/retrieve returns a warning listing the invalid field names and still processes valid metadata fields.

processingMetadataPath (string or object)

Defines the XPath location of customer-defined processing metadata within document bodies. This config field powers the processingMetadata field returned by /v1/retrieve/augment when "processingMetadata" is included in the request metadata array.

String form — a single XPath that must select an object or array node:

{
  "processingMetadataPath": "/root/processingInfo"
}

The matched node is returned as-is. If the XPath selects a text/scalar node rather than an object or array, null is returned and a warning is logged — point the XPath at the container object, not a leaf value.

Map form — a {key: xpath} object that assembles multiple XPaths into one result object:

{
  "processingMetadataPath": {
    "status": "/root/audit/status",
    "version": "/root/audit/schemaVersion",
    "entities": "/root/metadata"
  }
}

Each XPath is evaluated independently. Keys whose XPath matches zero nodes are omitted from the result. Text/scalar node matches are included as string values within the object.

Example augment response (when "processingMetadata" is requested):

{
  "uri": "/medical/doc001.json",
  "document": { ... },
  "processingMetadata": {
    "status": "approved",
    "version": "2.1",
    "entities": { "labels": ["urgent"], "relations": ["precedes"] }
  }
}

If processingMetadataPath is absent from the config, processingMetadata is always null.

labelDiscovery (object)

Controls which MarkLogic collections are reported as labels by /v1/retrieve/definition. Labels are discovered dynamically from the collections present in the content database; labelDiscovery narrows that list.

The property is optional. When it is absent, the defaults below apply.

Fields

• include — array of strings (optional): Allowlist of collection names. When non-empty, only these collections are reported as labels. Default: not set (all discovered collections are eligible).
• exclude — array of strings (optional): Collection names to omit from the label list. Applied after include. Default: not set.
• excludeSystemCollections — boolean (optional): Omit internal MarkLogic system collections. Default: true.
• excludeTemporalCollections — boolean (optional): Omit collections managed by MarkLogic temporal features. Default: true.
• maxResults — positive integer (optional): Maximum number of labels returned. Collections are sorted alphabetically and the list is truncated to this length. Default: not set (no limit).

Validation

labelDiscovery must be an object. include and exclude must be arrays of strings, the two exclude* properties must be booleans, and maxResults must be a positive integer. An invalid value causes /v1/retrieve/definition to fail with a 500 error rather than being silently ignored.

Effect on the definition response

When maxResults truncates the list, the definition response reports the pre-truncation count so clients can detect the limit:

• totalLabels — the number of eligible collections before truncation.
• labelsTruncated — true when fewer labels were returned than were eligible.

Example

{
  "labelDiscovery": {
    "exclude": ["staging", "scratch"],
    "excludeSystemCollections": true,
    "excludeTemporalCollections": true,
    "maxResults": 100
  }
}

baseFilter (object)

Defines default collection label constraints that are always applied to every /v1/retrieve request, regardless of what labels (if any) the caller provides. baseFilter labels take precedence when they overlap with request-level labels.

Currently supports a labels property with the same structure as the /v1/retrieve request labels parameter: a map of collection name to an object with a constraintValue of "MustHave" or "MustNotHave".

Example — always restrict results to documents in the "pubmed" collection:

{
  "baseFilter": {
    "labels": {
      "pubmed": { "constraintValue": "MustHave" }
    }
  }
}

scoring (object)

Controls the full-text search scoring algorithm. When omitted, the defaults below apply.

• scoreMethod — string: Default: "bm25". Scoring algorithm passed through to op.fromSearchDocs. Any scoring method accepted by op.fromSearchDocs for your MarkLogic Server version may be used; for example, "logtfidf" is used in the MarkLogic 11 configuration shipped with the API. The value is not validated by the Retrieval API, so an unrecognized method surfaces as a MarkLogic Server error at search time.
• bm25LengthWeight — number: Default: 0.5. BM25 length normalization weight. 0.0 = no length normalization; 1.0 = full normalization.

When to Adjust Scoring

The default BM25 algorithm with bm25LengthWeight: 0.5 works well for most deployments. Consider tuning only if you observe unexpected ranking behavior:

• Increase bm25LengthWeight to 0.75–1.0 if short documents are being unfairly downranked (e.g., dense technical abstracts or executive summaries)
• Decrease bm25LengthWeight to 0.0–0.25 if longer documents should be preferred (e.g., comprehensive reference materials or detailed research papers)

Most deployments use the default—adjust only if testing reveals ranking issues.

Example:

{
  "scoring": {
    "scoreMethod": "bm25",
    "bm25LengthWeight": 0.75
  }
}

Best Practices

Label Metadata Best Practices

Writing effective label metadata in descriptions.labels ensures that LLM systems can make intelligent collection selection decisions.

1. Write clear, concise descriptions (1–2 sentences)

   • Describe what documents are in the collection (domain, document type, subject matter)
   • Avoid overly technical jargon; use language an LLM would understand
   • Example: "Medical and biomedical literature including peer-reviewed research articles, clinical trial data, and pharmacological studies"

2. Provide realistic requireWhen hints (2–3 concrete scenarios)

   • List specific user intents or query patterns where this collection is appropriate
   • Use language an LLM can pattern-match against: query phrases, intent keywords, domain terminology
   • Example: ["clinical terminology", "drug names", "disease or treatment queries", "medical research"]

3. Define guardrail avoidWhen hints (1–2 scenarios)

   • Describe queries or contexts where this collection would mislead or be irrelevant
   • Helps LLM systems avoid poor collection selections for specific user intents
   • Example: ["purely administrative or procedural questions", "queries requiring financial or legal expertise"]

4. Test with actual queries

   • Once configured, verify that requireWhen and avoidWhen hints align with real user queries
   • Adjust hints if you find that LLM systems over-select or under-select the collection
   • Monitor /v1/retrieve/definition responses to ensure hints are clear

Troubleshooting

Issue: Schema not appearing in /v1/retrieve/definition

Cause: Schema URI is invalid or document doesn't exist in database

Solution:

• Verify schema document exists: fn.docAvailable(schemaUri)
• Check URI spelling and path
• Redeploy schemas: ./gradlew mlLoadSchemas

Issue: Vector search returns no results

Cause: Vector metadata doesn't match TDE view structure

Solution:

• Verify TDE view exists: op.fromView(schemaName, viewName)
• Check vector column name matches TDE template
• Confirm dimensions match vector data
• Validate TDE templates deployed: ./gradlew mlLoadSchemas

Issue: /v1/retrieve/definition returns empty vectorMetadata

Cause: retrieveConfig.json not loaded or corrupted

Solution:

• Verify document exists at /marklogic-retrieval-api/retrieveConfig.json
• Check JSON syntax is valid
• Redeploy: ./gradlew mlLoadSchemas