Excluded Paths

[v12.1.0 and up]

Excluded paths allow all XML elements, attributes, and JSON properties that match a given path expression, along with that expression’s children, to be excluded from having their words and values indexed. Excluded paths reduce the amount of indexing performed on all affected documents.

The universal index is populated as soon as documents are loaded into MarkLogic Server. By default, MarkLogic Server indexes all lexical content from the entire document. This content includes vector embeddings, hashes, and encoded content, which users do not normally search. Indexing content that does not need to be searched translates into higher resource requirements to extract and store these values as part of the universal index. As the generated term lists grow, the required memory to search using these indexes also increases. Using an excluded path reduces these resource requirements by preventing specified content in documents from becoming part of the universal index.

Security requirements

Managing excluded paths requires the http://marklogic.com/xdmp/privileges/admin/database privilege (or corresponding granular privilege). Additional roles or privileges may be required: for example, managing the excluded paths feature through the Admin Interface requires the admin-ui-user role, while invoking the Management REST API endpoints requires the manage-user role.

Components of an excluded path

1. path: A valid path expression that identifies the XML element or JSON property to exclude from indexing. Path expressions are based on XPath.

   • All descendant nodes are excluded from indexing, including XML attributes.

   • Namespace prefixes used as part of the path expressions must be defined in the database’s Path Namespaces.

2. collection: Optional. Limit the exclusion to documents that have the configured collections. Multiple values are ORed.

3. directory: Optional. Limit the exclusions to documents under the configured directories (and their subdirectories). Multiple values are ORed.

The combination of an excluded path's components determines its uniqueness. You can configure multiple excluded paths of the same path value, as long as they differ by collection or directory.

Each excluded path is evaluated independently. The effective exclusion is the union of all configured excluded paths.

Preconfigured excluded paths and namespaces

Newly created databases are preconfigured with these excluded paths:

• //vec:vector

• //vector

Newly created databases are also preconfigured with these Path Namespaces:

• vec maps to http://marklogic.com/vector.

For newly created MarkLogic Server clusters, only the Documents database is preconfigured with these excluded paths and namespaces.

During MarkLogic Server upgrades, these excluded paths and namespaces are not added to existing databases or clusters.

View excluded paths

To view a list of configured excluded paths through the Admin Interface, navigate to your database’s Excluded Paths configuration page (Databases > (your database) > Excluded Paths). The Excluded Paths configuration page appears with all currently configured excluded paths.

You can also view configured excluded paths programmatically:

• JavaScript: Use admin.databaseGetIndexExcludedPaths()
• XQuery: Use admin:database-get-index-excluded-paths()
• REST API: Use GET /manage/v2/databases/{id|name}/properties
  • Look for the index-exclusion-paths XML element or excluded-path JSON property.

Add excluded paths

To add an excluded path using the Admin Interface, follow these steps:

1. Navigate to your database’s Excluded Paths configuration page (Databases > (your database) > Excluded Paths).

2. Click Add. The Add Excluded Paths modal appears.

3. Enter the required Path to exclude and any optional Collections or Directories, then click Add. The modal closes. The new Excluded Path along with Action buttons to Edit or Delete it appears in the Excluded Paths configuration page table.

You can also add an excluded path programmatically:

• JavaScript: admin.databaseAddIndexExcludedPath()

• XQuery: admin:database-add-index-excluded-path()

• REST API: PUT /manage/v2/databases/{id|name}/properties

  Note:
  
  

  The REST API endpoint expects the index-exclusion-paths XML element or the excluded-path JSON property to contain all excluded paths expected to take effect. Any existing excluded path that is not included in the payload will be removed.

Delete excluded paths

To delete an excluded path using the Admin Interface, follow these steps:

1. Navigate to your database’s Excluded Paths configuration page (Databases > (your database) > Excluded Paths).

2. Click Delete for the excluded path that you want to delete. The confirmation modal appears.

3. Click Delete. The modal closes, and the excluded path is deleted from the Excluded Paths configuration page.

You can also delete an excluded path programmatically:

• JavaScript: admin.databaseDeleteIndexExcludedPath()

• XQuery: admin:database-delete-index-excluded-path()

• REST API: PUT /manage/v2/databases/{id|name}/properties

  Note:
  
  

  The REST API endpoint expects the index-exclusion-paths XML element or the excluded-path JSON property to contain all excluded paths expected to take effect. So, to delete an excluded path, you must specify all excluded paths in the payload except for the one that you want to delete.

Edit excluded paths

To edit an excluded path using the Admin Interface, follow these steps:

1. Navigate to your database’s Excluded Paths configuration page (Databases > (your database) > Excluded Paths).

2. Click Edit for the excluded path that you want to edit. The Edit Excluded Paths modal appears.

3. Edit the desired fields, and click Update. The modal closes, and the updated information appears on the Excluded Paths configuration page.

Programmatically, editing an excluded path involves deleting an existing excluded path and adding a new excluded path.

Scope of effect

Excluded paths apply only to indexing words, phrases, and values as part of the universal index. This effect extends to the terms' position, stem, wildcard, and one- to three-character term lists as part of the database configuration page.

Search

Queries and filters that rely solely on the universal index will no longer match content under the configured excluded paths. Queries like cts:word-query() and cts:element-value-query() (and their variants except for fields) are affected.

Word query

Excluded paths override any configured word query includes. So, queries like cts:word-query() and cts:element-value-query() (and their variants except for fields) will no longer match any content under the configured excluded paths—even if an element or property is configured as part of the included elements of the database’s word query settings.

Path resolution

Path expressions like /path/to/element[child = 'value'] that contain predicates (the filters between square brackets) resolve the filter using the universal index. If element, child, or any of their parent nodes are configured as an excluded path, then this expression will not resolve as expected.

The following features may be affected:

1. Path Range Index > Path Expression
2. Field > Field Path > Path
3. Geospatial Path Indexes > Path Expression
4. Geospatial Region Indexes > Path Expression
5. TDE Template > context and column > val
6. Protected Paths > Path Expression
7. Redaction > rule > path

Relevance score

Excluded paths can affect the relevance score of a document because they reduce the number of its terms included in the universal index. This reduction has a direct impact on the computed normalized term frequency and term density. In general, term density increases as excluded paths reduce the number of terms. This increase in term density increases a document’s relevance score. Expect this change in calculated relevance score when using score-logtfidf (default), score-logtf, and score-bm25.

When sorted using relevance score, the sequence of matching documents depends on the document coverage of the configured excluded paths:

• If the excluded paths equally affect all documents, then the sequence of matching documents is likely to remain the same despite the change in the calculated relevance score.

• If the excluded paths affect only a subset of documents, then the sequence of matching documents is likely to be different. The affected documents are likely to score higher because of their increased term density, so they are likely to appear before non-affected documents in the sequence of matching documents.

• The effect on term density is more prominent in XML documents because all content is treated as string.

Document length

Excluded paths do not affect a document’s length calculation or a database’s average document length calculation.

Features not affected by excluded paths

Despite an element, attribute, or property being in an excluded path, these elements and features function as usual:

• The structure of a document and the relationship of each node within a document is still indexed:
  • Scoping queries like cts:element-query() and cts:json-property-scope-query() continue to match elements, attributes, and properties covered by the excluded paths:
    • Word queries inside such scoping queries will not match content.
• Other indexes are not affected:
  • Range indexes continue to match content from excluded documents.
  • Queries using configured fields continue to match excluded document content.
  • Words and values from lexicons continue to be retrieved from content under excluded paths.
• Handling of managed and unmanaged triples is not affected.
• TDE continues to extract content.

Concerns and considerations

The following features may be affected if the configured queries make use of cts:word-query() (and non-field variants):

• Query-based partitions for tiered storage.
• additional-query of Query Options for Search API and Client REST API.