New features in MarkLogic 12.0

This section describes the new features in MarkLogic 12.0.

Native vector search

MarkLogic 12.0 introduces native support for Vector Search with Approximate Nearest Neighbor (ANN) indexing, enabling high-performance semantic search across large-scale unstructured data. This feature is designed to enhance AI-powered applications by allowing similarity-based retrieval using cosine distance as a ranking metric.

Key capabilities

• Native ANN indexing: Efficiently index and search vector embeddings stored in JSON and XML documents using a built-in ANN index.
• Top-K similarity search: Retrieve the top K most relevant results based on cosine similarity between query and indexed vectors.
• Hybrid search support: Seamlessly combine vector similarity with full-text, structured, graph, and geospatial queries using the Optic API.
• Flexible storage: Extract vector embeddings from arbitrary locations in documents using Template Driven Extraction (TDE).
• Scalable performance: Optimized for large-scale vector operations with tunable parameters to balance precision and performance.

See Building Vector Queries in Get Started with Optic.

To complement the new vector indexing capabilities, MarkLogic 12.0 includes a rich set of built-in functions for working with vectors in both JavaScript and XQuery:

• Vector creation and manipulation: Easily define, normalize, and transform vectors directly in your server-side code.
• Cosine and euclidean distance: Compute similarity scores between vectors using native functions—ideal for ranking and filtering.

See Vector Functions in Develop Server-Side Applications.

Virtual views

Template Driven Extraction (TDE), introduced in MarkLogic 9, generates views that enable you to define a relational lens over your document data. These views are extracted from documents, persisted as documents written to the system, and are optimized for fast retrieval. However, for large datasets, the extraction and storage of these views takes time to complete and incurs cost to maintain. These expenditures make relational exploration and ad hoc analytics hard to execute.

MarkLogic 12.0 introduces virtual views. Virtual views allow complete views or individual columns to be extracted at query time rather than write time and, therefore, do not require reindexing to make them available. They are accessible as soon as they are configured and do not increase write latency or storage costs. Virtual columns can be combined with non-virtual (materialized) columns for maximum flexibility. Materialize the columns that are needed for fast index resolution, such as primary keys for joins and scalar values for sorting, and leave the other columns as virtual.

Shortest path graph algorithm

MarkLogic 12.0 introduces support for the Shortest Path graph algorithm, enabling developers and data scientists to solve complex graph traversal problems directly within the database. This feature enhances semantic search, route planning, and AI-driven contextual retrieval by identifying the most efficient paths between entities in a knowledge graph.

The Shortest Path algorithm is available through magic property extensions in SPARQL as well as through a new operator in the Optic API.

See Graph Algorithms in Understand Semantic Graphs.

BM25 search scoring

MarkLogic 12.0 supports BM25 scoring of search results.

The BM25 method of scoring documents is widely used because of its effectiveness in ranking documents based on relevance to a query by taking into account the lengths of documents better than other algorithms. In versions prior to MarkLogic 12.0, the TF-IDF scoring algorithm allows for normalizing based on document length, but it has to be done at configuration time and has only a number of discrete levels. BM25 allows users to take into consideration the length of the matching document compared to the rest of available documents at query time using a weighting factor. This scoring method allows for shorter documents with similar term frequency to rank higher than longer documents.

Dynamic hosts

MarkLogic 12.0 introduces dynamic hosts, a major enhancement that simplifies and secures the process of adding and removing hosts to handle fluctuating workloads. Dynamic hosts allow a secure, lightweight addition and removal of evaluator nodes (E-nodes) in MarkLogic Server clusters. Additionally, since dynamic hosts do not participate in the cluster quorum calculation, new topologies for highly-available clusters with split evaluator and data nodes are now possible.

To join a cluster, a dynamic host must present a dynamic host token to one of the cluster’s Admin app servers. So, a new Security feature called Cluster API Tokens allows you to generate and manage dynamic host tokens through the Admin Interface. You can also generate and manage these tokens through new JavaScript and XQuery functions and through new REST API endpoints.

Dynamic hosts details how to configure dynamic hosts along with their dynamic host tokens.

Updated stemming dictionaries

MarkLogic 12.0 includes a significant update to the dictionaries that are used for stemmed searches in multiple languages. This update includes stemming changes like the handling of s and z in British English and the addition of thousands of words across 14 languages: English, French, Italian, Spanish, German, Russian, Dutch, Portuguese, Korean, Swedish, Nynorsk, Bokmal, Arabic, and Persian.

However, changing the stemming dictionaries does change the way content is indexed. So, to take full advantage of this update, a full or partial reindex is required. See Updated stemming dictionaries.

REST API content transform improvements

REST API content transforms have been enhanced to support updating document collections, permissions, quality, temporal collections, and metadata.

These context parameters are now supported in the transform API:

• collections
• permissions
• quality
• temporalCollection (temporal-collection in XQuery)
• metadata

See Working With Content Transformations.

Enhanced storage failure detection and failover

MarkLogic 11.0 introduced the ability to shut down based on delays in accessing file systems that store forest data. MarkLogic 12.0 introduces the additional ability to shut down based on disk failure. This improvement is achieved by periodically attempting to write then read a custom file to all configured file systems that store forest data (a forest’s default, fast, and large data directories). Failures are tracked until the configured timeout is exceeded. Once this timeout is exceeded, failover is triggered by shutting down MarkLogic Server.

Since a failure will trigger a host shutdown, if any database has Shutdown on Storage Failure enabled, then all forests that need to have high availability should have forest-level replication configured.

Section 508 accessibility improvements

The MarkLogic 12.0 Admin, Query Console, Monitoring Dashboard, and Monitoring History user interface includes major improvements towards level A and AA 508 accessibility compliance. This effort reflects our commitment to improving the accessibility of the MarkLogic Server user interfaces.

Enhanced usability and performance of the Admin Interface status pages

The Admin Interface now uses Kendo Data Grids for most of the tables on the status pages. This upgrade enhances the readability and usability of the status pages and allows users to refresh individual tables without reloading the entire page.

Here are the new table features:

1. Search: Begin typing in the search box to find any data in the entire table, not in just the visible part of the table.

2. Refresh: Refresh just this Data Grid, not the entire status page.

3. Sort: Click any column to sort the table in either ascending or descending order by that column.

4. Page navigator (available for tables displaying more than 1 page): Choose the page to display.

5. Items per page (available for tables with more than 10 rows): Choose the number of table rows to display: 10, 25, or 50.

6. Total: See the totals based on the records filtered by Search.

New modal confirmation dialogs in the Admin Interface

MarkLogic 12.0 replaces outdated confirmation pages with modal confirmation dialog boxes, improving user experience and reducing vulnerability to attacks like CSRF (Cross-Site Request Forgery).

These confirmation dialog boxes include data to help users make sure that they do, indeed, want to take the action.

After the user clicks to confirm, the dialog box blurs to show that the action is in process and to prevent further user interaction.

A message confirming that the action is in process or has been completed also appears.

If an error occurs, then the dialog boxes provide information about the error, guidance for correcting it, and error information to provide to customer support.

Admin Interface disabled during rolling upgrades

To prevent configuration changes during rolling upgrades, the Admin Interface is disabled on every host within the upgrading cluster.

To make obvious the fact that the Admin Interface is disabled, MarkLogic 12.0 grays out the Admin Interface on these hosts and disables all form fields and confirmation box OK buttons.

In addition, a bell icon with a badge appears on these hosts' status bars. Clicking the bell icon displays a message explaining why the Admin Interface is disabled.

Admin Interface includes index memory statistics

For optimal performance, MarkLogic Server leverages memory-mapped data structures for a number of system index types. To track which indexes are using system resources and to plan for future resource requirements, database administrators need to know how much memory their various configured indexes are using.

MarkLogic 12.0 adds the Show index memory usage button to access Kendo Data Grids that detail lexicon and range index memory usage. Using this information, database administrators can see the size on disk as well as in memory of each lexicon and range index configured in the database. The button appears on these status pages:

• Database Status (Server > Databases > (Choose database) > Status tab)

• Database-Forest Status (Server > Databases > (Choose database) > Forest Status tab)

• Forest Status (Server > Forests > (Choose forest) > Status tab)

Button access rather than scrolling access quickly provides this information while maintaining Admin Interface performance and simplicity.

JSON Web Key Set (JWKS) support

MarkLogic 12.0 can now automatically download, validate, and update OAuth JSON Web Key Set (JWKS) keys through a JWKS endpoint URL. Providing a JWKS endpoint URL through the new OAuth configuration field OAuth JWKS URI allows MarkLogic Server to automatically download, validate, and update JWKS keys, eliminating the need to manually provide them.

Import password-protected encrypted certificates

MarkLogic 12.0 adds the ability to import encrypted certificates by allowing a passphrase to be entered when importing.

A multi-step wizard approach simplifies importing each type of signed certificate into MarkLogic Server.

Upgraded OpenSSL

MarkLogic Server 12.0 has been updated to use OpenSSL 3.3.3. This update ensures continued compliance with modern security standards, including support for TLS 1.3 and FIPS 140-3.

This update renders LDAP MD5 bind authentication obsolete. See LDAP MD5 bind is no longer supported.

TLS 1.3 support

MarkLogic 12.0 now supports Transport Layer Security (TLS) 1.3, the latest version of the TLS protocol. This enhancement brings stronger security, improved performance, and streamlined handshakes for encrypted communications.

This update aligns MarkLogic Server with modern security standards and helps organizations meet compliance requirements while delivering a more secure and efficient experience.

Upgraded document filter libraries

MarkLogic 12.0 includes an upgrade to the libraries behind the document filter capability that provides text extraction for 1000+ binary file formats. This update enhances security, stability, and maintainability to ensure compatibility with future updates.

As part of this update, MarkLogic 12.0 also removes the legacy document conversion capability. See Document conversion functions and CPF pipelines have been removed.

Updated operating system support

MarkLogic 11.3.1 introduced support for Red Hat Enterprise Linux 9, Amazon Linux 2023, and Windows Server 2022. MarkLogic 12.0 is certified on these operating systems as well, but support for RHEL 7 has been dropped.

In MarkLogic 11, there is a separate RPM for RHEL 9 versus the RPM for RHEL 7 and 8. In MarkLogic 12.0, there is a single RPM that now supports RHEL 8 and 9. There is no RPM available for RHEL 7.

The startup scripts for MarkLogic 12.0 on RHEL 8 and 9 are now based on systemd, so the commands for starting, stopping, and managing MarkLogic Server are different than when running MarkLogic 11 on RHEL 8.

Accordingly, the service management scripts and service files have also changed:

Renamed AWS and Azure marketplace offerings

The Amazon Web Service (AWS) and Azure marketplace naming conventions for MarkLogic Server offerings have been updated to make them more clear. The naming conventions now make it clear which offerings require a license to be purchased separately (Bring Your Own License or BYOL) and which will be billed through the AWS Marketplace (Pay as You Go or PAYG).

AWS Marketplace

Azure Marketplace

New function xdmp.nodeGetInfo()

A new function xdmp.nodeGetInfo() (xdmp:node-get-info() in XQuery) was added that returns information about a given node. This function can tell you useful information such as whether the node is from a document in the database or a constructed node and how big the document node is in the compressed tree.