Issues fixed in MarkLogic Server 10.0-7

redaction on rows in the Optic API

An Optic query can redact a column by rebinding a column to an expression. The expression can either transform the column values or generate replacement values in some other way including based on random numbers or UUIDs. The Optic API now provides helper functions to build column rebindings for common redaction cases including maskDeterministic(), maskRandom(), redactDatetime(), redactEmail(), redactIpv4(), redactNumber(), redactRegex(), redactUsSsn(), and redactUsPhone().

Optic bindAs() deprecated in favor of bind()

The bindAs() operation adds a single column to a row. The new bind() operation takes multiple column definitions from as() and thus makes it possible to add multiple columns to a row and to use helper functions to define columns.

External Authenticate trace messages are not accurate in certain cases

The External Authenticate trace message with "bad password" is overly specific when the cause is a general authentication failure

MLCP dependencies have security vulnerablities

The following dependencies which have security vunlnerabilities have been updated: - Apache Hadoop: from 2.6.0 to 2.7.2 - junit:junit: from 4.12.0 to 4.13.1 - com.thoughtworks.xstream:xstream: from 1.4.12 to 1.4.16 - org.apache.libthrift:libthrift: from 0.13.0 to 0.14.0 - com.google.guava:guava: from 11.0.2 to 25.0-jre. - commons-httpclient: from 3.1 to 4.2.3 - jackson-mapper-asl: removed

XStream vulnerabilities in MLCP

Recent vulnerabilities (CVE-2020-26258 and CVE-2020-26259) were addressed in the 1.4.15 build of XStream on 13th December 2020. As XStream is an MLCP dependency, this release contains the fixed XStream library. More information can be found at http://x-stream.github.io/

MLCP throws IOExceptions during scale-out events

During scale-out events, MLCP throws IOExceptions if some of the splits have already finished importing.

SVC-EXTIME during Security upgrade

Under some rare circumstances a security upgrade will exceed the current time limit (600 seconds) specified for the Admin App Server on port 8001

groupToArrays() and groupByUnion() in Optic

The groupToArrays() operation in the Optic SJS library has been fixed to associate group rows with the correct group keys. The export of queries with the groupToArrays() and groupByUnion() operations in the Optic SJS library has been fixed to export correctly so the query can be imported or sent to the REST API.

Query Console: Importing a workspace puts the list in random order

Importing a workspace in Query Console puts the list in random order

Query-Based Views in the Optic and REST APIs

The Optic and REST APIs can now generate a QBV (Query-Based View) for persistence in the schemas database. The QBV is accessed in SQL or the Optic API in the same way as a view specified in a TDE. The Optic API produces the view with the generateView() method in SJS or the op:generate-view() function in XQuery. The /v1/rows endpoint of the REST API adds the generateView value to the enumeration of the output parameter.

Deploy roles report OK, while role is not created

In some situations /manage/v3 is reporting HTTP OK (200) for role deployment, while the role is not created because an underlying role does not exist.

Database Replication cycling between sync & async

Database Replication replica forests get stuck cycling between the sync & async states.

New operations op:exists-join() and op:not-exists-join()

These are equivalent to SQL semi join and anti join.

Optic import of random() and uuid() expressions

Previously, an Optic import of a query using sem.random(), sem.uuid(), sql.rand(), or xdmp.random() expressions called the function when building the query. As a result, the random value was a constant in every row of the result. Now, the Optic import calls such functions when executing the query so the random value is different in every row.

Ambiguous column error with IBM Information Analyzer

Binding a column in SQL selects, SPARQL, or Optic API with the same column name as an existing column results in ambiguous column error.

deadlock when reading document in multistatement transaction

The REST API and thus the Java and Node.js APIs can read a document for a request that's part of a multi-statement transaction. The REST API locked the document for shared access and as a result, different multistatement transactions could become deadlocked when trying to upgrade the shared lock to an exclusive lock.

500 error on /manage/v2/databases/database-name?view=status failure

500 error with message-code XDMP-AS on /manage/v2/databases/database-name?view=status failure when flexible replication is configured

SQL empty string

Some SQL queries do not correctly distinguish empty strings

Managed hosts in same AZ may mount incorrect data volume

When two hosts in the same AZ are restarted at around the same time, it is possible that the hosts will inadvertently swap data volumes, causing XDMP-WRONGHOST errors.

Call to cts:triple-range-query() causing Bad malloc

If too many values are passed to cts:triple-range-query(), a Critical error displaying "Bad malloc" may occur.

XCC closes connections instead of reusing

When XCC talks to load balancer/application gateway using HTTP 1.1, the keep-alive timeout header from the server is not forwarded. XCC will assign a zero timeout and close the connection, and the connection pool will not reuse connections.

loop on start: "let: not found"

When starting, if the mlcmd initialization script is executed in the XSH shell environment, it get stuck in an infinite loop: ... Detecting eth1 from operating system ... Waiting for eth1 to be plugged in. Retry 1/20 ... [/opt/MarkLogic/mlcmd/scripts/init-config.xsh line: 137] let: not found

Memory Leak when using Kerberos Authentication in 10.0-5.2 or later

A memory leak occurs with every Kerberos authenticated login.

Forest should log error if label file has invalid journal archive path

When forest label file has an invalid journal archive path, no Warning or Error is shown and the forest cannot be open the next time it is restarted.

Forced reindexing slow

In some circumstances, a forced reindexing can be slow

Examples in the POST /manage/v3 documentation do not work

Some of the documentation examples for /manage/v3 gave errors

cts:collection-match() performance

cts:collection-match() performs slow for large sets of documents (of the order of millions of documents)

LEFT OUTER JOIN in SQL or Optic with ORDER BY

Not all query plans generated from a LEFT OUTER JOIN in SQL or Optic with ORDER BY returns correct results; When the bug occurs, no results are found, despite there being valid results.

sampling in groupToArrays()

As with groupBy(), the Optic groupToArrays() operation takes grouping keys and aggregates. Unlike groupBy(), the initial version of groupToArrays() only supported aggregate functions and not implicit sampling of columns. Now, the list of aggregates passed to groupToArrays() can include columns for implicit sampling.

language-baseline not supported in manage/v2/properties endpoint

get and set of language-baseline are accepted by manage/v2/properties endpoint (GET/PUT) but have no effect

Monitoring Dashboard: Numeric values in excel file exported in string format

Monitoring Dashboard: Numeric values in excel file exported as string or "undefined" if not available

HTTP request to the server errors out when a request header contains multiple encoded words

The server responds to an HTTP request containing a request header that has multiple encoded words (E.g: ?UTF-8? Encoding) with a 400 error code.

xdmp:email can't decode diacritic characters correctly

xdmp:email can't decode diacritic characters correctly when user doesn't specify content-type.

JavaScript default app-server modules not supported

JavaScript default modules index.sjs and default.sjs weren't invoked by default

Restore validation of App-Services forest from different cluster.

Restore validation fails for restore operation of App-Services forest (by name) from a different cluster (by ID).

support for namedGroup() in facetBy()

The Optic API supports a facetBy() convenience for grouping to arrays. Previously, each facet had a system-assigned name. Now, the system-assigned name is a default that can be overridden by supplying a namedGroup() for the facet.

XDMP-TOOMANYARGS when changing modules database for triggers using Admin UI

After restoring triggers from a different cluster, the modules database ids do not match anymore on the new cluster, which is expected. On Admin UI screen under Databases-{db-name}->Triggers, it provides a button to map the unknown id to a database from the current cluster. However, when this is clicked, XDMP-TOOMANYARGS is thrown.

AssertionConsumerServiceURL has "http://" or https://", and port irrespective of value in SAML Assertion host.

If SAML Assertion host starts with "http://" or https://", AssertionConsumerServiceURL should not contain http:// or port information.