Default Stemming and Tokenization Libraries Changed for Most Languages

In MarkLogic 9 and later, the default tokenization and stemming libraries have been changed for all languages (except English tokenization). Consequently, some tokenization and stemming behavior changed between MarkLogic 8 and MarkLogic 9. In most cases, stemming and tokenization will be more precise in MarkLogic 9 and later.

You can now customize stemming and tokenization for a language by selecting one of several built-in stemmer and lexer plugins, or by creating your own stemmer and lexer plugins in C++.

You can also configure a custom stemming dictionary, a custom tokenization dictionary, or both for any language. Previously, you could not distinguish between stemming and tokenization dictionaries, and you could not install a dictionary for an unsupported language.

If you upgrade to MarkLogic 9 or later from an earlier version of MarkLogic Server, then your installation will continue to use the legacy stemming and tokenization libraries as the language baseline. Any fresh installation of MarkLogic Server will use the new libraries. You can change the baseline configuration using admin:cluster-set-language-baseline.

Unless you use the legacy language baseline, reindexing is required for content in the following languages:

• Chinese

• Danish

• Dutch, if you want to query with decompounding

• Finnish

• German

• Hungarian

• Japanese

• Korean, unless you use decompounding

• Norwegian (Bokmal and Nynorsk) if you want to query with decompounding

• Norwegian (generic ‘no’ lang code), though use of generic ‘no’ is not recommended

• Romanian

• Russian

• Swedish, if you want to query with decompounding

• Tamil

• Turkish

For other languages (except English), you might be able to avoid incompatibilities depending on the nature of your queries, but reindexing is still strongly recommended.

Tokenization and stemming are significantly different for Japanese. Tokenization is significantly different for Chinese (both simplified and traditional). The impact on other languages is more nuanced, but should lead to better results, overall. You might observe some relevance score changes on stemmed searches due to the higher degree of precision. If you require low-level details about the impact on a specific language and you have an active maintenance contract, you can contact MarkLogic Technical Support.

For more details on incompatibilities related to the changes to stemming and tokenization, see MarkLogic Server v9 Tokenization and Stemming from MarkLogic Technical Support.

For more details on stemming and tokenization, see these topics in Develop Search Applications:

• Tokenization and Stemming
• User-Defined Lexer Plugins
• Using a User-Defined Stemmer Plugin
• Custom Dictionaries for Tokenizing and Stemming