C# (CSharp) org.apache.lucene.analysis.ngram Namespace

Although highly discouraged, it is still possible to use the old behavior through Lucene43EdgeNGramTokenizer.

You must specify the required Version compatibility when creating a NGramTokenFilter. As of Lucene 4.4, this token filters:

• handles supplementary characters correctly,
• emits all n-grams for the same token at the same position,
• does not modify offsets,
• sorts n-grams by their offset in the original token first, then increasing length (meaning that "abc" will give "a", "ab", "abc", "b", "bc", "c").

You can make this filter use the old behavior by providing a version < Version#LUCENE_44 in the constructor but this is not recommended as it will lead to broken TokenStreams that will cause highlighting bugs.

If you were using this TokenFilter to perform partial highlighting, this won't work anymore since this filter doesn't update offsets. You should modify your analysis chain to use NGramTokenizer, and potentially override NGramTokenizer#isTokenChar(int) to perform pre-tokenization.

This tokenizer changed a lot in Lucene 4.4 in order to:

• tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
• count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
• give the ability to #isTokenChar(int) pre-tokenize the stream before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Although highly discouraged, it is still possible to use the old behavior through Lucene43NGramTokenizer.