C# Class org.apache.lucene.analysis.ngram.NGramTokenizer

Tokenizes the input into n-grams of the given size(s).

On the contrary to NGramTokenFilter, this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.

For example, "abcde" would be tokenized as (minGram=2, maxGram=3):

Termababcbcbcdcdcdede
Position increment1111111
Position length1111111
Offsets[0,2[[0,3[[1,3[[1,4[[2,4[[2,5[[3,5[

This tokenizer changed a lot in Lucene 4.4 in order to:

  • tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
  • count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
  • give the ability to #isTokenChar(int) pre-tokenize the stream before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Although highly discouraged, it is still possible to use the old behavior through Lucene43NGramTokenizer.

Inheritance: Tokenizer
Afficher le fichier Open project: paulirwin/lucene.net Class Usage Examples

Méthodes publiques

Méthode Description
NGramTokenizer ( System.Version version, AttributeFactory factory, Reader input, int minGram, int maxGram ) : System

Creates NGramTokenizer with given min and max n-grams.
NGramTokenizer ( System.Version version, Reader input ) : System

Creates NGramTokenizer with default min and max n-grams.
NGramTokenizer ( System.Version version, Reader input, int minGram, int maxGram ) : System

Creates NGramTokenizer with given min and max n-grams.
end ( ) : void
incrementToken ( ) : bool
reset ( ) : void

Méthodes protégées

Méthode Description
isTokenChar ( int chr ) : bool

Only collect characters which satisfy this condition.

Private Methods

Méthode Description
NGramTokenizer ( System.Version version, AttributeFactory factory, Reader input, int minGram, int maxGram, bool edgesOnly ) : System
NGramTokenizer ( System.Version version, Reader input, int minGram, int maxGram, bool edgesOnly ) : System
consume ( ) : void

Consume one code point.
init ( System.Version version, int minGram, int maxGram, bool edgesOnly ) : void
updateLastNonTokenChar ( ) : void

Method Details

NGramTokenizer() public méthode

Creates NGramTokenizer with given min and max n-grams.
public NGramTokenizer ( System.Version version, AttributeFactory factory, Reader input, int minGram, int maxGram ) : System
version System.Version the lucene compatibility version
factory AttributeFactory to use
input Reader holding the input to be tokenized
minGram int the smallest n-gram to generate
maxGram int the largest n-gram to generate
Résultat System

NGramTokenizer() public méthode

Creates NGramTokenizer with default min and max n-grams.
public NGramTokenizer ( System.Version version, Reader input ) : System
version System.Version the lucene compatibility version
input Reader holding the input to be tokenized
Résultat System

NGramTokenizer() public méthode

Creates NGramTokenizer with given min and max n-grams.
public NGramTokenizer ( System.Version version, Reader input, int minGram, int maxGram ) : System
version System.Version the lucene compatibility version
input Reader holding the input to be tokenized
minGram int the smallest n-gram to generate
maxGram int the largest n-gram to generate
Résultat System

end() public méthode

public end ( ) : void
Résultat void

incrementToken() public méthode

public incrementToken ( ) : bool
Résultat bool

isTokenChar() protected méthode

Only collect characters which satisfy this condition.
protected isTokenChar ( int chr ) : bool
chr int
Résultat bool

reset() public méthode

public reset ( ) : void
Résultat void