C# 클래스 Lucene.Net.Analysis.Ngram.NGramTokenizer

Tokenizes the input into n-grams of the given size(s).

On the contrary to NGramTokenFilter, this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.

For example, "abcde" would be tokenized as (minGram=2, maxGram=3):

Termababcbcbcdcdcdede
Position increment1111111
Position length1111111
Offsets[0,2[[0,3[[1,3[[1,4[[2,4[[2,5[[3,5[

This tokenizer changed a lot in Lucene 4.4 in order to:

  • tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
  • count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
  • give the ability to #isTokenChar(int) pre-tokenize the stream before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Although highly discouraged, it is still possible to use the old behavior through Lucene43NGramTokenizer.

상속: Tokenizer
파일 보기 프로젝트 열기: apache/lucenenet 1 사용 예제들

공개 메소드들

메소드 설명
End ( ) : void
IncrementToken ( ) : bool
NGramTokenizer ( LuceneVersion version, AttributeFactory factory, TextReader.TextReader input, int minGram, int maxGram ) : Lucene.Net.Analysis.Tokenattributes

Creates NGramTokenizer with given min and max n-grams.
NGramTokenizer ( LuceneVersion version, TextReader.TextReader input ) : Lucene.Net.Analysis.Tokenattributes

Creates NGramTokenizer with default min and max n-grams.
NGramTokenizer ( LuceneVersion version, TextReader.TextReader input, int minGram, int maxGram ) : Lucene.Net.Analysis.Tokenattributes

Creates NGramTokenizer with given min and max n-grams.
Reset ( ) : void

보호된 메소드들

메소드 설명
IsTokenChar ( int chr ) : bool

Only collect characters which satisfy this condition.

비공개 메소드들

메소드 설명
Init ( LuceneVersion version, int minGram, int maxGram, bool edgesOnly ) : void
NGramTokenizer ( LuceneVersion version, AttributeFactory factory, TextReader.TextReader input, int minGram, int maxGram, bool edgesOnly ) : Lucene.Net.Analysis.Tokenattributes
NGramTokenizer ( LuceneVersion version, TextReader.TextReader input, int minGram, int maxGram, bool edgesOnly ) : Lucene.Net.Analysis.Tokenattributes
UpdateLastNonTokenChar ( ) : void
consume ( ) : void

Consume one code point.

메소드 상세

End() 공개 메소드

public End ( ) : void
리턴 void

IncrementToken() 공개 최종 메소드

public final IncrementToken ( ) : bool
리턴 bool

IsTokenChar() 보호된 메소드

Only collect characters which satisfy this condition.
protected IsTokenChar ( int chr ) : bool
chr int
리턴 bool

NGramTokenizer() 공개 메소드

Creates NGramTokenizer with given min and max n-grams.
public NGramTokenizer ( LuceneVersion version, AttributeFactory factory, TextReader.TextReader input, int minGram, int maxGram ) : Lucene.Net.Analysis.Tokenattributes
version LuceneVersion the lucene compatibility version
factory AttributeFactory to use
input System.IO.TextReader.TextReader holding the input to be tokenized
minGram int the smallest n-gram to generate
maxGram int the largest n-gram to generate
리턴 Lucene.Net.Analysis.Tokenattributes

NGramTokenizer() 공개 메소드

Creates NGramTokenizer with default min and max n-grams.
public NGramTokenizer ( LuceneVersion version, TextReader.TextReader input ) : Lucene.Net.Analysis.Tokenattributes
version LuceneVersion the lucene compatibility version
input System.IO.TextReader.TextReader holding the input to be tokenized
리턴 Lucene.Net.Analysis.Tokenattributes

NGramTokenizer() 공개 메소드

Creates NGramTokenizer with given min and max n-grams.
public NGramTokenizer ( LuceneVersion version, TextReader.TextReader input, int minGram, int maxGram ) : Lucene.Net.Analysis.Tokenattributes
version LuceneVersion the lucene compatibility version
input System.IO.TextReader.TextReader holding the input to be tokenized
minGram int the smallest n-gram to generate
maxGram int the largest n-gram to generate
리턴 Lucene.Net.Analysis.Tokenattributes

Reset() 공개 메소드

public Reset ( ) : void
리턴 void