C# Class Lucene.Net.Analysis.Cjk.CJKBigramFilter

Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

CJK types are set by these tokenizers, but you can also use #CJKBigramFilter(TokenStream, int) to explicitly control which of the CJK scripts are turned into bigrams.

By default, when a CJK character has no adjacent characters to form a bigram, it is output in unigram form. If you want to always output both unigrams and bigrams, set the outputUnigrams flag in CJKBigramFilter#CJKBigramFilter(TokenStream, int, boolean). This can be used for a combined unigram+bigram approach.

In all cases, all non-CJK input is passed thru unmodified.

Inheritance: TokenFilter

Show file Open project: apache/lucenenet Class Usage Examples

Private Properties

Property	Type	Description
DoNext	bool
FlushBigram	void
FlushUnigram	void
Refill	void

Public Methods

Method	Description
CJKBigramFilter ( TokenStream @in ) : Lucene.Net.Analysis.Standard	Calls {@link CJKBigramFilter#CJKBigramFilter(TokenStream, int) CJKBigramFilter(in, HAN \| HIRAGANA \| KATAKANA \| HANGUL)}
CJKBigramFilter ( TokenStream @in, int flags ) : Lucene.Net.Analysis.Standard	Calls {@link CJKBigramFilter#CJKBigramFilter(TokenStream, int, boolean) CJKBigramFilter(in, flags, false)}
CJKBigramFilter ( TokenStream @in, int flags, bool outputUnigrams ) : Lucene.Net.Analysis.Standard	Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.
IncrementToken ( ) : bool
Reset ( ) : void

Private Methods

Method	Description
DoNext ( ) : bool	looks at next input token, returning false is none is available
FlushBigram ( ) : void	Flushes a bigram token to output from our buffer This is the normal case, e.g. ABC -> AB BC
FlushUnigram ( ) : void	Flushes a unigram token to output from our buffer. This happens when we encounter isolated CJK characters, either the whole CJK string is a single character, or we encounter a CJK character surrounded by space, punctuation, english, etc, but not beside any other CJK.
Refill ( ) : void	refills buffers with new data from the current token.

Method Details

CJKBigramFilter() public method

Calls {@link CJKBigramFilter#CJKBigramFilter(TokenStream, int) CJKBigramFilter(in, HAN | HIRAGANA | KATAKANA | HANGUL)}

public CJKBigramFilter ( TokenStream @in ) : Lucene.Net.Analysis.Standard
@in	TokenStream
return	Lucene.Net.Analysis.Standard

CJKBigramFilter() public method

Calls {@link CJKBigramFilter#CJKBigramFilter(TokenStream, int, boolean) CJKBigramFilter(in, flags, false)}

public CJKBigramFilter ( TokenStream @in, int flags ) : Lucene.Net.Analysis.Standard
@in	TokenStream
flags	int
return	Lucene.Net.Analysis.Standard

CJKBigramFilter() public method

Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.

public CJKBigramFilter ( TokenStream @in, int flags, bool outputUnigrams ) : Lucene.Net.Analysis.Standard
@in	TokenStream
flags	int	OR'ed set from , , /// ,
outputUnigrams	bool	true if unigrams for the selected writing systems should also be output. /// when this is false, this is only done when there are no adjacent characters to form /// a bigram.
return	Lucene.Net.Analysis.Standard

IncrementToken() public method

public IncrementToken ( ) : bool
return	bool

Reset() public method

public Reset ( ) : void
return	void