C# Class Lucene.Net.Analysis.CommonGrams.CommonGramsFilter

Inheritance: TokenFilter

ファイルを表示 Open project: apache/lucenenet Class Usage Examples

Private Properties

Property	Type	Description
GramToken	void
SaveTermBuffer	void

Public Methods

Method	Description
CommonGramsFilter ( LuceneVersion matchVersion, TokenStream input, CharArraySet commonWords ) : Lucene.Net.Analysis.Tokenattributes	Construct a token stream filtering the given input using a Set of common words to create bigrams. Outputs both unigrams with position increment and bigrams with position increment 0 type=gram where one or both of the words in a potential bigram are in the set of common words .
IncrementToken ( ) : bool	Inserts bigrams for common words into a token stream. For each input token, output the token. If the token and/or the following token are in the list of common words also output a bigram with position increment 0 and type="gram" TODO:Consider adding an option to not emit unigram stopwords as in CDL XTF BigramStopFilter, CommonGramsQueryFilter would need to be changed to work with this. TODO: Consider optimizing for the case of three commongrams i.e "man of the year" normally produces 3 bigrams: "man-of", "of-the", "the-year" but with proper management of positions we could eliminate the middle bigram "of-the"and save a disk seek and a whole set of position lookups.
Reset ( ) : void	{@inheritDoc}

Private Methods

Method	Description
GramToken ( ) : void	Constructs a compound token.
SaveTermBuffer ( ) : void	Saves this information to form the left part of a gram

Method Details

CommonGramsFilter() public method

Construct a token stream filtering the given input using a Set of common words to create bigrams. Outputs both unigrams with position increment and bigrams with position increment 0 type=gram where one or both of the words in a potential bigram are in the set of common words .

public CommonGramsFilter ( LuceneVersion matchVersion, TokenStream input, CharArraySet commonWords ) : Lucene.Net.Analysis.Tokenattributes
matchVersion	LuceneVersion
input	TokenStream	TokenStream input in filter chain
commonWords	CharArraySet	The set of common words.
return	Lucene.Net.Analysis.Tokenattributes

IncrementToken() public method

Inserts bigrams for common words into a token stream. For each input token, output the token. If the token and/or the following token are in the list of common words also output a bigram with position increment 0 and type="gram" TODO:Consider adding an option to not emit unigram stopwords as in CDL XTF BigramStopFilter, CommonGramsQueryFilter would need to be changed to work with this. TODO: Consider optimizing for the case of three commongrams i.e "man of the year" normally produces 3 bigrams: "man-of", "of-the", "the-year" but with proper management of positions we could eliminate the middle bigram "of-the"and save a disk seek and a whole set of position lookups.

public IncrementToken ( ) : bool
return	bool

Reset() public method

{@inheritDoc}

public Reset ( ) : void
return	void