C# Class Lucene.Net.Analysis.Shingle.ShingleFilter

A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.

For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".

This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_"). It does not handle a position increment of 0.

Inheritance: Lucene.Net.Analysis.TokenFilter

Exibir arquivo Open project: apache/lucenenet Class Usage Examples

Public Methods

Method	Description
End ( ) : void
IncrementToken ( ) : bool
Reset ( ) : void
ShingleFilter ( TokenStream input ) : System	Construct a ShingleFilter with default shingle size: 2.
ShingleFilter ( TokenStream input, int maxShingleSize ) : System	Constructs a ShingleFilter with the specified shingle size from the TokenStream `input`
ShingleFilter ( TokenStream input, int minShingleSize, int maxShingleSize ) : System	Constructs a ShingleFilter with the specified shingle size from the TokenStream `input`
ShingleFilter ( TokenStream input, string tokenType ) : System	Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2

Private Methods

Method	Description
GetNextToken ( InputWindowToken target ) : InputWindowToken	Get the next token from the input stream. If the next token has `positionIncrement > 1`, `positionIncrement - 1` #fillerTokens are inserted first.
ShiftInputWindow ( ) : void	Fills #inputWindow with input stream tokens, if available, shifting to the right if the window was previously full. Resets #gramSize to its minimum value.

Method Details

End() public method

public End ( ) : void
return	void

IncrementToken() public method

public IncrementToken ( ) : bool
return	bool

Reset() public method

public Reset ( ) : void
return	void

ShingleFilter() public method

Construct a ShingleFilter with default shingle size: 2.

public ShingleFilter ( TokenStream input ) : System
input	Lucene.Net.Analysis.TokenStream	input stream
return	System

ShingleFilter() public method

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

public ShingleFilter ( TokenStream input, int maxShingleSize ) : System
input	Lucene.Net.Analysis.TokenStream	input stream
maxShingleSize	int	maximum shingle size produced by the filter.
return	System

ShingleFilter() public method

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

public ShingleFilter ( TokenStream input, int minShingleSize, int maxShingleSize ) : System
input	Lucene.Net.Analysis.TokenStream	input stream
minShingleSize	int	minimum shingle size produced by the filter.
maxShingleSize	int	maximum shingle size produced by the filter.
return	System

ShingleFilter() public method

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2

public ShingleFilter ( TokenStream input, string tokenType ) : System
input	Lucene.Net.Analysis.TokenStream	input stream
tokenType	string	token type for shingle tokens
return	System