C# Class Lucene.Net.Analysis.Shingle.ShingleFilter

A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.

For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".

This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_"). It does not handle a position increment of 0.

Inheritance: Lucene.Net.Analysis.TokenFilter
Mostra file Open project: apache/lucenenet Class Usage Examples

Public Methods

Method Description
End ( ) : void
IncrementToken ( ) : bool
Reset ( ) : void
ShingleFilter ( TokenStream input ) : System

Construct a ShingleFilter with default shingle size: 2.
ShingleFilter ( TokenStream input, int maxShingleSize ) : System

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
ShingleFilter ( TokenStream input, int minShingleSize, int maxShingleSize ) : System

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
ShingleFilter ( TokenStream input, string tokenType ) : System

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2

Private Methods

Method Description
GetNextToken ( InputWindowToken target ) : InputWindowToken

Get the next token from the input stream.

If the next token has positionIncrement > 1, positionIncrement - 1 #fillerTokens are inserted first.

ShiftInputWindow ( ) : void

Fills #inputWindow with input stream tokens, if available, shifting to the right if the window was previously full.

Resets #gramSize to its minimum value.

Method Details

End() public method

public End ( ) : void
return void

IncrementToken() public method

public IncrementToken ( ) : bool
return bool

Reset() public method

public Reset ( ) : void
return void

ShingleFilter() public method

Construct a ShingleFilter with default shingle size: 2.
public ShingleFilter ( TokenStream input ) : System
input Lucene.Net.Analysis.TokenStream input stream
return System

ShingleFilter() public method

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
public ShingleFilter ( TokenStream input, int maxShingleSize ) : System
input Lucene.Net.Analysis.TokenStream input stream
maxShingleSize int maximum shingle size produced by the filter.
return System

ShingleFilter() public method

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
public ShingleFilter ( TokenStream input, int minShingleSize, int maxShingleSize ) : System
input Lucene.Net.Analysis.TokenStream input stream
minShingleSize int minimum shingle size produced by the filter.
maxShingleSize int maximum shingle size produced by the filter.
return System

ShingleFilter() public method

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2
public ShingleFilter ( TokenStream input, string tokenType ) : System
input Lucene.Net.Analysis.TokenStream input stream
tokenType string token type for shingle tokens
return System