C# Class Lucene.Net.Analysis.Shingle.ShingleFilter

A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.

For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".

This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_"). It does not handle a position increment of 0.

Inheritance: Lucene.Net.Analysis.TokenFilter
Afficher le fichier Open project: apache/lucenenet Class Usage Examples

Méthodes publiques

Méthode Description
End ( ) : void
IncrementToken ( ) : bool
Reset ( ) : void
ShingleFilter ( TokenStream input ) : System

Construct a ShingleFilter with default shingle size: 2.
ShingleFilter ( TokenStream input, int maxShingleSize ) : System

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
ShingleFilter ( TokenStream input, int minShingleSize, int maxShingleSize ) : System

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
ShingleFilter ( TokenStream input, string tokenType ) : System

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2

Private Methods

Méthode Description
GetNextToken ( InputWindowToken target ) : InputWindowToken

Get the next token from the input stream.

If the next token has positionIncrement > 1, positionIncrement - 1 #fillerTokens are inserted first.

ShiftInputWindow ( ) : void

Fills #inputWindow with input stream tokens, if available, shifting to the right if the window was previously full.

Resets #gramSize to its minimum value.

Method Details

End() public méthode

public End ( ) : void
Résultat void

IncrementToken() public méthode

public IncrementToken ( ) : bool
Résultat bool

Reset() public méthode

public Reset ( ) : void
Résultat void

ShingleFilter() public méthode

Construct a ShingleFilter with default shingle size: 2.
public ShingleFilter ( TokenStream input ) : System
input Lucene.Net.Analysis.TokenStream input stream
Résultat System

ShingleFilter() public méthode

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
public ShingleFilter ( TokenStream input, int maxShingleSize ) : System
input Lucene.Net.Analysis.TokenStream input stream
maxShingleSize int maximum shingle size produced by the filter.
Résultat System

ShingleFilter() public méthode

Constructs a ShingleFilter with the specified shingle size from the TokenStream input
public ShingleFilter ( TokenStream input, int minShingleSize, int maxShingleSize ) : System
input Lucene.Net.Analysis.TokenStream input stream
minShingleSize int minimum shingle size produced by the filter.
maxShingleSize int maximum shingle size produced by the filter.
Résultat System

ShingleFilter() public méthode

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2
public ShingleFilter ( TokenStream input, string tokenType ) : System
input Lucene.Net.Analysis.TokenStream input stream
tokenType string token type for shingle tokens
Résultat System