C# Class Yamool.CWSharp.BigramTokenizer

The standard bigram tokenizer that converting text into a sequence of tokens.
Inheritance: ITokenizer
Mostra file Open project: zhengchun/CWSharp

Public Methods

Method Description
BigramTokenizer ( ) : System
Traverse ( string text ) : IEnumerable

Method Details

BigramTokenizer() public method

public BigramTokenizer ( ) : System
return System

Traverse() public method

public Traverse ( string text ) : IEnumerable
text string
return IEnumerable