C# Class SILUBS.ScriptureChecks.UnitTestTokenizer

Afficher le fichier Open project: sillsdev/FieldWorks

Méthodes publiques

Méthode Description
Tokenize ( string text ) : List

Split text for book into TextTokens. Populate Tokens and chapters.

Private Methods

Méthode Description
CategorizeTokens ( List tokens ) : void
DivideText ( string text ) : List

Divide text for book into TextTokens. Set Offset, Length, BookText, AnnotationOffset, Chapter, Verse Tricky things needing done: 1) Split \v N abc... into two tokens, first containing just verse number 2) \f X abc... don't return caller as part of the token

GetCVNumber ( string text, int offset ) : string

Return the text of a chapter or verse number starting at the specified offset.

IsParagraphStart ( string marker ) : bool

Determines whether the specified marker is a paragraph start marker.

NextToken ( string text, int ind ) : UnitTestUSFMTextToken

Method Details

Tokenize() public méthode

Split text for book into TextTokens. Populate Tokens and chapters.
public Tokenize ( string text ) : List
text string
Résultat List