C# (CSharp) Lucene.Net.Analysis.Miscellaneous Namespace

• C1 Controls and Latin-1 Supplement: http://www.unicode.org/charts/PDF/U0080.pdf
• Latin Extended-A: http://www.unicode.org/charts/PDF/U0100.pdf
• Latin Extended-B: http://www.unicode.org/charts/PDF/U0180.pdf
• Latin Extended Additional: http://www.unicode.org/charts/PDF/U1E00.pdf
• Latin Extended-C: http://www.unicode.org/charts/PDF/U2C60.pdf
• Latin Extended-D: http://www.unicode.org/charts/PDF/UA720.pdf
• IPA Extensions: http://www.unicode.org/charts/PDF/U0250.pdf
• Phonetic Extensions: http://www.unicode.org/charts/PDF/U1D00.pdf
• Phonetic Extensions Supplement: http://www.unicode.org/charts/PDF/U1D80.pdf
• General Punctuation: http://www.unicode.org/charts/PDF/U2000.pdf
• Superscripts and Subscripts: http://www.unicode.org/charts/PDF/U2070.pdf
• Enclosed Alphanumerics: http://www.unicode.org/charts/PDF/U2460.pdf
• Dingbats: http://www.unicode.org/charts/PDF/U2700.pdf
• Supplemental Punctuation: http://www.unicode.org/charts/PDF/U2E00.pdf
• Alphabetic Presentation Forms: http://www.unicode.org/charts/PDF/UFB00.pdf
• Halfwidth and Fullwidth Forms: http://www.unicode.org/charts/PDF/UFF00.pdf

 <fieldType name="text_ascii" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.ASCIIFoldingFilterFactory" preserveOriginal="false"/> </analyzer> </fieldType>

This filter is particularly useful to build nice looking facet parameters. This filter is not appropriate if you intend to use a prefix query.

The factory takes parameters:
"onlyFirstWord" - should each word be capitalized or all of the words?
"keep" - a keep word list. Each word that should be kept separated by whitespace.
"keepIgnoreCase - true or false. If true, the keep list will be considered case-insensitive.
"forceFirstLetter" - Force the first letter to be capitalized even if it is in the keep list
"okPrefix" - do not change word capitalization if a word begins with something in this list. for example if "McK" is on the okPrefix list, the word "McKinley" should not be changed to "Mckinley"
"minWordLength" - how long the word needs to be to get capitalization applied. If the minWordLength is 3, "and" > "And" but "or" stays "or"
"maxWordCount" - if the token contains more then maxWordCount words, the capitalization is assumed to be correct.

 <fieldType name="text_cptlztn" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.CapitalizationFilterFactory" onlyFirstWord="true" keep="java solr lucene" keepIgnoreCase="false" okPrefix="McK McD McA"/>    </analyzer> </fieldType>

Note: Length is calculated as the number of Unicode codepoints.

 <fieldType name="text_lngth" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.CodepointCountFilterFactory" min="0" max="1" /> </analyzer> </fieldType>

 <fieldType name="text_hyphn" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.HyphenatedWordsFilterFactory"/> </analyzer> </fieldType>

 <fieldType name="text_keepword" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.KeepWordFilterFactory" words="keepwords.txt" ignoreCase="false"/> </analyzer> </fieldType>

 <fieldType name="text_keyword" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protectedkeyword.txt" pattern="^.+er$" ignoreCase="false"/> </analyzer> </fieldType>

Note: Length is calculated as the number of UTF-16 code units.

 <fieldType name="text_lngth" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.LengthFilterFactory" min="0" max="1" /> </analyzer> </fieldType>

By default, this filter ignores any tokens in the wrapped {@code TokenStream} once the limit has been reached, which can result in {@code reset()} being called prior to {@code incrementToken()} returning {@code false}. For most {@code TokenStream} implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a {@code TokenStream} which requires that the full stream of tokens be exhausted in order to function properly, use the #LimitTokenCountFilter(TokenStream,int,boolean) consumeAllTokens option.

By default, this filter ignores any tokens in the wrapped {@code TokenStream} once the limit has been exceeded, which can result in {@code reset()} being called prior to {@code incrementToken()} returning {@code false}. For most {@code TokenStream} implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a {@code TokenStream} which requires that the full stream of tokens be exhausted in order to function properly, use the #LimitTokenPositionFilter(TokenStream,int,boolean) consumeAllTokens option.

 <fieldType name="text_limit_pos" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.LimitTokenPositionFilterFactory" maxTokenPosition="3" consumeAllTokens="false" /> </analyzer> </fieldType>

The {@code consumeAllTokens} property is optional and defaults to {@code false}. See LimitTokenPositionFilter for an explanation of its use.

Example usage:

 {@code Map analyzerPerField = new HashMap<>(); analyzerPerField.put("firstname", new KeywordAnalyzer()); analyzerPerField.put("lastname", new KeywordAnalyzer()); PerFieldAnalyzerWrapper aWrapper = new PerFieldAnalyzerWrapper(new StandardAnalyzer(version), analyzerPerField); } 

In this example, StandardAnalyzer will be used for all fields except "firstname" and "lastname", for which KeywordAnalyzer will be used.

A PerFieldAnalyzerWrapper can be used like any other analyzer, for both indexing and query parsing.

NOTE: This filter might not behave correctly if used with custom Attributes, i.e. Attributes other than the ones located in org.apache.lucene.analysis.tokenattributes.

NOTE: This filter might not behave correctly if used with custom Attributes, i.e. Attributes other than the ones located in Lucene.Net.Analysis.TokenAttributes.

 <fieldType name="text_rmdup" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>

It's is a semantically more destructive solution than ScandinavianNormalizationFilter but can in addition help with matching raksmorgas as räksmörgås.

blåbærsyltetøj == blåbärsyltetöj == blaabaarsyltetoej == blabarsyltetoj räksmörgås == ræksmørgås == ræksmörgaos == raeksmoergaas == raksmorgas

Background: Swedish åäö are in fact the same letters as Norwegian and Danish åæø and thus interchangeable when used between these languages. They are however folded differently when people type them on a keyboard lacking these characters.

In that situation almost all Swedish people use a, a, o instead of å, ä, ö.

Norwegians and Danes on the other hand usually type aa, ae and oe instead of å, æ and ø. Some do however use a, a, o, oo, ao and sometimes permutations of everything above.

This filter solves that mismatch problem, but might also cause new.

 <fieldType name="text_scandfold" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.ScandinavianFoldingFilterFactory"/> </analyzer> </fieldType>

It's a semantically less destructive solution than ScandinavianFoldingFilter, most useful when a person with a Norwegian or Danish keyboard queries a Swedish index and vice versa. This filter does not the common Swedish folds of å and ä to a nor ö to o.

blåbærsyltetøj == blåbärsyltetöj == blaabaarsyltetoej but not blabarsyltetoj räksmörgås == ræksmørgås == ræksmörgaos == raeksmoergaas but not raksmorgas

 <fieldType name="text_scandnorm" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.ScandinavianNormalizationFilterFactory"/> </analyzer> </fieldType>

 <fieldType name="text_dicstem" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StemmerOverrideFilterFactory" dictionary="dictionary.txt" ignoreCase="false"/> </analyzer> </fieldType>

As of Lucene 4.4, this filter does not support updateOffsets=true anymore as it can lead to broken token streams.

 <fieldType name="text_trm" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.NGramTokenizerFactory"/> <filter class="solr.TrimFilterFactory" /> </analyzer> </fieldType>

• split on intra-word delimiters (by default, all non alpha-numeric characters): "Wi-Fi" → "Wi", "Fi"
• split on case transitions: "PowerShot" → "Power", "Shot"
• split on letter-number transitions: "SD500" → "SD", "500"
• leading and trailing intra-word delimiters on each subword are ignored: "//hello---there, 'dude'" → "hello", "there", "dude"
• trailing "'s" are removed for each subword: "O'Neil's" → "O", "Neil"
  • Note: this step isn't performed in a separate filter because of possible subword combinations.

• combinations="0" causes no subword combinations: "PowerShot" → 0:"Power", 1:"Shot" (0 and 1 are the token positions)
• combinations="1" means that in addition to the subwords, maximum runs of non-numeric subwords are catenated and produced at the same position of the last subword in the run:
  • "PowerShot" → 0:"Power", 1:"Shot" 1:"PowerShot"
  • "A's+B's&C's" -gt; 0:"A", 1:"B", 2:"C", 2:"ABC"
  • "Super-Duper-XL500-42-AutoCoder!" → 0:"Super", 1:"Duper", 2:"XL", 2:"SuperDuperXL", 3:"500" 4:"42", 5:"Auto", 6:"Coder", 6:"AutoCoder"

 <fieldType name="text_wd" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" protected="protectedword.txt" preserveOriginal="0" splitOnNumerics="1" splitOnCaseChange="1" catenateWords="0" catenateNumbers="0" catenateAll="0" generateWordParts="1" generateNumberParts="1" stemEnglishPossessive="1" types="wdfftypes.txt" /> </analyzer> </fieldType>