Solr CJK Parser

CJK with Solr for Libraries, part 1
http://discovery-grindstone.blogspot.com/2013/10/cjk-with-solr-for-libraries-part-1.html

http://www.cnblogs.com/nayitian/archive/2013/01/18/2866733.html
<fieldType name="text_ik" class="solr.TextField"> 
<analyzer type="index" isMaxWordLength="false" class="org.wltea.analyzer.lucene.IKAnalyzer"/> 
<analyzer type="query" isMaxWordLength="true" class="org.wltea.analyzer.lucene.IKAnalyzer"/> 
</fieldType>  

1.拷贝~/solr/contrib/analysis-extras/lucene-libs/lucene-analyzers-smartcn-4.0.0.jar到$CATALINA_HOME/webapps/solr/WEB-INF/lib/目录下
2.拷贝上述包中的stopwords.txt到$SOLR_HOME/collection1/conf/lang/stopwords_zh-cn.txt
3.修改$SOLR_HOME/collection1/conf/schema.xml文件，在<type></types>中增加如下内容:
<fieldType name="text_zh_smart" class="solr.TextField" positionIncrementGap="100">
 <analyzer>
<tokenizer class="solr.SmartChineseSentenceTokenizerFactory"/>
<filter class="solr.SmartChineseWordTokenFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PositionFilterFactory" />
<filter class="solr.StopFilterFactory" ignoreCase="false" words="lang/stopwords_zh-cn.txt" enablePositionIncrements="true"/>
 </analyzer>
</fieldType>

<fieldType name="text_zh_smart" class="solr.TextField" positionIncrementGap="100">
   <analyzer class="org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer"/>
</fieldType>

http://wiki.apache.org/solr/LanguageAnalysis
<analyzer class="org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer"/>
to configure your own analysis setup, use the SmartChineseSentenceTokenizerFactory along with your custom filter setup. The sentence tokenizer tokenizes on sentence boundaries and the SmartChineseWordTokenFilter breaks this further up into words.
<analyzer>
<tokenizer class="solr.SmartChineseSentenceTokenizerFactory"/>
<filter class="solr.SmartChineseWordTokenFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PositionFilterFactory" />
</analyzer>

https://code.google.com/p/ik-analyzer/source/browse/trunk/src/org/wltea/analyzer/lucene/IKAnalyzer.java?r=55
isMaxWordLength 当为true时，分词器进行最大词长切分

当为true时，分词器进行智能切分
https://github.com/yqingp/elasticsearch-analysis-ik/blob/master/src/main/java/org/wltea/analyzer/lucene/IKAnalyzer.java


http://java.dzone.com/articles/indexing-chinese-solr
#1:  Separate your Chinese text into its own field(s). 
#2: Use the CJK or Paoding analyzers for your Chinese Text. 


SmartChineseAnalyzer 源码分析
http://liuhaifeng.com/2010/12/smartcn-source-code-analysis.html

SmartChineseAnalyzer
https://drupal.org/node/1081850
<fieldType name="text_chinese" class="solr.TextField">
      <analyzer class="org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer"/>
</fieldType>

paoding
http://git.oschina.net/zhzhenqin/paoding-analysis
<fieldType name="text_general" class="solr.TextField">
  <analyzer class="net.paoding.analysis.analyzer.PaodingAnalyzer" />
</fieldType>

http://stanbol.apache.org/docs/trunk/components/enhancer/nlp/paoding
<fieldType name="text_zh" class="solr.TextField">
    <analyzer class="net.paoding.analysis.analyzer.PaodingAnalyzer"/>
</fieldType>

http://blog.csdn.net/ivanhxy/article/details/5409541
paoding-dic-home.properties
paoding.dic.home=classpath:dic // dic 所在目录  
paoding.dic.home.config-fisrt=this //应用当前的配置

PaodingAnalysisException: not found the dic home dirctory
http://blog.csdn.net/mengxianhua/article/details/7856340
方法：找到\solr\WEB-INF\classes路径下的paoding-dic-home.properties文件，下的paoding.dic.home，去掉注释，修改为你的dic路径即可。  

http://wiki.apache.org/solr/LanguageAnalysis#Chinese.2C_Japanese.2C_Korean
To use the default setup with fallback to English Porter stemmer for english words, use:
<analyzer class="org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer"/>
Or to configure your own analysis setup, use the SmartChineseSentenceTokenizerFactory along with your custom filter setup. The sentence tokenizer tokenizes on sentence boundaries and the SmartChineseWordTokenFilter breaks this further up into words.
<analyzer>
<tokenizer class="solr.SmartChineseSentenceTokenizerFactory"/>
<filter class="solr.SmartChineseWordTokenFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PositionFilterFactory" />
</analyzer>
<!> Note: Be sure to use PositionFilter at query-time (only) as these languages do not use spaces between words.
Script Translations
Han Traditional <--> Simplified
<filter class="solr.ICUTransformFilterFactory" id="Traditional-Simplified"/>
Katakana <--> Hiragana
<filter class="solr.ICUTransformFilterFactory" id="Katakana-Hiragana"/> 

<filter class="solr.CJKBigramFilterFactory"/>

SmartChineseSentenceTokenizerFactory
http://blog.csdn.net/shirdrn/article/details/7054633
http://qindongliang1922.iteye.com/blog/1894002
org.apache.lucene.analysis.cn.smart.TestSmartChineseAnalyzer


http://discovery-grindstone.blogspot.com/2013/11/cjk-with-solr-for-libraries-part-2.html
Script Translations
<filter class="solr.ICUTransformFilterFactory" id="Traditional-Simplified"/>
<fieldType name="transformed" class="solr.TextField">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.ICUTransformFilterFactory" id="Traditional-Simplified"/>
  </analyzer>
</fieldType>
Katakana <--> Hiragana
<filter class="solr.ICUTransformFilterFactory" id="Katakana-Hiragana"/> 

PositionFilter
http://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/position/PositionFilter.html
Deprecated
PositionFilter makes TokenStream graphs inconsistent which can cause highlighting bugs. Its main use-case being to make QueryParser generate boolean queries instead of phrase queries, it is now advised to use QueryParser.setAutoGeneratePhraseQueries(boolean) (for simple cases) or to override QueryParser.newFieldQuery.

http://git.oschina.net/zhzhenqin/paoding-analysis.git
Paoding分词器for Lucene4.x-Solr4.x

Hidden Markov model隐马尔可夫模型
CS188Spring2013: - Introduction to Artificial Intelligence

http://www.youtube.com/user/CS188Spring2013