Nutch的配置（使用MySQL作为数据存储）

这里假定nutch的根目录为：${APACHE_NUTCH_HOME}

配置${APACHE_NUTCH_HOME}/ivy/ivy.xml，确保Nutch使用MySQL作为数据存储

1. <dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>  

<dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>

1. <dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>  

<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>

1. <dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />  

<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />

1. <!-- Uncomment this to use MySQL as database with SQL as Gora store. -->  
2. <dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>  

<!-- Uncomment this to use MySQL as database with SQL as Gora store. --><dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>

编辑${APACHE_NUTCH_HOME}/conf/gora.properties 

添加以下代码激活MySQL的配置

1. ###############################  
2. # MySQL properties #  
3. ###############################  
4. gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver  
5. gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true  
6. gora.sqlstore.jdbc.user=xxxxx  
7. gora.sqlstore.jdbc.password=xxxxx  

################################ MySQL properties ################################gora.sqlstore.jdbc.driver=com.mysql.jdbc.Drivergora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=truegora.sqlstore.jdbc.user=xxxxxgora.sqlstore.jdbc.password=xxxxx

1. <primarykey column="id" length="767"/>  

<primarykey column="id" length="767"/>

配置${APACHE_NUTCH_HOME}/conf/nutch-site.xml 

在 http.agent.name字段下增加一个名字，可以是任意值但不能为空！  如果需要的话可以添加额外的语言（例如en为英语），同时也可以设置默认编码格式为utf-8

1. <property>  
2. <name>http.agent.name</name>  
3. <value>YourNutchSpider</value>  
4. </property>  
5.   
6. <property>  
7. <name>http.accept.language</name>  
8. <value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>  
9. <description>Value of the "Accept-Language" request header field.  
10. This allows selecting non-English language as default one to retrieve.  
11. It is a useful setting for search engines build for certain national group.  
12. </description>  
13. </property>  
14.   
15. <property>  
16. <name>parser.character.encoding.default</name>  
17. <value>utf-8</value>  
18. <description>The character encoding to fall back to when no other information  
19. is available</description>  
20. </property>  
21.   
22. <property>  
23. <name>storage.data.store.class</name>  
24. <value>org.apache.gora.sql.store.SqlStore</value>  
25. <description>The Gora DataStore class for storing and retrieving data.  
26. Currently the following stores are available: ....  
27. </description>  
28. </property>  

<property><name>http.agent.name</name><value>YourNutchSpider</value></property><property><name>http.accept.language</name><value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value><description>Value of the "Accept-Language" request header field.This allows selecting non-English language as default one to retrieve.It is a useful setting for search engines build for certain national group.</description></property><property><name>parser.character.encoding.default</name><value>utf-8</value><description>The character encoding to fall back to when no other informationis available</description></property><property><name>storage.data.store.class</name><value>org.apache.gora.sql.store.SqlStore</value><description>The Gora DataStore class for storing and retrieving data.Currently the following stores are available: ....</description></property>

1. cd ${APACHE_NUTCH_HOME}/runtime/local  
2. mkdir -p urls  
3. echo ‘http://nutch.apache.org/‘ > urls/seed.txt  

cd ${APACHE_NUTCH_HOME}/runtime/localmkdir -p urlsecho ‘http://nutch.apache.org/‘ > urls/seed.txt

1. bin/nutch crawl urls -threads 30  

bin/nutch crawl urls -threads 30

翻译源：http://wiki.apache.org/nutch/#Nutch_2.X_tutorial.28s.29

Nutch的配置（使用MySQL作为数据存储）

标签：and   ted   name   alt   inline   track   编码格式   根目录   mod