/×××××××××××××××××××××××××××××××××××××××××/
Author：xxx0624
HomePage：http://www.cnblogs.com/xxx0624/
===============File===============
配置1：
<property> name>file.content.limit</value>65536description>The length limit for downloaded content using the file protocol,in bytes. If this value is nonnegative (>=0),content longer than it will be truncated; otherwise,no truncation at all. Do not confuse this setting with the http.content.limit setting. > >
当使用file协议下载的时候，用来限制下载文件的大小，默认为65535个字节。如果超过大小限制，内容会被截断。
配置2：
>file.content.ignored>true>If true,no file content will be saved during fetch.
And it is probably what we want to set most of time,since file:// URLs
are meant to be local and we can always use them directly at parsing
and indexing stages. Otherwise file contents will be saved.
!! NO IMPLEMENTED YET !!
如果这个设置为true，当nutch在爬取文件的时候不会下载文件内容。
===============HTTP===============
配置1：(重要！)
>http.agent.name></>HTTP 'User-Agent' request header. MUST NOT be empty -
please set this to a single word uniquely related to your organization.
NOTE: You should also check other related properties:
http.robots.agents
http.agent.description
http.agent.url
http.agent.email
http.agent.version
and set their values appropriately.
这个用于配置HTTP代理。
定义HTTP header中的User-Agent相关属性，一定需要配置！
配置2：(重要！)
>http.robots.agents>*>The agent strings we'll look for in robots.txt files,comma-separated,in decreasing order of precedence. You should
put the value of http.agent.name as the first agent name,and keep the
default * at the end of the list. E.g.: BlurflDev,Blurfl,*
有些网页会有robots设置，robots.txt设置为了规范爬虫。
设置这个后，nutch会在相应的爬取网站中的robots.txt内寻找是否存在这个agent，否则无法爬取到该网页。
配置3：
>http.robots.403.allow>Some servers return HTTP status 403 (Forbidden) if
/robots.txt doesn't exist. This should probably mean that we are
allowed to crawl the site nonetheless. If this is set to false,then such sites will be treated as forbidden. 有些服务器在没有robots文件的时候会返回403错误，这时我们就能随意爬取内容。
但是如果这个值被设置成false，我们就无法爬取这个网站。
配置4：
>http.timeout>10000>The default network timeout,in milliseconds.>
默认的网络超时时间是10000ms。
配置5（这一点可以考虑用于优化Nutch的爬取速度）：
>http.max.delays>100>The number of times a thread will delay when trying to
fetch a page. Each time it finds that a host is busy,it will wait
fetcher.server.delay. After http.max.delays attepts,it will give
up on the page for now. 在爬取网页的时候，线程的最多等待次数。每次线程发现主机繁忙的时候，线程就会等待fetch.server.delay这么长的时间，如果总的等待次数超过了http.max.delays，nutch则不再爬取该网页。
配置6：
>http.content.limit>The length limit for downloaded content using the http
protocol,content longer
than it will be truncated; otherwise,no truncation at all. Do not
confuse this setting with the file.content.limit setting.
在使用HTTP协议下载网页的时候，用来限制下载网页的内容大小，最多是65536个字节。
超过则会被截断。
配置7：（代理服务部分）
>http.proxy.host>The proxy hostname. If empty,no proxy is used.>
>http.proxy.port>The proxy port.>http.proxy.username>Username for proxy. This will be used by
'protocol-httpclient',if the proxy server requests basic,digest
and/or NTLM authentication. To use this,'protocol-httpclient' must
be present in the value of 'plugin.includes' property.
NOTE: For NTLM authentication,do not prefix the username with the
domain,i.e. 'susam' is correct whereas 'DOMAIN\susam' is incorrect.
>http.proxy.password>Password for proxy. This will be used by
'protocol-httpclient','protocol-httpclient' must
be present in the value of 'plugin.includes' property.
分别是代理的主机名，端口号，代理用户名，代理密码。
这几个配置都和protocol-httpclient插件有关。
===============FTP=================
暂无
===============web db===============
（1）Fetch过程中配置（只列出部分配置）
>
db.fetch.interval.default>2592000>The default number of seconds between re-fetches of a page (30 days). 这个设置为了定期重新爬取网页的时间间隔，默认是30天。单位是秒。
>
db.fetch.interval.max>7776000>The maximum number of seconds between re-fetches of a page (90 days). After this period every page in the db will be re-tried,no matter what is its status. 这个设置表示在db.fetch.interval.max这段时间过后，数据库中的每个网页都肯定会被重新抓取，不管它目前是什么状态。>
db.fetch.schedule.class>org.apache.nutch.crawl.DefaultFetchSchedule>The implementation of fetch schedule. DefaultFetchSchedule simply adds the original fetchInterval to the last fetch time,regardless of page changes. 这个指定的类是实现了网页下载时间安排。DefaultFetchSchedule 只是简单的将原来的下载时间间隔加到上次下载时间上，不管当前每个网页的改变。
配置4：
>db.fetch.schedule.adaptive.inc_rate>0.4>If a page is unmodified,its fetchInterval will be
increased by this rate. This value should not
exceed 0.5,otherwise the algorithm becomes unstable. 如果重新下载网页并更新数据库的时候，发现这个网页没有发生变化，那么这个网页的更新时间间隔会变成：原来的时间间隔+设置的这个值（这个值不能超过0.5）
配置5：
>db.fetch.schedule.adaptive.dec_rate>0.2>If a page is modified,its fetchInterval will be
decreased by this rate. This value should not
exceed 0.5,sans-serif; line-height:21px"> 如果重新下载网页并更新数据库的时候，发现这个网页发生了变化，那么这个网页的更新时间间隔会变成：原来的时间间隔-设置的这个值（这个值不能超过0.5）
>
db.fetch.schedule.adaptive.min_interval>60.0>Minimum fetchInterval,in seconds. 最小的网页更新时间间隔。>db.fetch.schedule.adaptive.max_interval>31536000.0>Maximum fetchInterval,in seconds (365 days).
NOTE: this is limited by db.fetch.interval.max. Pages with
fetchInterval larger than db.fetch.interval.max
will be fetched anyway. 最大的网页更新时间间隔。
（2）Generate配置
配置7：
>generate.max.count>-1>The maximum number of urls in a single
fetchlist. -1 if unlimited. The urls are counted according
to the value of the parameter generator.count.mode.
设置下载队列的url数量，-1表示无限。
配置8：
>generate.count.mode>host>Determines how the URLs are counted for generator.max.count.
Default value is 'host' but can be 'domain'. Note that we do not count
per IP in the new version of the Generator.
设置用来根据host不同来判断该URL是否抓取其内容
配置9：
>generate.update.crawldb>false>For highly-concurrent environments,where several
generate/fetch/update cycles may overlap,setting this to true ensures
that generate will create different fetchlists even without intervening
updatedb-s,at the cost of running an additional job to update CrawlDB.
If false,running generate twice without intervening
updatedb will generate identical fetchlists. 对于高并发的环境来说，可能发生generate/fetch/update循环重叠的情况。
如果设置为true，即使没有中间updatedb，可以以运行一个额外的job来更新crawldb达到目的。
如果设置为false，在没有中间updatedb的情况下，则有可能产生两个相同的下载队列。
（3）partitioner（分发策略的配置）
配置10：
>partition.url.mode>byHost>Determines how to partition URLs. Default value is 'byHost',also takes 'byDomain' or 'byIP'.
设置根据Host不同来分发url。
（4）fetcher（下载的配置）
配置11：
>fetcher.server.delay>5.0>The number of seconds the fetcher will delay between
successive requests to the same server. 设置对同一server成功请求的时间间隔。
配置12：（重要！）
>fetcher.threads.fetch>10>The number of FetcherThreads the fetcher should use.
This is also determines the maximum number of requests that are
made at once (each FetcherThread handles one connection). The total
number of threads running in distributed mode will be the number of
fetcher threads * number of nodes as fetcher has one map task per node.
默认10个下载线程
配置13：（重要！）这个可以考虑用于加速nutch爬虫
>fetcher.threads.per.queue>1>This number is the maximum number of threads that
should be allowed to access a queue at one time. 设置同一时间内，同一队列能有几个线程访问
配置14：
>fetcher.store.content>If true,fetcher will store content. 设置true表示下载线程会下载内容
配置15：
>fetcher.throughput.threshold.pages>The threshold of minimum pages per second. If the fetcher downloads less
pages per second than the configured threshold,the fetcher stops,preventing slow queue's
from stalling the throughput. This threshold must be an integer. This can be useful when
fetcher.timelimit.mins is hard to determine. The default value of -1 disables this check.
这个是设置fetcher的下载能力。如果每秒下载少于这个设置值，则下载线程会停止。
-1表示这个设置无效
===============index===============
索引部分不是很懂，只列举出部分配置...
<property> <name>indexer.max.title.length</name> <value>100</value> <description>The maximum number of characters of a title that are indexed. A value of -1 disables this check. Used by index-basic. </description> </property>
设置能索引的标题的最大长度
================plugin===============
<property> <name>plugin.folders</name> <value>plugins</value> <description>Directories where nutch plugins are located. Each element may be a relative or absolute path. If absolute,it is used as is. If relative,it is searched for on the classpath.</description> </property>
设置nutch插件的地址
<property> <name>plugin.auto-activation</name> <value>true</value> <description>Defines if some plugins that are not activated regarding the plugin.includes and plugin.excludes properties must be automaticaly activated if they are needed by some actived plugins. </description> </property>
默认true为自动启动。
<property> <name>plugin.includes</name> <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</value> <description>Regular expression naming plugin directory names to include. Any plugin not matching this expression is excluded. In any case you need at least include the nutch-extensionpoints plugin. By default Nutch includes crawling just HTML and plain text via HTTP,and basic indexing and search plugins. In order to use HTTPS please enable protocol-httpclient,but be aware of possible intermittent problems with the underlying commons-httpclient library. </description> </property>
插件4：
<property>
<name>plugin.excludes</name>
<value></value>
<description>Regular expression naming plugin directory names to exclude.
</description>
</property>
===============parse===============
<property>
<name>parse.plugin.file</name>
<value>parse-plugins.xml</value>
<description>The name of the file that defines the associations between
content-types and parsers.</description>
</property>
这个用于设置文件类型和相应的解析器
<property> <name>htmlparsefilter.order</name> <value></value> <description>The order by which HTMLParse filters are applied. If empty,all available HTMLParse filters (as dictated by properties plugin-includes and plugin-excludes above) are loaded and applied in system defined order. If not empty,only named filters are loaded and applied in given order. HTMLParse filter ordering MAY have an impact on end result,as some filters could rely on the Metadata generated by a prevIoUs filter. </description> </property>
设置HTML解析器的顺序。默认是按照plugin-includes and plugin-excludes来进行加载的
配置3：（重要！）
<property>
<name>urlfilter.regex.file</name>
<value>regex-urlfilter.txt</value>
<description>Name of file on CLASSPATH containing regular expressions
used by urlfilter-regex (RegexURLFilter) plugin.</description>
</property>
设置url过滤，支持正则表达式
===============solr & elasticSearch================
配置1：（重要！）
<property> <name>solr.mapping.file</name> <value>solrindex-mapping.xml</value> <description> Defines the name of the file that will be used in the mapping of internal nutch field names to solr index fields as specified in the target Solr schema. </description> </property>
设置solr索引的映射关系
>
solr.commit.index> When closing the indexer,trigger a commit to the Solr server. 当关闭索引器时，提交结果到solr服务器>
elastic.index>index> The name of the elasticsearch index. Will normally be autocreated if it doesn't exist. 设置es索引的默认名字>
elastic.max.bulk.docs>500> The number of docs in the batch that will trigger a flush to elasticsearch. 设置bulk方式提交索引文件的数目==================store==================
>
storage.data.store.class>org.apache.gora.memory.store.MemStore>The Gora DataStore class for storing and retrieving data. Currently the following stores are available: org.apache.gora.sql.store.sqlStore Default store. A DataStore implementation for RDBMS with a sql interface. sqlStore uses JDBC drivers to communicate with the DB. As explained in ivy.xml,currently >= gora-core 0.3 is not backwards compatable with sqlStore. org.apache.gora.cassandra.store.CassandraStore Gora class for storing data in Apache Cassandra. org.apache.gora.hbase.store.HBaseStore Gora class for storing data in Apache HBase. org.apache.gora.accumulo.store.AccumuloStore Gora class for storing data in Apache Accumulo. org.apache.gora.avro.store.AvroStore Gora class for storing data in Apache Avro. org.apache.gora.avro.store.DataFileAvroStore Gora class for storing data in Apache Avro. DataFileAvroStore is a file based store which uses Avro's DataFile{Writer,Reader}'s as a backend. This datastore supports mapreduce. org.apache.gora.memory.store.MemStore Gora class for storing data in a Memory based implementation for tests. 指定存储的方式，如hbase，avro等方式方式2：还可以更改gora文件
原文链接：https://www.f2er.com/xml/296781.html