Nutch配置:nutch-default.xml详解

前端之家收集整理的这篇文章主要介绍了Nutch配置:nutch-default.xml详解前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

/×××××××××××××××××××××××××××××××××××××××××/

Author:xxx0624

HomePage:http://www.cnblogs.com/xxx0624/

===============File===============

配置1:

<property>
  name>file.content.limit</value>65536description>The length limit for downloaded content using the file
   protocol,in bytes. If this value is nonnegative (>=0),content longer
   than it will be truncated; otherwise,no truncation at all. Do not
   confuse this setting with the http.content.limit setting.
  >
>

当使用file协议下载的时候,用来限制下载文件的大小,默认为65535个字节。如果超过大小限制,内容会被截断。

配置2:

>file.content.ignored>true>If true,no file content will be saved during fetch. And it is probably what we want to set most of time,since file:// URLs are meant to be local and we can always use them directly at parsing and indexing stages. Otherwise file contents will be saved. !! NO IMPLEMENTED YET !! 如果这个设置为true,当nutch在爬取文件的时候不会下载文件内容

===============HTTP===============

配置1:(重要!)

>http.agent.name></>HTTP 'User-Agent' request header. MUST NOT be empty - please set this to a single word uniquely related to your organization. NOTE: You should also check other related properties: http.robots.agents http.agent.description http.agent.url http.agent.email http.agent.version and set their values appropriately. 这个用于配置HTTP代理。

定义HTTP header中的User-Agent相关属性一定需要配置

配置2:(重要!)

>http.robots.agents>*>The agent strings we'll look for in robots.txt files,comma-separated,in decreasing order of precedence. You should put the value of http.agent.name as the first agent name,and keep the default * at the end of the list. E.g.: BlurflDev,Blurfl,* 有些网页会有robots设置,robots.txt设置为了规范爬虫。

设置这个后,nutch会在相应的爬取网站中的robots.txt内寻找是否存在这个agent,否则无法爬取到该网页。

配置3:

>http.robots.403.allow>Some servers return HTTP status 403 (Forbidden) if /robots.txt doesn't exist. This should probably mean that we are allowed to crawl the site nonetheless. If this is set to false,then such sites will be treated as forbidden. 有些服务器在没有robots文件的时候会返回403错误,这时我们就能随意爬取内容

但是如果这个值被设置成false,我们就无法爬取这个网站。

配置4:

>http.timeout>10000>The default network timeout,in milliseconds.>

默认的网络超时时间是10000ms。

配置5(这一点可以考虑用于优化Nutch的爬取速度):

>http.max.delays>100>The number of times a thread will delay when trying to fetch a page. Each time it finds that a host is busy,it will wait fetcher.server.delay. After http.max.delays attepts,it will give up on the page for now. 在爬取网页的时候,线程的最多等待次数。每次线程发现主机繁忙的时候,线程就会等待fetch.server.delay这么长的时间,如果总的等待次数超过了http.max.delays,nutch则不再爬取该网页。

配置6:

>http.content.limit>The length limit for downloaded content using the http protocol,content longer than it will be truncated; otherwise,no truncation at all. Do not confuse this setting with the file.content.limit setting. 在使用HTTP协议下载网页的时候,用来限制下载网页的内容大小,最多是65536个字节。

超过则会被截断。

配置7:(代理服务部分)

>http.proxy.host>The proxy hostname. If empty,no proxy is used.> >http.proxy.port>The proxy port.>http.proxy.username>Username for proxy. This will be used by 'protocol-httpclient',if the proxy server requests basic,digest and/or NTLM authentication. To use this,'protocol-httpclient' must be present in the value of 'plugin.includes' property. NOTE: For NTLM authentication,do not prefix the username with the domain,i.e. 'susam' is correct whereas 'DOMAIN\susam' is incorrect. >http.proxy.password>Password for proxy. This will be used by 'protocol-httpclient','protocol-httpclient' must be present in the value of 'plugin.includes' property. 分别是代理的主机名,端口号,代理用户名,代理密码。

这几个配置都和protocol-httpclient插件有关。

===============FTP=================

暂无

===============web db===============

(1)Fetch过程中配置(只列出部分配置)

>db.fetch.interval.default>2592000>The default number of seconds between re-fetches of a page (30 days). 这个设置为了定期重新爬取网页的时间间隔,默认是30天。

单位是秒。

>db.fetch.interval.max>7776000>The maximum number of seconds between re-fetches of a page (90 days). After this period every page in the db will be re-tried,no matter what is its status. 这个设置表示在db.fetch.interval.max这段时间过后,数据库中的每个网页都肯定会被重新抓取,不管它目前是什么状态。

>db.fetch.schedule.class>org.apache.nutch.crawl.DefaultFetchSchedule>The implementation of fetch schedule. DefaultFetchSchedule simply adds the original fetchInterval to the last fetch time,regardless of page changes. 这个指定的类是实现了网页下载时间安排。

DefaultFetchSchedule 只是简单的将原来的下载时间间隔加到上次下载时间上,不管当前每个网页的改变。

配置4:

>db.fetch.schedule.adaptive.inc_rate>0.4>If a page is unmodified,its fetchInterval will be increased by this rate. This value should not exceed 0.5,otherwise the algorithm becomes unstable. 如果重新下载网页并更新数据库的时候,发现这个网页没有发生变化,那么这个网页的更新时间间隔会变成:原来的时间间隔+设置的这个值(这个值不能超过0.5)

配置5:

>db.fetch.schedule.adaptive.dec_rate>0.2>If a page is modified,its fetchInterval will be decreased by this rate. This value should not exceed 0.5,sans-serif; line-height:21px"> 如果重新下载网页并更新数据库的时候,发现这个网页发生了变化,那么这个网页的更新时间间隔会变成:原来的时间间隔-设置的这个值(这个值不能超过0.5)

>db.fetch.schedule.adaptive.min_interval>60.0>Minimum fetchInterval,in seconds. 最小的网页更新时间间隔。

>db.fetch.schedule.adaptive.max_interval>31536000.0>Maximum fetchInterval,in seconds (365 days). NOTE: this is limited by db.fetch.interval.max. Pages with fetchInterval larger than db.fetch.interval.max will be fetched anyway. 最大的网页更新时间间隔。

(2)Generate配置

配置7:

>generate.max.count>-1>The maximum number of urls in a single fetchlist. -1 if unlimited. The urls are counted according to the value of the parameter generator.count.mode. 设置下载队列的url数量,-1表示无限。

配置8:

>generate.count.mode>host>Determines how the URLs are counted for generator.max.count. Default value is 'host' but can be 'domain'. Note that we do not count per IP in the new version of the Generator. 设置用来根据host不同来判断该URL是否抓取其内容

配置9:

>generate.update.crawldb>false>For highly-concurrent environments,where several generate/fetch/update cycles may overlap,setting this to true ensures that generate will create different fetchlists even without intervening updatedb-s,at the cost of running an additional job to update CrawlDB. If false,running generate twice without intervening updatedb will generate identical fetchlists. 对于高并发的环境来说,可能发生generate/fetch/update循环重叠的情况。

如果设置为true,即使没有中间updatedb,可以以运行一个额外的job来更新crawldb达到目的。

如果设置为false,在没有中间updatedb的情况下,则有可能产生两个相同的下载队列。

(3)partitioner(分发策略的配置)

配置10:

>partition.url.mode>byHost>Determines how to partition URLs. Default value is 'byHost',also takes 'byDomain' or 'byIP'. 设置根据Host不同来分发url。

(4)fetcher(下载的配置)

配置11:

>fetcher.server.delay>5.0>The number of seconds the fetcher will delay between successive requests to the same server. 设置对同一server成功请求的时间间隔。

配置12:(重要!)

>fetcher.threads.fetch>10>The number of FetcherThreads the fetcher should use. This is also determines the maximum number of requests that are made at once (each FetcherThread handles one connection). The total number of threads running in distributed mode will be the number of fetcher threads * number of nodes as fetcher has one map task per node. 默认10个下载线程

配置13:(重要!)这个可以考虑用于加速nutch爬虫

>fetcher.threads.per.queue>1>This number is the maximum number of threads that should be allowed to access a queue at one time. 设置同一时间内,同一队列能有几个线程访问

配置14:

>fetcher.store.content>If true,fetcher will store content. 设置true表示下载线程会下载内容

配置15:

>fetcher.throughput.threshold.pages>The threshold of minimum pages per second. If the fetcher downloads less pages per second than the configured threshold,the fetcher stops,preventing slow queue's from stalling the throughput. This threshold must be an integer. This can be useful when fetcher.timelimit.mins is hard to determine. The default value of -1 disables this check. 这个是设置fetcher的下载能力。如果每秒下载少于这个设置值,则下载线程会停止。

-1表示这个设置无效

===============index===============

索引部分不是很懂,只列举出部分配置...

<property>
  <name>indexer.max.title.length</name>
  <value>100</value>
  <description>The maximum number of characters of a title that are indexed. A value of -1 disables this check.
  Used by index-basic.
  </description>
</property>

设置能索引的标题的最大长度

================plugin===============

<property>
  <name>plugin.folders</name>
  <value>plugins</value>
  <description>Directories where nutch plugins are located.  Each
  element may be a relative or absolute path.  If absolute,it is used
  as is.  If relative,it is searched for on the classpath.</description>
</property>

设置nutch插件的地址

<property>
  <name>plugin.auto-activation</name>
  <value>true</value>
  <description>Defines if some plugins that are not activated regarding
  the plugin.includes and plugin.excludes properties must be automaticaly
  activated if they are needed by some actived plugins.
  </description>
</property>

插件不加载时,但是又被其他加载的插件依赖,是否自动启动。

默认true为自动启动。

<property>
  <name>plugin.includes</name>
 <value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</value>
 <description>Regular expression naming plugin directory names to
  include.  Any plugin not matching this expression is excluded.
  In any case you need at least include the nutch-extensionpoints plugin. By
  default Nutch includes crawling just HTML and plain text via HTTP,and basic indexing and search plugins. In order to use HTTPS please enable 
  protocol-httpclient,but be aware of possible intermittent problems with the 
  underlying commons-httpclient library.
  </description>
</property>

要包含的插件名称列表(支持正则表达式)

插件4:

<property>
  <name>plugin.excludes</name>
  <value></value>
  <description>Regular expression naming plugin directory names to exclude.  
  </description>
</property>

要排除的插件名称列表(支持正则表达式)

===============parse===============

<property>
  <name>parse.plugin.file</name>
  <value>parse-plugins.xml</value>
  <description>The name of the file that defines the associations between
  content-types and parsers.</description>
</property>

这个用于设置文件类型和相应的解析器

<property>
  <name>htmlparsefilter.order</name>
  <value></value>
  <description>The order by which HTMLParse filters are applied.
  If empty,all available HTMLParse filters (as dictated by properties
  plugin-includes and plugin-excludes above) are loaded and applied in system
  defined order. If not empty,only named filters are loaded and applied
  in given order.
  HTMLParse filter ordering MAY have an impact
  on end result,as some filters could rely on the Metadata generated by a prevIoUs filter.
  </description>
</property>

设置HTML解析器的顺序。默认是按照plugin-includes and plugin-excludes来进行加载的

配置3:(重要!)

<property>
  <name>urlfilter.regex.file</name>
  <value>regex-urlfilter.txt</value>
  <description>Name of file on CLASSPATH containing regular expressions
  used by urlfilter-regex (RegexURLFilter) plugin.</description>
</property>

设置url过滤,支持正则表达式

===============solr & elasticSearch================

配置1:(重要!)

<property>
  <name>solr.mapping.file</name>
  <value>solrindex-mapping.xml</value>
  <description>
  Defines the name of the file that will be used in the mapping of internal
  nutch field names to solr index fields as specified in the target Solr schema.
  </description>
</property>

设置solr索引的映射关系

>solr.commit.index> When closing the indexer,trigger a commit to the Solr server. 关闭索引器时,提交结果到solr服务器

>elastic.index>index> The name of the elasticsearch index. Will normally be autocreated if it doesn't exist. 设置es索引的默认名字

>elastic.max.bulk.docs>500> The number of docs in the batch that will trigger a flush to elasticsearch. 设置bulk方式提交索引文件的数目

==================store==================

>storage.data.store.class>org.apache.gora.memory.store.MemStore>The Gora DataStore class for storing and retrieving data. Currently the following stores are available: org.apache.gora.sql.store.sqlStore Default store. A DataStore implementation for RDBMS with a sql interface. sqlStore uses JDBC drivers to communicate with the DB. As explained in ivy.xml,currently >= gora-core 0.3 is not backwards compatable with sqlStore. org.apache.gora.cassandra.store.CassandraStore Gora class for storing data in Apache Cassandra. org.apache.gora.hbase.store.HBaseStore Gora class for storing data in Apache HBase. org.apache.gora.accumulo.store.AccumuloStore Gora class for storing data in Apache Accumulo. org.apache.gora.avro.store.AvroStore Gora class for storing data in Apache Avro. org.apache.gora.avro.store.DataFileAvroStore Gora class for storing data in Apache Avro. DataFileAvroStore is a file based store which uses Avro's DataFile{Writer,Reader}'s as a backend. This datastore supports mapreduce. org.apache.gora.memory.store.MemStore Gora class for storing data in a Memory based implementation for tests. 指定存储的方式,如hbase,avro等方式

方式2:还可以更改gora文件

原文链接:https://www.f2er.com/xml/296781.html

猜你在找的XML相关文章