我有一个mysql数据库,在一个表中有超过400,000,000条记录.

当我设置限制时,Solr导入运行正常.我的最后一次测试是10,它在大约5分钟内成功运行.在Solr中拥有完整的数据集是必需的,当我删除限制时,solr报告它已经丢失了与MysqL的连接.

我相信我遇到的是当前配置中的错误配置或限制.

这是来自tomcat的catalina.out日志

Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT id as solr_id,id,firstname,lastname,state from names Processing Document # 1
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:621)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:327)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:225)
        ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT id as solr_id,state from names Processing Document # 1
        at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.sqlEntityProcessor.initQuery(sqlEntityProcessor.java:59)
        at org.apache.solr.handler.dataimport.sqlEntityProcessor.nextRow(sqlEntityProcessor.java:73)
        at org.apache.solr.handler.dataimport.EntityProcessorWrapper.pullRow(EntityProcessorWrapper.java:330)
        at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:296)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:683)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:619)
        ... 5 more
Caused by: com.MysqL.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 353,135 milliseconds ago.  The last packet sent successfully to the server was 353,135 milliseconds ago.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
        at com.MysqL.jdbc.Util.handleNewInstance(Util.java:411)
        at com.MysqL.jdbc.sqlError.createCommunicationsException(sqlError.java:1121)
        at com.MysqL.jdbc.MysqLIO.nextRowFast(MysqLIO.java:2243)
        at com.MysqL.jdbc.MysqLIO.nextRow(MysqLIO.java:1999)
        at com.MysqL.jdbc.MysqLIO.readSingleRowSet(MysqLIO.java:3504)
        at com.MysqL.jdbc.MysqLIO.getResultSet(MysqLIO.java:490)
        at com.MysqL.jdbc.MysqLIO.readResultsForQueryOrUpdate(MysqLIO.java:3198)
        at com.MysqL.jdbc.MysqLIO.readAllResults(MysqLIO.java:2366)
        at com.MysqL.jdbc.MysqLIO.sqlQueryDirect(MysqLIO.java:2789)
        at com.MysqL.jdbc.ConnectionImpl.execsql(ConnectionImpl.java:2809)
        at com.MysqL.jdbc.ConnectionImpl.execsql(ConnectionImpl.java:2758)
        at com.MysqL.jdbc.StatementImpl.execute(StatementImpl.java:894)
        at com.MysqL.jdbc.StatementImpl.execute(StatementImpl.java:732)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.MysqL.jdbc.MysqLIO.readFully(MysqLIO.java:3119)
        at com.MysqL.jdbc.MysqLIO.nextRowFast(MysqLIO.java:2224)
        ... 24 more
Jun 16,2013 4:51:37 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Jun 16,2013 4:51:37 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback

MysqL日志

130616 16:15:29 MysqLd_safe Starting MysqLd daemon with databases from /var/lib/MysqL
130616 16:15:29 [Warning] option 'max_allowed_packet': unsigned value 53687091200 adjusted to 1073741824
130616 16:15:29  InnoDB: Initializing buffer pool,size = 22.0G
130616 16:15:30  InnoDB: Completed initialization of buffer pool
130616 16:15:31  InnoDB: Started; log sequence number 49 1885610091
130616 16:15:31 [Note] Event Scheduler: Loaded 0 events
130616 16:15:31 [Note] /usr/libexec/MysqLd: ready for connections.
Version: '5.1.69'  socket: '/var/lib/MysqL/MysqL.sock'  port: 3306  Source distribution

MysqL表

CREATE TABLE `names` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,`state` varchar(12) DEFAULT NULL,`source` varchar(128) DEFAULT NULL,`lastname` varchar(128) DEFAULT NULL,`firstname` varchar(128) DEFAULT NULL,PRIMARY KEY (`id`),KEY `source-state` (`source`,`state`),KEY `first-last` (`firstname`,`lastname`),KEY `first-last-state` (`firstname`,`lastname`,`state`)
) ENGINE=InnoDB AUTO_INCREMENT=433924211 DEFAULT CHARSET=utf8;

MysqL配置

[MysqLd]
datadir=/var/lib/MysqL
socket=/var/lib/MysqL/MysqL.sock
user=MysqL
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
wait_timeout=66600
net_read_timeout=66600
interactive_timeout=666666
max_allowed_packet=50G
max_connect_errors=666666
innodb_buffer_pool_size=22GB
innodb_flush_log_at_trx_commit = 2
thread_cache_size=4
[MysqLd_safe]
log-error=/var/log/MysqLd.log
pid-file=/var/run/MysqLd/MysqLd.pid

Solr data-config.xml

MysqL.jdbc.Driver"
                url="jdbc:MysqL://127.0.0.1:3306/databasename?zeroDateTimeBehavior=convertToNull"
                user="root"
                password="password"
                stream="true"/>