pg_bulkload批量载入工具(初探)

前端之家收集整理的这篇文章主要介绍了pg_bulkload批量载入工具(初探)前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。


安装:

  1. #tar-zxvfpg_bulkload-3.1.6.tar.gz
  2. #cdpg_bulkload-3.1.6
  3. #./home/postgres/.bash_profile
  4. #make
  5. #makeinstall


引入扩展:

  1. $psql
  2. psql(9.3.4)
  3. Type"help"forhelp.
  4. postgres=#createdatabasetest;
  5. CREATEDATABASE
  6. postgres=#\ctest
  7. Youarenowconnectedtodatabase"test"asuser"postgres".
  8. test=#createextensionpg_bulkload;
  9. CREATEEXTENSION
  10. test=#\dx
  11. Listofinstalledextensions
  12. Name|Version|Schema|Description
  13. -------------+---------+------------+-----------------------------------------------------------------
  14. pg_bulkload|1.0|public|pg_bulkloadisahighspeeddataloadingutilityforPostgresql
  15. plpgsql|1.0|pg_catalog|PL/pgsqlprocedurallanguage
  16. (2rows)
  17. test=#\df
  18. Listoffunctions
  19. Schema|Name|Resultdatatype|Argum
  20. entdatatypes|Type
  21. --------+-------------+------------------+------------------------------------------------------------------------------------------------------------
  22. ---------------------------------------------------------------------------------------------------------------------+--------
  23. public|pg_bulkload|record|optionstext[],OUTskipbigint,OUTcountbigint,OUTparse_errorsbigint,OUTduplicate_newbigint,OUTd
  24. uplicate_oldbigint,OUTsystem_timedoubleprecision,OUTuser_timedoubleprecision,OUTdurationdoubleprecision|normal
  25. (1row)


创造测试数据:

  1. test=#createtablet1(idint,nametext);
  2. CREATETABLE
  3. test=#insertintot1selectgenerate_series(1,50000000),'HighGo';
  4. INSERT050000000
  5. test=#\d+
  6. Listofrelations
  7. Schema|Name|Type|Owner|Size|Description
  8. --------+------+-------+----------+---------+-------------
  9. public|t1|table|postgres|2112MB|
  10. (1row)
  11. test=#copyt1to'/opt/pg93/data/t1.csv'with(format'csv');
  12. COPY50000000
  1. $du-sh/opt/pg93/data/t1.csv
  2. 754M/opt/pg93/data/t1.csv


copy方式载入数据:

(with logged):

  1. test=#truncatet1;
  2. TRUNCATETABLE
  3. test=#\timing
  4. Timingison.
  5. test=#
  6. test=#copyt1from'/opt/pg93/data/t1.csv'with(format'csv');
  7. COPY50000000
  8. Time:139038.099ms

(without logged):

  1. test=#truncatet1;
  2. TRUNCATETABLE
  3.  
  4. 修改t1表为unlogged
  5. test=#updatepg_classsetrelpersistence='u'whererelname='t1';
  6. UPDATE1
  7.  
  8. test=#copyt1from'/opt/pg93/data/t1.csv'with(format'csv');
  9. COPY50000000
  10. Time:110796.480ms


pg_blukload载入数据:

(without logged):

  1. $pg_bulkload-i/opt/pg93/data/t1.csv-Ot1-lt1_bulkload.log-o"TYPE=CSV"-o"WRITER=PARALLEL"-dtest
  2. NOTICE:BULKLOADSTART
  3. NOTICE:BULKLOADEND
  4. 0Rowsskipped.
  5. 50000000Rowssuccessfullyloaded.
  6. 0Rowsnotloadedduetoparseerrors.
  7. 0Rowsnotloadedduetoduplicateerrors.
  8. 0Rowsreplacedwithnewrows.
  9. Runbeganon2014-06-1605:53:45.025377+08
  10. Runendedon2014-06-1605:55:00.625057+08
  11. cpu2.80s/33.00usecelapsed75.60sec
  1. $pg_bulkload-i/opt/pg93/data/t1.csv-Ot1-lt1_bulkload.log-o"TYPE=CSV"-o"WRITER=DIRECT"-dtest
  2. NOTICE:BULKLOADSTART
  3. NOTICE:BULKLOADEND
  4. 0Rowsskipped.
  5. 50000000Rowssuccessfullyloaded.
  6. 0Rowsnotloadedduetoparseerrors.
  7. 0Rowsnotloadedduetoduplicateerrors.
  8. 0Rowsreplacedwithnewrows.
  9. Runbeganon2014-06-1606:05:40.267198+08
  10. Runendedon2014-06-1606:07:05.08921+08
  11. cpu6.88s/34.25usecelapsed84.82sec


(with logged):

  1. test=#truncatet1;
  2. TRUNCATETABLE
  3. 修改t1表为logged
  4. test=#updatepg_classsetrelpersistence='p'whererelname='t1';
  5. UPDATE1
  1. $pg_bulkload-i/opt/pg93/data/t1.csv-Ot1-lt1_bulkload.log-o"TYPE=CSV"-o"WRITER=PARALLEL"-dtest
  2. NOTICE:BULKLOADSTART
  3. NOTICE:BULKLOADEND
  4. 0Rowsskipped.
  5. 50000000Rowssuccessfullyloaded.
  6. 0Rowsnotloadedduetoparseerrors.
  7. 0Rowsnotloadedduetoduplicateerrors.
  8. 0Rowsreplacedwithnewrows.
  9. Runbeganon2014-06-1605:57:05.620751+08
  10. Runendedon2014-06-1605:58:20.458029+08
  11. cpu2.80s/33.02usecelapsed74.84sec
  12.  
  13.  
  14. $pg_bulkload-i/opt/pg93/data/t1.csv-Ot1-lt1_bulkload.log-o"TYPE=CSV"-o"WRITER=DIRECT"-dtest
  15. NOTICE:BULKLOADSTART
  16. NOTICE:BULKLOADEND
  17. 0Rowsskipped.
  18. 50000000Rowssuccessfullyloaded.
  19. 0Rowsnotloadedduetoparseerrors.
  20. 0Rowsnotloadedduetoduplicateerrors.
  21. 0Rowsreplacedwithnewrows.
  22. Runbeganon2014-06-1606:02:10.33344+08
  23. Runendedon2014-06-1606:03:36.986382+08
  24. cpu7.15s/34.93usecelapsed86.65sec


pg_bulkload执行过程中查看进程和连接:
(WRITER=PARALLEL)

  1. $ps-ef|greppost
  2. postgres2404422690005:53pts/200:00:00pg_bulkload-i/opt/pg93/data/t1.csv-Ot1-lt1_bulkload.log-oTYPE=CSV-oWRITER=PARALLEL-dtest
  3. postgres2404522364805:53?00:00:04postgres:postgrestest[local]SELECT
  4. postgres2404622362005:53?00:00:02postgres:postgrestest[local]SELECT
  1. postgres=#selectdatname,application_name,queryfrompg_stat_activity;
  2. datname|application_name|query
  3.  
  4.  
  5. ----------+------------------+------------------------------------------------------------------------------------------------------------------------
  6. -----------------------------------------------------------------------------------------------------------
  7. postgres|psql|selectdatname,queryfrompg_stat_activity;
  8. test|pg_bulkload|SELECT*FROMpg_bulkload($1)
  9. test||SELECT*FROMpg_bulkload(ARRAY['TYPE=TUPLE','INPUT='||$1,'WRITER=DIRECT','OUTPUT='||$2,'ON_DUPLICATE_KEEP='||$3,'DUPLICATE_ERRORS='||$4,'DUPLICATE_BADFILE='||$5,'LOGFILE='||$6,'VERBOSE='||$7,'TRUNCATE='||$8])
  10. (3rows)


(WRITER=PARALLEL)

  1. $ps-ef|greppost
  2. postgres2412422690006:02pts/200:00:00pg_bulkload-i/opt/pg93/data/t1.csv-Ot1-lt1_bulkload.log-oTYPE=CSV-oWRITER=DIRECT-dtest
  3. postgres2412522364906:02?00:00:13postgres:postgrestest[local]SELECT
  1. test=#selectdatname,queryfrompg_stat_activity;
  2. datname|application_name|query
  3. ---------+------------------+--------------------------------------------------------------
  4. test|pg_bulkload|SELECT*FROMpg_bulkload($1)
  5. test|psql|selectdatname,queryfrompg_stat_activity;
  6. (2rows)

猜你在找的Postgre SQL相关文章