我正在使用pandas和read_csv方法加载一个非常庞大的csv文件,如1000万条记录,我想知道是否有办法显示该加载的进度,如：
100,000 lines read
150,000 lines read
谢谢.
最佳答案
要显示这样的进度：
Completed 1 %
Completed 2 %
...
Completed 99 %
Completed 100 %
你可以试试这个：
import os,pandas
filename = "VeryLong.csv"
lines_number = sum(1 for line in open(filename))
lines_in_chunk = 500 # I don't know what size is better
counter = 0
completed = 0
reader = pandas.read_csv(filename,chunksize=lines_in_chunk)
for chunk in reader:
# < ... reading the chunk somehow... >
# showing progress:
counter += lines_in_chunk
new_completed = int(round(float(counter)/lines_number * 100))
if (new_completed > completed):
completed = new_completed
print "Completed",completed,"%"