配置要求:
系统:centos6.5
目标:基于CUDA8.0+Opencv3.1+Cudnnv5.1+python3.6接口的caffe框架
综合来说,caffe的配置并没有想象中的那么难。还是那句话已官方文档为准,网上的教程很难找到完全对应的。
Centos系统下配置caffe 的官方文档,
http://caffe.berkeleyvision.o...
1.安装前准备
一般依赖项:
sudo yum install protobuf-devel leveldb-devel snappy-devel opencv-devel boost-devel hdf5-devel
剩下的依赖项:
sudo yum install gflags-devel glog-devel lmdb-devel
以上就是caffe配置所需要的依赖包了,这里我采用的方法是全部手动安装,这样成功率要比直接用yum高上非常多.
① Protobuf
由于我配置的是python3.6
的接口,因此protobuf
的版本必须大于3.0以上
https://github.com/google/pro...
分别去下载cpp
和python
的包。
我选的是3.2版本,因此下载了protobuf-cpp-3.2.0.tar.gz
和protobuf-python-3.2.0.tar.gz
包,选好目录
tar -zxvf protobuf-cpp-3.2.0.tar.gz cd protobuf-3.2.0 ./configure make make check make install ldconfig
tar -zxvf protobuf-python-3.2.0.tar.gz
进入目录之后
cd python python setup.py build python setup.py test python setup.py install
编译完成后可以用一下命令确认是否安装成功
conda list | grep protobuf
② boost
http://www.boost.org/users/hi...
下载boost_1_65_0.tar.gz
tar -zxvf boost_1_65_0.tar.gz cd boost_1_65_0 ./bootstrap.sh ./b2 ./b2 install
完成后 若发现没有libboost_python
生成
重新
cd boost_1_65_0 ./bootstrap.sh ./b2 –with-python include=”你pyconfig.h的路径”←可用locate去寻找pyconfig.h的路径
在终端输入
locate libboost_python3
查看/usr/local/lib/
下有没有
/usr/local/lib/libboost_python3.a /usr/local/lib/libboost_python3.so /usr/local/lib/libboost_python3.so.1.65.0
直接创建软链接
ln -s /usr/local/lib/libboost_python3.so.1.65.0 /usr/local/lib/libboost_python3.so
这三个如果没有,就从boost
的stage
文件夹下的lib
文件夹当中把这三个文件拷贝到/usr/local/lib/
目录下,然后创建软链接
③ glog gflags lmdb
这三个依赖项直接根据caffe的官方文档的命令进行安装编译即可
1.glog
wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/google-glog/glog-0.3.3.tar.gz tar zxvf glog-0.3.3.tar.gz cd glog-0.3.3 ./configure make && make install
2. gflags
wget https://github.com/schuhschuh/gflags/archive/master.zip unzip master.zip cd gflags-master mkdir build && cd build export CXXFLAGS="-fPIC" && cmake .. && make VERBOSE=1 make && make install
3.lmdb
git clone https://github.com/LMDB/lmdb cd lmdb/libraries/liblmdb make && make install
④ hdf5
建议安装1.8.17版本,因为anaconda自带的hdf5也是这个版本
http://download.csdn.net/down...
tar -zxvf hdf5-1.8.17.tar.gz cd hdf5-1.8.17 ./configure --prefix=/usr/local/hdf5-1.8.17/ make make check make install make check-install
⑤ snappy
yum install snappy
⑥ leveldb
http://download.csdn.net/down...
tar –xvf leveldb-1.7.0.tar.gz cd leveldb-1.7.0 make cp libleveldb* /usr/lib/ cp –r include/leveldb /usr/local/include
⑦atlas-devel
直接使用yum install atlas-devel
安装
caffe编译
1.caffe下载
git clone https://github.com/bvlc/caffe.git
2.caffe编译
cd caffe vi Makefile
找到Configure build
其下的COMMON_FLAGS +=
后面加上-I/usr/local/hdf5/include
LDFLAGS +=
后面加上-L/usr/local/hdf5/lib
当然如果你之前路径配的都没问题的话,没可以不加
修改Makefile.config
如果没有Makefile.config
,
cp Makefile.config.example Makefile.config vi Makefile.config
以下是我修改后的完整的Makefile.config
,左箭头(←)所指部分是需要修改的地方
## Refer to http://caffe.berkeleyvision.org/installation.html # Contributions simplifying and improving our build system are welcome! # cuDNN acceleration switch (uncomment to build with cuDNN). USE_CUDNN := 1← # cpu-only switch (uncomment to build without GPU support). # cpu_ONLY := 1 # uncomment to disable IO dependencies and corresponding data layers # USE_OPENCV := 0 # USE_LEVELDB := 0 # USE_LMDB := 0 # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary) # You should not set this flag if you will be reading LMDBs with any # possibility of simultaneous read and write # ALLOW_LMDB_NOLOCK := 1 # Uncomment if you're using OpenCV 3 OPENCV_VERSION := 3← # To customize your choice of compiler,uncomment and set the following. # N.B. the default for Linux is g++ and the default for OSX is clang++ # CUSTOM_CXX := g++ # CUDA directory contains bin/ and lib/ directories that we need. CUDA_DIR := /usr/local/cuda-8.0← # On Ubuntu 14.04,if cuda tools are installed via # "sudo apt-get install nvidia-cuda-toolkit" then use this instead: # CUDA_DIR := /usr # CUDA architecture setting: going with all of them. # For CUDA < 6.0,comment the *_50 through *_61 lines for compatibility. # For CUDA < 8.0,comment the *_60 and *_61 lines for compatibility. CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \ -gencode arch=compute_35,code=sm_35 \ -gencode arch=compute_50,code=sm_50 \ -gencode arch=compute_52,code=sm_52 \ -gencode arch=compute_60,code=sm_60 \ -gencode arch=compute_61,code=sm_61 \ -gencode arch=compute_61,code=compute_61 # BLAS choice: # atlas for ATLAS (default) # mkl for MKL # open for OpenBlas BLAS := atlas← # Custom (MKL/ATLAS/OpenBLAS) include and lib directories. # Leave commented to accept the defaults for your choice of BLAS # (which should work)! # BLAS_INCLUDE := /path/to/your/blas # BLAS_LIB := /path/to/your/blas # Homebrew puts openblas in a directory that is not on the standard search path # BLAS_INCLUDE := $(shell brew --prefix openblas)/include # BLAS_LIB := $(shell brew --prefix openblas)/lib # This is required only if you will compile the matlab interface. # MATLAB directory should contain the mex binary in /bin. # MATLAB_DIR := /usr/local # MATLAB_DIR := /Applications/MATLAB_R2012b.app # NOTE: this is required only if you will compile the python interface. # We need to be able to find Python.h and numpy/arrayobject.h. # PYTHON_INCLUDE := /usr/include/python2.7 \← /usr/lib/python2.7/dist-packages/numpy/core/include← # Anaconda Python distribution is quite popular. Include path: # Verify anaconda location,sometimes it's in root. ANACONDA_HOME := /root/anaconda3← PYTHON_INCLUDE := $(ANACONDA_HOME)/include \← $(ANACONDA_HOME)/include/python3.6m \← $(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include← # Uncomment to use Python 3 (default is Python 2) PYTHON_LIBRARIES := boost_python3 python3.6m← # PYTHON_INCLUDE := /usr/include/python3.6m \← /usr/lib/python3.6/dist-packages/numpy/core/include← # We need to be able to find libpythonX.X.so or .dylib. # PYTHON_LIB := /usr/lib← PYTHON_LIB := $(ANACONDA_HOME)/lib \← $(ANACONDA_HOME)/pkgs/python-3.6.1-2/lib← # Homebrew installs numpy in a non standard path (keg only) # PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include # PYTHON_LIB += $(shell brew --prefix numpy)/lib # Uncomment to support layers written in Python (will link against Python libs) WITH_PYTHON_LAYER := 1← # Whatever else you find you need goes here. INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/local/hdf5/include← LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib64/atlas /usr/local/hdf5/lib← # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies # INCLUDE_DIRS += $(shell brew --prefix)/include # LIBRARY_DIRS += $(shell brew --prefix)/lib # NCCL acceleration switch (uncomment to build with NCCL) # https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0) # USE_NCCL := 1 # Uncomment to use `pkg-config` to specify OpenCV library paths. # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.) # USE_PKG_CONFIG := 1 # N.B. both build and distribute dirs are cleared on `make clean` BUILD_DIR := build DISTRIBUTE_DIR := distribute # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171 # DEBUG := 1 # The ID of the GPU that 'make runtest' will use to run unit tests. TEST_GPUID := 0 # enable pretty build (comment to see full commands) Q ?= @
修改完成后保存退出。
主要需要指出的是PYTHON_LIBRARIES
: 同意确保boost_python3
这个动态链接库在ld.so.conf
文件中(记住ldconfig
)或LD_LIBRARY_PATH
中能找到。INCLUDE_DIRS
和LIBRARY_DIRS
的话需要加上的路径是没有放在local
或者usr
文件夹下的include
或者lib
文件夹中的依赖项。
然后
make all -jn make test -jn make runtest -jn make pycaffe -jn
在安装完成之后,如果想要导入caffePython
模块,则添加模块路径到你的环境变量 $PYTHONPATH
中。比如在你的~/.bashrc
中添加如下一行:
export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH
打开终端
python import caffe
如果没有错误,表示caffe的python接口配置完成
碰到的问题
①
[root@localhost caffe]# make runtest .build_release/tools/caffe .build_release/tools/caffe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /home/HY/caffe/caffe/.build_release/tools/../lib/libcaffe.so.1.0.0) .build_release/tools/caffe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/local/lib/libopencv_core.so.3.1) .build_release/tools/caffe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/local/lib/libopencv_imgcodecs.so.3.1) make: *** [runtest] Error 1
问题产生的原因:新安装的高版本的gcc生成的动态库没有替换老版本的gcc的动态库导致的
解决方案参考这个博主写的博客:http://blog.chinaunix.net/uid...
②
Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can,at your own risk,disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.8.3,library is 1.8.17
这个问题产生的原因是hdf5版本不一致所造成的系统本身已经安装的是1.8.3版本,但是anaconda3所带的hdf5的版本的是1.8.17,所以在编译的时候会发生版本冲突。
为了解决这个兼容性问题,只能是重新编译hdf5-1.8.17版本,具体方法参考上文安装依赖项中的hdf5编译安装方法
③
[root@localhost caffe]# make pycaffe CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp python/caffe/_caffe.cpp:1:52: fatal error: Python.h: No such file or directory #include <Python.h> // NOLINT(build/include_alpha) ^ compilation terminated. make: *** [python/caffe/_caffe.so] Error 1
顾名思义找不到Python.h
这个头文件,于是我利用
find / -name "Python.h"
去找文件的路径发现
/root/anaconda2/include/python2.7/Python.h /root/anaconda2/pkgs/python-2.7.13-0/include/python2.7/Python.h /root/anaconda3/include/python3.6m/Python.h /root/anaconda3/pkgs/python-3.6.1-2/include/python3.6m/Python.h
可以看到第三个路径是上面修改Makefile.config
时修改PYTHON_INCLUDE
时应该要改的路径,打开Makefile.config
到指定位置,果然发现自己的路径配置错误,当时指向了python3.6
而不是python3.6m
。改成python3.6m
之后重新make
不再出现这个问题,因此这个问题出现的原因就是python的路径配置错误。
④
The following directory should be added to compiler include paths: /home/HY/boost_1_59_0 The following directory should be added to linker library paths: /home/HY/boost_1_59_0/stage/lib
这个问题是最早几次编译caffe的时候出现的问题,查了好久是因为在编译boost的时候没有把python模块编译出来,如果安装上文正确编译libboost_python3
的话 并不会出现这个问题,出现这个问题的朋友,可以进入boost文件夹编译一下python模块,因为boost是可以重复编译的,命令参考上文
⑤ 编译boost库时
...Failed gcc.compile.c++ bin.v2/libs/python/build/gcc-4.8.2/release/link-static/threading-multi/numpy/scalars.o... gcc.compile.c++ bin.v2/libs/python/build/gcc-4.8.2/release/link-static/threading-multi/numpy/ufunc.o In file included from ./boost/python/detail/prefix.hpp:13:0,from ./boost/python/args.hpp:8,from ./boost/python.hpp:11,from ./boost/python/numpy/internal.hpp:17,from libs/python/src/numpy/ufunc.cpp:8: ./boost/python/detail/wrap_python.hpp:50:23: fatal error: pyconfig.h: No such file or directory # include <pyconfig.h> ^ compilation terminated.
这个问题和第四个问题出现的地方差不多,都是在编译boost的python模块的时候发生的,这个问题是因为编译的时候找不到pyconfig.h
。
解决方法:编译boost_python
的时候use
./b2 --with-python include="path/to/pyconfig.h"
引号里面的路径可以利用
locate pyconfig.h
去确定
⑥ 在make runtest
的时候
error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
顾名思义找不到libpython3.6m.so.1.0
解决方法利用locate
去找到libpython3.6m.so.1.0
所在位置
然后复制libpython3.6m.so.1.0
到/usr/local/lib
目录下
⑦ NVCC的警告
产生原因好像在CUDA8.0以后把compute 20,21
都弃用了,因此解决方法也很简单。
vi Makefile.conf
把CUDA_ARCH
中-gencode arch=compute_20,code=sm_20
和-gencode arch=compute_20,code=sm_21
直接去掉