swift分布式存储多节点部署

前端之家收集整理的这篇文章主要介绍了swift分布式存储多节点部署前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

1、机器

192.168.1.211Proxy Node

192.168.1.212Storage Node

192.168.1.213Storage Node

192.168.1.214Storage Node


系统为SLES11sp1


2、配置软件源

因为公司服务器无法连外网,所以配置局域网源和本地源来搭建环境


上传ISO镜像文件到各台机器

SLES-11-SP4-DVD-x86_64-GM-DVD1.iso


每台机器挂载镜像,配置本地源

# mkdir /mnt/cdrom

# mount -o loop SLES-11-SP4-DVD-x86_64-GM-DVD1.iso /mnt/cdrom

# mount

/home/SLES-11-SP4-DVD-x86_64-GM-DVD1.iso on /mnt/cdrom type iso9660 (ro)


# vi /etc/zypp/repos.d/Local-iso.repo

[local-iso]

name=local iso repo

enabled=1

autorefresh=1

baseurl=file:/mnt/cdrom/

type=yast2

gpgcheck=0


http局域网源

/etc/zypp/repos.d # cat Icehouse.repo

[Icehouse]

name=Icehouse

enabled=1

autorefresh=1

baseurl=http://192.168.1.206:8080/download.opensuse.org/repositories/Cloud/OpenStack/Icehouse/SLE_11_SP3/

type=rpm-md


3、创建swift用户并配置权限

创建组

# groupadd swift

创建用户

# useradd -g swift -d /home/swift -s /bin/bash -m swift

为swift赋予sudo

# vi /etc/sudoers

在root ALL=(ALL) ALL一行下添加

swift ALL=(ALL) NOPASSWD:ALL


# passwd swift


5、基础配置

注意关闭SElinux和防火墙


分别在四台机器上安装依赖

# su - swift

sudo zypper install curlgcc memcached rsync sqlite3 xfsprogs Git-core libffi-dev python-setuptools

sudo zypper install python-coverage python-dev python-nose python-simplejson python-xattr python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip python-dnspython python-mock python-swiftclient openstack-swift


注:

python-swiftclient是客户端工具,可以在服务器上安装使服务器只充当客户端。


每个节点上配置

~> sudo chown -R swift:swift /etc/swift


~> /etc/swift> cat swift.conf

[swift-hash]

#random unique strings that can neverchange(DO NOT LOSE)

swift_hash_path_prefix = 'od -t x8 -N 8 -An < /dev/random'

swift_hash_path_suffix = 'od -t x8 -N 8 -An < /dev/random'


6、安装配置proxy

sudo zypper install openstack-swift-proxy memcached


启动memcached

/usr/sbin # # ./memcached -d -m 10 -u swift -l 192.168.1.211 11211 -c 256 -P /tmp/memcached/pid


注:

memcached为什么这样启动,而不是在配置文件里写入是因为我发现写入配置文件后启动服务并没有生效。


/usr/sbin # service memcached status

Checking for service memcached

running


/usr/sbin # netstat -an | grep 11211

tcp 0 0 192.168.1.211:11211 0.0.0.0:* LISTEN

udp 0 0 192.168.1.211:11211 0.0.0.0:*

查看配置后的文件

/etc/swift> grep -v "^#" /etc/swift/proxy-server.conf | grep -v "^$"

[DEFAULT]

user = swift

bind_port = 8090

workers = 8

[pipeline:main]

pipeline= healthcheck proxy-logging cache tempauth proxy-logging proxy-server

[app:proxy-server]

use = egg:swift#proxy

allow_account_management = ture

account_autocreate = true

[filter:tempauth]

use = egg:swift#tempauth

user_system_root= testpass .admin http://192.168.1.211:8090/v1/AUTH_system

user_admin_admin = admin .admin .reseller_admin

user_test_tester = testing .admin

user_test2_tester2 = testing2 .admin

user_test_tester3 = testing3

[filter:healthcheck]

use = egg:swift#healthcheck

[filter:cache]

use = egg:swift#memcache

memcache_servers = 192.168.1.211:11211

[filter:ratelimit]

use = egg:swift#ratelimit

[filter:domain_remap]

use = egg:swift#domain_remap

[filter:catch_errors]

use = egg:swift#catch_errors

[filter:cname_lookup]

use = egg:swift#cname_lookup

[filter:staticweb]

use = egg:swift#staticweb

[filter:tempurl]

use = egg:swift#tempurl

[filter:formpost]

use = egg:swift#formpost

[filter:name_check]

use = egg:swift#name_check

[filter:list-endpoints]

use = egg:swift#list_endpoints

[filter:proxy-logging]

use = egg:swift#proxy_logging

[filter:bulk]

use = egg:swift#bulk

[filter:container-quotas]

use = egg:swift#container_quotas

[filter:slo]

use = egg:swift#slo

[filter:dlo]

use = egg:swift#dlo

[filter:account-quotas]

use = egg:swift#account_quotas

[filter:gatekeeper]

use = egg:swift#gatekeeper

[filter:container_sync]

use = egg:swift#container_sync


创建account、container以及object rings

> cd/etc/swift

/etc/swift> sudo swift-ring-builder account.builder create 18 3 1

/etc/swift> sudo swift-ring-builder container.builder create 18 3 1

/etc/swift> sudo swift-ring-builder object.builder create 18 3 1


注:

18代表2的18次幂,这个数字取决与你希望一个ring中会有多少个partition,3代表object的副本数,1代表至少一个小时后才能被移动。


让ring记录每个storage存储设备

/etc/swift> export ZONE=1

/etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.212

/etc/swift> export WEIGHT=100

/etc/swift> export DEVICE=sdb1


/etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT

WARNING: No region specified for z1-192.168.1.212:6002/sdb1. Defaulting to region 1.

Device d4r1z1-192.168.1.212:6002R192.168.1.212:6002/sdb1_"" with 100.0 weight got id 4


/etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT

WARNING: No region specified for z1-192.168.1.212:6001/sdb1. Defaulting to region 1.

Device d0r1z1-192.168.1.212:6001R192.168.1.212:6001/sdb1_"" with 100.0 weight got id 0


/etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT

WARNING: No region specified for z1-192.168.1.212:6000/sdb1. Defaulting to region 1.

Device d0r1z1-192.168.1.212:6000R192.168.1.212:6000/sdb1_"" with 100.0 weight got id 0


/etc/swift> export ZONE=2

/etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.213


/etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT

WARNING: No region specified for z2-192.168.1.213:6002/sdb1. Defaulting to region 1.

Device d1r1z2-192.168.1.213:6002R192.168.1.213:6002/sdb1_"" with 100.0 weight got id 1


/etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT

WARNING: No region specified for z2-192.168.1.213:6001/sdb1. Defaulting to region 1.

Device d1r1z2-192.168.1.213:6001R192.168.1.213:6001/sdb1_"" with 100.0 weight got id 1


/etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT

WARNING: No region specified for z2-192.168.1.213:6000/sdb1. Defaulting to region 1.

Device d1r1z2-192.168.1.213:6000R192.168.1.213:6000/sdb1_"" with 100.0 weight got id 1


/etc/swift> export ZONE=3

/etc/swift> export STORAGE_LOCAL_NET_IP=192.168.1.214


/etc/swift> sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT

WARNING: No region specified for z3-192.168.1.214:6002/sdb1. Defaulting to region 1.

Device d12r1z3-192.168.1.214:6002R192.168.1.214:6002/sdb1_"" with 100.0 weight got id 12


/etc/swift> sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT

WARNING: No region specified for z3-192.168.1.214:6001/sdb1. Defaulting to region 1.

Device d2r1z3-192.168.1.214:6001R192.168.1.214:6001/sdb1_"" with 100.0 weight got id 2


/etc/swift> sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT

WARNING: No region specified for z3-192.168.1.214:6000/sdb1. Defaulting to region 1.

Device d2r1z3-192.168.1.214:6000R192.168.1.214:6000/sdb1_"" with 100.0 weight got id 2


平衡ring

/etc/swift> sudo swift-ring-builder account.builder rebalance

Reassigned 262144 (100.00%) partitions. Balance is now 0.00.

/etc/swift> sudo swift-ring-builder container.builder rebalance

Reassigned 262144 (100.00%) partitions. Balance is now 0.00.

/etc/swift> sudo swift-ring-builder object.builder rebalance

Reassigned 262144 (100.00%) partitions. Balance is now 0.00.


如果操作失误也可以移除

/etc/swift> sudo swift-ring-builder account.builder remove z1-192.168.1.212:6020

d0r1z1-192.168.1.212:6020R192.168.1.212:6020/sdb1_"" marked for removal and will be removed next rebalance.


不过我发现最后一个无法删除,只能在新建另外一个,然后删除想要删除


删除后需要rebalance

/etc/swift> sudo swift-ring-builder account.builder rebalance

Reassigned 262144 (100.00%) partitions. Balance is now 0.00.


当然也可以加入设备名

/etc/swift> sudo swift-ring-builder account.builder remove z1-192.168.1.212:6020/sdb1

d0r1z1-192.168.1.212:6020R192.168.1.212:6020/sdb1_"" marked for removal and will be removed next rebalance.


拷贝ring到其他所有节点

所有节点

sudo chown -R swift:swift /etc/swift


scp swift/*.ring.gz swift@192.168.1.212:/etc/swift

scp swift/*.ring.gz swift@192.168.1.213:/etc/swift

scp swift/*.ring.gz swift@192.168.1.214:/etc/swift


开启proxy

/etc/swift> sudo swift-init proxy start


需要注意的是对于account.builder 、container.builder以及object.builder的命名一定要严格按照小写,并且名字不能修改,不然会出现类似下面的错误

sudo swift-init proxy start

Starting proxy-server...(/etc/swift/proxy-server.conf)

WARNING: SSL should only be enabled for testing purposes. Use external SSL termination for a production deployment.

Traceback (most recent call last):

File "/usr/bin/swift-proxy-server",line 23,in <module>

sys.exit(run_wsgi(conf_file,'proxy-server',default_port=8080,**options))

......


IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz'


因为/etc/swift/目录下是Container.ring.gz


7、安装配置storage节点(三个节点安装配置)

sudo zypper install openstack-swift-account openstack-swift-container openstack-swift-object python-xml

sudo mkdir -p /srv/node/sdb1

sudo chown swift:swift /srv/node/sdb1


设备配置xfs卷

机器为底层lvm逻辑卷格式

# lvcreate -n swiftlv -L 10G vg0


# mkfs.xfs /dev/mapper/vg0-swiftlv


把挂载信息写入配置文件

# vi /etc/fstab

/dev/mapper/vg0-swiftlv /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0


重新加载配置文件

# mount -a


查看

# mount

/dev/mapper/vg0-swiftlv on /srv/node/sdb1 type xfs (rw,noatime,logbufs=8)


按实际情况修改rsync配置文件

/etc> sudo vi rsyncd.conf

uid = swift

gid = swift

log file = /var/log/rsyncd.log

pid file = /var/run/rsyncd.pid

address = 192.168.1.212


[account]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/account.lock


[container]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/container.lock


[object]

max connections = 2

path = /srv/node/

read only = False

lock file = /var/lock/object.lock


其他两台注意:

address = 192.168.1.213

address = 192.168.1.214


/etc> sudo vi default/rsync

RSYNC_ENABLE=true


# service rsyncd start


/etc>rsync rsync://pub@192.168.1.212

account

container

object


/etc> rsync rsync://pub@192.168.1.213

account

container

object


/etc> rsync rsync://pub@192.168.1.214

account

container

object


修改account、container以及object配置文件

~> grep -v "^#" /etc/swift/account-server.conf |grep -v "^$"

[DEFAULT]

bind_ip = 192.168.1.212

bind_port = 6002

swift_dir = /etc/swift

devices = /srv/node

workers = 2

[pipeline:main]

pipeline = account-server

[app:account-server]

use = egg:swift#account

[filter:healthcheck]

use = egg:swift#healthcheck

[filter:recon]

use = egg:swift#recon

[account-replicator]

[account-auditor]

[account-reaper]


~> grep -v "^#" /etc/swift/container-server.conf |grep -v "^$"

[DEFAULT]

bind_port = 6001

workers = 2

[pipeline:main]

pipeline = container-server

[app:container-server]

use = egg:swift#container

[filter:healthcheck]

use = egg:swift#healthcheck

[filter:recon]

use = egg:swift#recon

[container-replicator]

[container-updater]

[container-auditor]

[container-sync]


~> grep -v "^#" /etc/swift/object-server.conf |grep -v "^$"

[DEFAULT]

bind_port = 6000

workers = 2

[pipeline:main]

pipeline = object-server

[app:object-server]

use = egg:swift#object

[filter:healthcheck]

use = egg:swift#healthcheck

[filter:recon]

use = egg:swift#recon

[object-replicator]

[object-updater]

[object-auditor]


拷贝配置文件到其他两台机器,记得其他两台IP做修改


启动存储服务

etc/swift> sudo swift-init all start

Starting container-updater...(/etc/swift/container-server.conf)

Starting account-auditor...(/etc/swift/account-server.conf)

Starting object-replicator...(/etc/swift/object-server.conf)

Unable to locate config for proxy-server

Starting container-replicator...(/etc/swift/container-server.conf)

Starting object-auditor...(/etc/swift/object-server.conf)

Starting object-expirer...(/etc/swift/object-expirer.conf)

Starting container-auditor...(/etc/swift/container-server.conf)

Starting container-server...(/etc/swift/container-server.conf)

Starting account-server...(/etc/swift/account-server.conf)

Starting account-reaper...(/etc/swift/account-server.conf)

Starting container-sync...(/etc/swift/container-server.conf)

Starting account-replicator...(/etc/swift/account-server.conf)

Starting object-updater...(/etc/swift/object-server.conf)

Starting object-server...(/etc/swift/object-server.conf)


可以使用curl来和存储进行交互

curl -k -v -H'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.211:8080/auth/v1.0

curl -k -v -H'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass'


http://192.168.1.211:8080/auth/v1.0


如果curl出错:

swift@x-shcs-creditcard-v01:~> curl -k -v -H'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.211:8080/auth/v1.0

* Protocol "https" not supported or disabled in libcurl

* Closing connection -1

curl: (1) Protocol "https" not supported or disabled in libcurl


则需要编译curl

# tar jxf curl-7.19.0.tar.bz2

# cd curl-7.19.0/

# ./configure --prefix=/usr/local/curl --with-ssl=/usr/local/openssl

curl version: 7.19.0

Host setup: x86_64-unknown-linux-gnu

Install prefix: /usr/local

Compiler: gcc

SSL support: enabled (OpenSSL)

SSH support: no (--with-libssh2)

zlib support: enabled

krb4 support: no (--with-krb4*)

GSSAPI support: no (--with-gssapi)

SPNEGO support: no (--with-spnego)

c-ares support: no (--enable-ares)

ipv6 support: enabled

IDN support: no (--with-libidn)

Build libcurl: Shared=yes,Static=yes

Built-in manual: no (--enable-manual)

Verbose errors: enabled (--disable-verbose)

SSPI support: no (--enable-sspi)

ca cert bundle: no

ca cert path: /etc/ssl/certs/

LDAP support: no (--enable-ldap / --with-ldap-lib / --with-lber-lib)

LDAPS support: no (--enable-ldaps)


# make && make install


/usr/local/bin # mv curl curl.bak

/usr/bin # mv curl curl.bak


# ln -s /usr/local/curl/bin/curl /usr/bin/curl


检测swift是否正常工作,出现像下面的输出就说明正常

/etc/swift> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass stat

Account: AUTH_system

Containers: 0

Objects: 0

Bytes: 0

Content-Type: text/plain; charset=utf-8

X-Timestamp: 1490690568.46981

X-Trans-Id: tx59301de1e9244b70a2065-0058da2208

X-Put-Timestamp: 1490690568.46981


新建一级子目录

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass post container1


忘存储目录上传文件

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container1 /etc/swift/*.ring.gz


查看存储里的目录和文件

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list


下载存储目录里的文件到本地

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass download container1

猜你在找的Swift相关文章