In my lab I am working on a springXD project and I needed a linux host to mount a ViPR s3 bucket. Google led me to the s3fs project. S3fs is a FUSE-based file system backed by Amazon S3 or s3 compatable storage. It allows a linux host to mount a bucket as a local file system. This blog will show how to install s3fs and configure an EMC ViPR s3 bucket for use. This install is done with CentOS 6.4 and ViPR 1.1. You can see my previous post on how to set up object servies for ViPR HERE
On the CentOS Host install the following packages with YUM:
yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap
After the YUM install we install FUSE with the following commands:
NOTE: FUSE version is latest as of March 5, 2014
cd /usr/local/src
wget http://sourceforge.net/projects/fuse/files/latest/fuse-2.9.3.tar.gz
tar -xzvf fuse-2.9.3.tar.gz
mv fuse-2.9.3 fuse
cd fuse/
./configure --prefix=/usr
make
make install
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
ldconfig
modprobe fuse
pkg-config --modversion fuse
Next install s3fs with the following commands:
NOTE: s3fs version is latest as of March 5, 2014
cd /usr/local/src
wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
tar -xzvf s3fs-1.74.tar.gz
mv s3fs-1.74 s3fs
cd s3fs
./configure --prefix=/usr
make
make install
From the emc community page https://community.emc.com/docs/DOC-27823 we can find the networking guide for ViPR Data services. For applications to access ViPR s3 buckets we must create DNS or local host records. First we need an “A” record for our data services VM that was deployed to host ViPR data services. For this blog we have the following
Object.cto.emc.local 10.10.81.160
Next we need to create a 2nd “A” record for the bucket. In this case my bucket will be called buckets3 so the DNS record is
Buckets3.object.cto.emc.local 10.10.81.160
Note that the bucket IP address is the same as the data services IP address. Every bucket we create that we want an s3 application to have access to will need an associated “A” record created.
Next we need to create a Base URL in ViPR and a bucket. The base url is the FQDN of the data services VM. It is used by application to prepend the base URL with the bucket name.
|
Under Data Services, Base URL, create a URL with the FQDN of the ViPR data services VM |
|
Next we need to create a bucket. From the user menu select service catalog and click on Data Services |
|
Select Create Bucket for Data Services |
|
Give the bucket a name. From the drop down menus choose a virtual pool, project, object access and HDFS access. Click Save Note : In this example my bucket will have both object and S3 for access. |
Back on the CentOS box we can finish the s3fs setup by creating a password file
vim /etc/passwd-s3fs
Enter the following into the file:
root:HzoWC4x2/bSt+CySRdkeqMbrajjHZPZyoKg4OQn8
Above is the AccessKeyID:SecretAccessKey The Secret key can be accessed from ViPR and is shown below.
chmod 640 /etc/passwd-s3fs
|
Retrieve the Secret Data store key from the ViPR web page under root->Manage data store Keys
|
Now we can mount the s3 bucket
|
The command to mount the bucket is: s3fs -o host=http://object.cto.emc.local:9020 buckets3 /mnt/s3 Note that the –o option points to the Base url and needs the ViPR data service port 9020 for s3 HTTP A df –h shows the s3fs mount |
Comments