TheRuddyDuck

How to build a Data Lake Part 1 –The Pilot

So I’ve seen a lot of blogs recently talking about the Data Lake. What it is and what it means. My favorite has been Steve Todd's which gives a good high level over of what a data lake is. In the EMC open innovations lab (OIL) we are constantly working with Internal EMC BU’s, different parts of the federation (RSA, VMware, Pivotal), and customers to explore and pilot new technologies. Recently we began working with a customer who had a unique challenge. Their old legacy analytic systems and databases could not help them with their manufacturing needs. In real time they want to be able to run analytics on data coming off of the assembly line while also comparing it to historical data. This is the perfect use case for a data lake! As I prepped for this pilot I found lots of blogs that talked about the theory and the “why” you should build a data lake. I also found lots of internal EMC/Pivotal docs about how to install different piece parts, but no overarching documentation on how to build a data lake. As always this blog focuses on the how. This Pilot is just kicking off and as it proceeds I will be updating this blog with the architecture, how to deploy and configure it, load data, and run analytic applications. Should be a fun project!

Openstack Icehouse Cinder and EMC ViPR

Last week EMC released their newest storage plugin for Icehouse Cinder. This version FOUND HERE now supports Havana, Icehouse, and EMC ViPR 1.1. I have previously blogged on how to setup the ViPR driver for version 1.0 and Havana (HERE). The plugin also now supports multiple backends simultaneously. The instillation of the newest version of the plugin has one major change, and that is the use of the ViPR CLI client to create volumes on ViPR virtual storage pools. This blog will show you how to setup the new driver

Enviornment

3 nodes installed with CentOS and RDO Icehouse

ViPR is backed by 2 storage systems. Isilon for NFS and VNX for iSCSI Block

The Physical arrays have been abstracted into Virtual array’s.

From the Virtual array we have created Virtual pools. This blog we will be using the Tier0-SSD Virtual pool to create cinder volumes.

We have alos created a project called ObjectProject. 

Install Cinder Driver

To install the cinder driver we first have to install the ViPR CLI on the Cinder node.

mkdir cli

cd cli

Download the CLI from your VIPR vm using the wget command

wget http://10.10.81.161:9998/cli

Extract the installer with the Tar command

tar -xf cli

Install the ViPR CLI

./Installer_viprcli.linux

The ViPR cli install begins. Note the instillation directory, Port number, and the ViPR hostname, which must be a FQDN.

Set the path environment using the source command

source /opt/vipr/cli/viprcli.profile

With the CLI installed we can begin the driver installation

cd /tmp

wget github.com/emcvipr/controller-openstack-cinder/archive/master.zip

Unzip the driver

unzip master

Copy the ViPR subdirectory into the cinder volume directory

cd controller-openstack-cinder-master/cinder/volume/drivers/emc/

cp -avr vipr /usr/lib/python2.6/site-packages/cinder/volume/drivers/emc/

Edit the cinder.conf file

vim /etc/cinder/cinder.conf

At the top of the file in the default section enter the following

volume_driver = cinder.volume.drivers.emc.vipr.emc_vipr_iscsi.EMCViPRISCSIDriver

vipr_hostname=vipr.ebc.emc.local

vipr_port=4443

vipr_username=root

vipr_password=password

vipr_cli_path=/opt/vipr/cli/bin

vipr_tenant=Provider Tenant

vipr_project=ObjectProject

vipr_varray=CinderiSCSI

vipr_cookiedir=/usr/cookie

rpc_response_timeout=300

Enter the correct information for your environment. Note that the default tenant in ViPR is “Provider Tenant”

Restart the cinder service

service openstack-cinder-volume restart

Create a volume type

cinder --os-username admin --os-tenant-name admin type-create Tier0

Map the volume type to a virual pool in ViPR

cinder --os-username admin --os-tenant-name admin type-key Tiero set ViPR:VPOOL=Tier0-SSD

Note type key can be the ID number or name.

On the ViPR GUI we add the cinder host. Click Add

  

Enter the host information. Click Save

The host is added to ViPR

Under the network tab select the netwok associated with the array. In this example it’s the ISCSI-NET network

You will see the 2 IP ports from the array already assigned to this network. Click the Add button and select Add Host Ports

Scroll through the list till you find the iqn of the cinder host. Select it and click Add

The Port is added to the network. Click Save

On the Openstack Horizon dashboard we can now create a volume. Give the volume a name and select the type created earlier. Click Create Volume

On the ViPR GUI you can see the volume being created by viewing the resource tab under users.

We can also view the creation of the volume in Unisphere.

We now have a ViPR iSCSI block volume in Cinder for use.

If you have any issues with LUN creation check the /var/log/cinder/volume.log file. Details of volume creation are logged there.

Install Configure and Use s3fs with the EMC ViPR s3 API

In my lab I am working on a springXD project and I needed a linux host to mount a ViPR s3 bucket. Google led me to the s3fs project. S3fs is a FUSE-based file system backed by Amazon S3 or s3 compatable storage. It allows a linux host to mount a bucket as a local file system. This blog will show how to install s3fs and configure an EMC ViPR s3 bucket for use. This install is done with CentOS 6.4 and ViPR 1.1. You can see my previous post on how to set up object servies for ViPR HERE 

On the CentOS Host install the following packages with YUM:

yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap

After the YUM install we install FUSE with the following commands:

NOTE: FUSE version is latest as of March 5, 2014

cd /usr/local/src

wget http://sourceforge.net/projects/fuse/files/latest/fuse-2.9.3.tar.gz

tar -xzvf fuse-2.9.3.tar.gz

mv fuse-2.9.3 fuse

cd fuse/

./configure --prefix=/usr

make

make install

export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/

ldconfig

modprobe fuse

pkg-config --modversion fuse

Next install s3fs with the following commands:

NOTE: s3fs version is latest as of March 5, 2014

cd /usr/local/src

wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz

tar -xzvf s3fs-1.74.tar.gz

mv s3fs-1.74 s3fs

cd s3fs

./configure --prefix=/usr

make

make install

From the emc community page https://community.emc.com/docs/DOC-27823 we can find the networking guide for ViPR Data services. For applications to access ViPR s3 buckets we must create DNS or local host records. First we need an “A” record for our data services VM that was deployed to host ViPR data services. For this blog we have the following

Object.cto.emc.local                          10.10.81.160

Next we need to create a 2^nd “A” record for the bucket. In this case my bucket will be called buckets3 so the DNS record is

Buckets3.object.cto.emc.local         10.10.81.160

Note that the bucket IP address is the same as the data services IP address. Every bucket we create that we want an s3 application to have access to will need an associated “A” record created.

Next we need to create a Base URL in ViPR and a bucket. The base url is the FQDN of the data services VM. It is used by application to prepend the base URL with the bucket name.

Under Data Services, Base URL, create a URL with the FQDN of the ViPR data services VM

Next we need to create a bucket. From the user menu select service catalog and click on Data Services

Select Create Bucket for Data Services

Give the bucket a name. From the drop down menus choose a virtual pool, project, object access and HDFS access. Click Save Note : In this example my bucket will have both object and S3 for access.

Back on the CentOS box we can finish the s3fs setup by creating a password file

vim /etc/passwd-s3fs

Enter the following into the file:

root:HzoWC4x2/bSt+CySRdkeqMbrajjHZPZyoKg4OQn8

Above is the AccessKeyID:SecretAccessKey The Secret key can be accessed from ViPR and is shown below.

chmod 640 /etc/passwd-s3fs

 

Retrieve the Secret Data store key from the ViPR web page under root->Manage data store Keys

Now we can mount the s3 bucket

The command to mount the bucket is: s3fs -o host=http://object.cto.emc.local:9020 buckets3 /mnt/s3 Note that the –o option points to the Base url and needs the ViPR data service port 9020 for s3 HTTP A df –h shows the s3fs mount

Configure Object Services for EMC ViPR 1.1

This blog is an update to my previous post on configuring object services for ViPR found HERE. That post was for ViPR 1.0. This post is for ViPR 1.1 and includes new configurations of Object Buckets and HDFS. The deployment for 1.1 is the same as 1.0 and can be found HERE

On the web client, under Data Services-> setup select the IP network to use. Use the same network that you have previously assigned to your file services.

Click Virtual Pools. Click Add

Create an Object Virtual pool. Give the virtual pool a name and select the type. Note this pool can support 3 different access types. Click Save

We now have a data services virtual pool that supports both Object (s3 and swift) and HDFS access.

Click Data Stores. Click Add

Give the data store a name. Use the drop down menu’s to select the Virtual array, File storage pool, and Virtual pool. An object file system will be created with in the virtual file pool. Choose the size of this file system. Click Save

The data store is created. Notice the mount point. You can verify its creation on your file array using putty or file system explore. All objects placed on the object store will go into this directory.

Click Tenant configuration. Give a tenant Namespace and use the drop downs to select the virtual pool and default project. Click Save.      A data pool is now setup on ViPR that can support HDFS, S3, and Swift access.