TheRuddyDuck

EMC ViPR- Block and Object Storage for Openstack DEMO

This blog features a Demo of Using EMC ViPR for all your storage needs in an Openstack environment. Block storage for Openstack Cinder using iscsi and Object based storage for Glance, all through one management interface. Couple of things to note as you watch the demo. 1st the real coolness about Object based storage in Openstack is its ability to take a storage bucket and make it accessible to multiple API's like Swift, S3, and Atmos. So you can load data however you choose and its then available to all these object interfaces. Besides offering Block and Object, ViPR can also offer NFS services. So if you interested in supporting Live Migration features in KVM, you can set these services up dynamically (the video Doesn’t show this part). Finally, in my demo I only have one block storage array, but ViPR has the ability to setup virtual pools that span multiple array’s.

Here is the Demo:

EMC Vipr- Block and Object Storage for Openstack from James Ruddy on Vimeo.

 

Demo of Integrating EMC Isilon with Cloudera Manager for HDFS using the Cloudera API

In my last blog post I showed how to integrate EMC Isilon into Cloudera Manager for native HDFS connections using the Cloudera API. This blog features of demo of that integration. It shows how easy it is to configure Cloudera to use Isilon for HDFS, and highlights the value of using a shared storage model for HDFS. 

 

 

Configure Cloudera Manager HDFS Service to Integrate with EMC Isilon

Cloudera Manager provides an overall platform to manage and monitor all the nodes and services that make up a CDH cluster. It accomplishes this task through the use of agents. Each managed object has an agent that allows it to receive configuration information from Cloudera Manager, and to send performance stats. EMC Isilon cannot have the Cloudera Manager agent installed on it. Fortunately the Cloudera API ( http://cloudera.github.io/cm_api/) allows an unmanaged object to be added to Cloudera Manager. Not only can it be added as a host, but it can be assigned roles. Using the API we can add Isilon as a host, and assign it the necessary roles to run HDFS. Cloudera manager can then push the configuration to all the nodes in the cluster.

Prerequisits

A running Cloudera Cluster with Cloudera Manager. See these links:

Deploy Cloudera with Puppet

Configure Hadoop with Cloudera Manager

EMC Isilon configured for HDFS with correct permissions for Cloudera. See these links:

Configure HDFS on EMC Isilon

Cloudera permission on EMC Isilon

Integrate Isilon with the HDFS service

From the main page click the drop down arrow to the right of the Cluster name. Select “Rename Cluster”

Rename the default cluster name to a name without any spaces in it. This makes it easier to use the API in later steps. In this example we name the cluster “Cluster1”

The cluster name is now changed.

The Cloudera API client code is need to interact with the Cloudera API. From the Manager server download it from github with the following command: wget https://github.com/cloudera/cm_api/archive/master.zip

Unzip the file with the following command: unzip master

Change directory into the cm_api-master/python directory with the following command: cd cm_api-master/python

Install the API client with the following command: python setup.py install The CM API client is now installed on your system

Now we use the python and the CM API client to add our Isilon cluster to Cloudera Manager. It’s added to the cluster, hdfs and mapreduce service, default rack, and give the NameNode role in hdfs. The commands to do this are below: python import socket from cm_api.api_client import ApiResource api = ApiResource("cdhmanager.ebc.emc.local",username="admin",password="admin",version=1) cluster = api.get_cluster("Cluster1") hdfs = cluster.get_service("hdfs1") mapred = cluster.get_service("mapreduce1") name = 'isilon1.cto.emc.local' host = api.create_host(name, name, socket.gethostbyname(name), "/default") nn = hdfs.create_role("hdfs01-nn", "NAMENODE", name) After running the API commands, we can see some errors and warnings pop up in Cloudrea Manager. Since the Isilon cluster does not have an agent the error in the host service will always be present. The warnings in the HDFS service will be rectified in later steps. Click on the HDFS service

Once the HDFS service page opens, click on the Instances tab.

We see all the nodes and their roles listed. Notice the Isilon host is stopped but health is good. It also has the namenode role assigned to it.

In the middle of the page click “Add”

Add the Datanode and SecondaryNameNode roles to Isilon.

Now Isilon has 3 roles.

Click the box next to “Name” so that all hosts are selected.

Click “Actions for Selected” and choose stop.

Click “Stop”. The HDFS service is stopped on all nodes.

The stop command runs successfully on all nodes.

Using the drop down, sort the hosts by the role. Select “NameNode”. You will see the role listed on both Isilon and the Cloudera Manager as we defined in Table1. Select the Manager.

Click “Actions for Selected” and choose Delete.

Confirm the deletion of the NameNode role from the Cloudera Manager server by clicking “Delete”

Using the drop down, sort the hosts by the role. Select “SecondaryNameNode”. You will see the role listed on both Isilon and the Cloudera Manager as we defined in Table1. Select the Manager.

Click “Actions for Selected” and choose Delete.

The roles are now set under the HDFS service so that all HDFS connections will be made to Isilon

Click the “Home” button in the top left corner. Notice we have more warnings under the HDFS service but no errors.

Click the arrow to the right of the cluster name. Select “Deploy Client Configuration”. The new HDFS configuration will be pushed out to the cluster.

Confirm deployment by clicking “Deploy Client Configuration”

The configuration is successfully deployed.

Confirm the configuration on any of the nodes with the following command: cat /etc/hadoop/conf.cloudera.hdfs1/core-site.xmk |grep isilon1 NOTE: replace isilon1 with the name of your Isilon cluster

Back on the home screen select the arrow to the right of the madreduce service. Click “Restart”

Confirm restart by clicking “Restart”. The oozie and hive services will also restart. The Cloudera Hadoop cluster is now ready to use Isilon for HDFS.

Resolve HDFS warnings

On the home screen click the warning wrench next to the HDFS service.

Of the 6 warning listed, 3 can be resolved. The 3 warning related to Isilon cannot be resolved because of the llack of agent on the Isilon cluster. Clicking on the warning will bring you to the configuration page to resolve the warning. Select the defaults and the warning will be resolved.          On the home page we have now resolved all the warnings non Isilon related.

Multi-Node Glance with EMC ViPR

In a previous blog post I showed how to setup Openstack glance to use the EMC ViPR Swift API. That post was done on a single node. If you want to use Glance in an Openstack enviornment with more than one node you only have to make a few changes to the glance-api.conf file. And this change only has to be made if the messaging service is not running on the same node. 

In my lab I’ve deployed Openstack Havana in a 3 node configuration using packstack. I created an answer file and edited it so that services (Glance, Neutron, Cinder, Nova, etc) were split amongst the nodes. Nova2 has the Glance service, and Nova1 has the qpid service.

 I follow the directions from my previous post to configure both EMC ViPR and the /etc/glance/glance-api.conf. 

I then edit the following in the /etc/glance/glance-api.conf file:

The setting above address the issue found in bug 1020025

Glance notification service wouldn't connect to qpid nodes. Fix: Replaced qpid_host with qpid_hostname in glance-api.

Also the username and password were set in my anwser file for packstack.

Restart the openstack-glance-api service and the EMC ViPR Swift API will be used for Glance image repositories.

 

Multi-Node Cinder with EMC ViPR

Multi-Node Cinder with EMC ViPR

In a previous series of blogs I showed how to setup EMC Vipr to be used for Cinder block storage in an Openstack environment (the series starts here). The blog series showed how to create storage on a single node instance of Openstack. In real world scenarios there will be multiple nodes in your Openstack environment, and you may want to make different storage pools available to the nodes. To access multiple pools of storage with EMC Vipr, we need to setup the cinder volume service on multiple nodes. Each instance of the ViPR Cinder Driver can be used to manage only one virtual array and one virtual pool within ViPR.  The ViPR virtual array can be accessed by multiple Cinder driver instances, but a virtual pool can only be accessed by one driver.

In my lab I’ve deployed Openstack Havana in a 3 node configuration using packstack. I created an answer file and edited it so that services (Glance, Neutron, Cinder, Nova, etc) were split amongst the nodes. Nova3 has the Cinder service. When deploying the Cinder service on subsequent nodes you need to create a connection to the mysql database. I also edited to the answer file to set the sql connection passwords instead of using an auto generated password.

 

 

 

 

Above you can see my lab enviornment. 3 nodes with Nova installed, the 2 virtual arrays, and the 4 pools physical pools that have been abstracted into virtual pools. I have already setup the EMC ViPR drive on Node3 which is using the VNX-Tier0 pool. The rest of this blog will show how to setup additional Cinder volme services and use the EMC ViPR driver to access other virtual pools.

On the nova1 node, install cinder yum install openstack-cinder

Edit the /etc/cinder/cinder.conf file and add the following lines: sql_connection=mysql://cinder:[email protected]/cinder qpid_hostname=nova1.cto.emc.local The sql connection is to the server with the mysql database, in this case nova1. The connection is made with the user “cinder” with a password of “password” The qpd service is also located on nova1

Set the SCSI service to turn on at boot, start it, and start the openstack-cinder-volume service: chkconfig tgtd on service tgtd start service openstack-cinder-volume start cinder-manage host list Check that the connection is working with the cinder-manage host list command.

Now we install the ViPR Cinder driver as normal. See this post for details (HERE). A summary of the commands is below: cd /tmp wget github.com/emcvipr/controller-openstack-cinder/archive/master.zip unzip master cd controller-openstack-cinder-master/cinder/volume/drivers/emc cp -avr vipr /usr/lib/python2.6/site-packages/cinder/volume/drivers/emc mkdir /usr/cookie chown cinder /usr/cookie

Edit the cinder.conf file again and add the Vipr Driver information. The screen shot above shows cinder.conf file with all the correct information for my environment. volume_driver = cinder.volume.drivers.emc.vipr.emc_vipr_iscsi.EMCViPRISCSIDriver vipr_hostname=10.10.81.53 vipr_port=4443 vipr_username=root vipr_password=password vipr_tenant=Provider Tenant vipr_project=Cinder vipr_varray=VNX-Iscsi-Block vipr_cookiedir=/usr/cookie rpc_response_timeout=300

Create the volume in Openstack and back it with a virtual pool. In this example we are creating the Openstack volume “Tier1” with the first command and backing it with the virtual pool “VNX-Tier1” with the second command. Copy the ID number listed after the first command and insert it as the “Type-key” for the second command: cinder --os-username admin --os-password password --os-tenant-name admin --os-auth-url=http://nova1.cto.emc.local:35357/v2.0 type-create Tier2 cinder --os-username admin --os-password password --os-tenant-name admin --os-auth-url=http://nova1.cto.emc.local:35357/v2.0 type-key 162c6219-8cc7-43c8-86e3-db60f24641cb set ViPR:VPOOL=VNX-Tier2

On the Horizon dashboard you will see the Volume type listed

The next series of commands authenticate the driver against the ViPR vapp, adds the host as a physical asset, inserts the iscsi initator into the correct network pool and verifies connectivity of the host to the physical array: python viprcli.py authenticate -u root -d /usr/cookie cat /etc/iscsi/initiatorname.iscsi ---ADD INITIATOR NAME IN THE NEXT COMMAND--- python viprcli.py openstack add_host -name nova1.cto.emc.local -wwpn iqn.1994-05.com.redhat:f4a0e539edc iscsiadm -m discovery -t sendtargets -p 192.168.1.101 --ADDRESS of VNX service openstack-cinder-volume restart The pool can now be used by Openstack

Using the horizon dashboard, create a volume

Use the volume “Type” we just created.

Verify the creation on Vipr by checking the resources for the Cinder project

In unisphere we can see the LUN was created in the pool.

The storage environment is a VNX with 3 tiers of storage, deployed as 3 pools. A Virtual array has been created and the physical pools have been abstracted into virtual pools. To add subsequent nodes repeat the steps above.

I then go on to add the Cinder service to Node2 and give it access to the VNX-Tier2 virtual pool. Above you can see 3 volumes that have been created in Horizon. One per pool.