TheRuddyDuck

Data Lake Demo with Pivotal and Isilon

This is a continuing series on how to build a data lake. Welcome to part7

Part1

Part2

Part3

Part4

Part5

Part 6

Over the past couple of weeks Ive been blogging on how to create a data lake. These blogs included the architecture and how to install of a Data Lake with PivotalHD, GemfireXD,  and Hawq all running as VM's within a VMware infrastructure using EMC Isilon for HDFS.

The final step is to create an application and run some analytics. This demo is based off the retail demo that can be found on github:

https://github.com/Pivotal-Field-Engineering/retail-demo-xd

What you will see in the demo is an application in SpringXD that uses MAdLib,  built with Redis that shows data flow from ingestion, tapping data to the in memory data grid GemfireXD, with all data ending up in Isilon HDFS that can be queried using Hawq PXF.

A big thanks to Louis Mugano for help with the application build.

Data Lake Demo with Pivotal and Isilon from James Ruddy on Vimeo.

 

 

 

OpenStack Cinder and Software-Defined Storage (SDS)

OpenStack Cinder and Software-Defined Storage (SDS)

So a week after the OpenStack Summit in Atlanta kicked off I’ve had some time to digest all I saw and heard. Having had the chance to present with John Griffith, the PTL of Cinder, was an amazing experience. John recently published a blog on his thoughts on Software-Defined Storage (SDS) and Cinder http://griffithscorner.wordpress.com/2014/05/16/the-problem-with-sds-under-cinder/

This is a great article and one of the reasons that I’ve been so attracted to OpenStack is the diversity of opinions that drive the projects. I do however disagree with his thoughts on using SDS with Cinder. Let me give you some background.

I’ll start by saying that within the walls of EMC I have no hand in what drives our strategy or when/where we develop and contribute. What I have been doing within OCTO is taking the work of the different BU’s into the lab, looking at how all the piece parts work together, and feed back my thoughts to the PM’s. What I’ve seen is that as a whole we have been developing and maintaining drivers/plugins that we work towards getting into trunk. I agree with John in that this is what vendors should be doing. The table below shows what we are working on at EMC.

Where I disagree with John is that SDS has no place within Cinder. Particularly I disagree with his reasoning that SDS will allow vendors to short cut developing multiple drivers. I can’t see EMC, NetApp or HDS doing this. All vendors want to give their customers choice. More importantly, at least within EMC, each product group is responsible for their product having options for implementation in multiple environments. VNX isn’t going to stop developing its direct Cinder driver just because ViPR has gotten its driver into trunk. This is a personal opinion, but I just can’t see EMC CEO David Goulden telling the VMAX group to stop developing their Cinder driver just so EMC only has to develop and maintain one driver for OpenStack.

More importantly I disagree with John on the use of SDS with Cinder because of my experience working directly with customers. When building a OpenStack environment, as a storage administrator, having a central control plane for all my Cinder needs, commodity or enterprise, while also being able to offer Object services just makes everything easy peasy. I know it may sound disingenuous as an employee of EMC, but the use case for ViPR with OpenStack is my favorite. I also think that the community has validated this by their love of Ceph which had a Cinder driver integrated in the Folsom release.

Obviously in an open project you are going to conflict with vendors who each may have their own agenda or ideas on architecture and direction I just don’t think that we should assume that vendors have evil intentions or that their contributions to the community don’t help drive OpenStack adoption. As John stated during the Storage Experts panel, one of the best ways to advance OpenStack is to get more customers using it through different modes – some from trunk and some from vendor-specific contributions.

Install and integrate GemfireXD with EMC Isilon

This is a continuing series on how to build a data lake. Welcome to part6

Part1

Part2

Part3

Part4

Part5

GemFire XD is provided as a Pivotal HD installable component, for use with the Pivotal Command Center CLI installer. The CLI installation process installs multiple instances of GemFire XD. You can use ICM to install GemfireXD when you deploy your PHD environment or you can reconfigure a cluster to support it after deployment. In this example below we are reconfiguring an environment to support Gemfire after deploying PHD, HAWQ, and PFX with EMC Isilon for HDFS. Previous blogs have shown the initial deployment

Install

Download the PRTS tarball to the /tmp/PHD directory

Prepare the gemfire nodes

icm_client preparehosts --hostfile=/tmp/PHD/gemxdnode.txt --java=/tmp/PHD/jdk-6u26-linux-x64-rpm.bin

confirm they are prepared:

icm_client scanhosts -f /tmp/PHD/gemxdnode.txt

Unzip the tar files:

tar -zxf Pivotal_GemFireXD_05Beta2_b44694.tar.gz

tar -xf PRTS-1.0.0-9.tar.gz

Import the files:

icm_client import -s PRTS-1.0.0-9/

Stop the cluster and fetch the configuration file

icm_client stop -l DataLakeCluster

icm_client fetch-configuration -l DataLakeCluster -o /tmp/firstreconfig

Using vim edit file /tmp/firstreconfig.xml

add gfxd in service tag

add the following to <hostRoleMapping>

<gfxd>

<gfxd-locator>gemxdnode1.cto.emc.local</gfxd-locator>

<gfxd-server>gemxdnode2.cto.emc.local,gemxdnode3.cto.emc.local,gemxdnode4.cto.emc.local</gfxd-server>

</gfxd>

 

Save changes to the file. Now we will use this config file to reconfigure the cluster to use GemfireXD

icm_client reconfigure -l DataLakeCluster -c /tmp/firstreconfig/ -j /tmp/PHD/jdk-6u26-linux-x64-rpm.bin -d -i -y /usr/lib/gphd/gphdmgr/hawq_sys_config/

The cluster will be reconfigured with GemfireXD

Start GemfireXD

We defined the locater and server nodes in the cluster reconfig. Now we need to enable theses services on the servers.

On locator server

mkdir in /home for locator [gemxdnode1] and dir for locator [locator]

on locatore server:

sqlf locator start -dir=/home/locator -peer-discovery-port=10101 -client-port=1527 -jmx-manager-start=true -jmx-manager-http-port=7075

sqlf server start -dir=/home/gemxdnode1 -locators=localhost[10101] -client-port=1528

on node2:

/home/gemxdnode2

sqlf server start -dir=/home/gemxdnode2 -locators=gemxdnode2.cto.emc.local[10101] -client-port=1529

on node3:

/home/gemxdnode3

sqlf server start -dir=/home/gemxdnode3 -locators=gemxdnode1.cto.emc.local[10101] -client-port=1530

on node 4:

/home/gemxdnode4

sqlf server start -dir=/home/gemxdnode4 -locators=gemxdnode1.cto.emc.local[10101] -client-port=1531

When complete get this:

Starting SQLFire Server using locators for peer discovery: gemxdnode1.cto.emc.local[10101]

Starting network server for SQLFire Server at address localhost/127.0.0.1[1531]

Logs generated in /home/gemxdnode4/sqlfserver.log

SQLFire Server pid: 31246 status: running

Distributed system now has 5 members.

Other members: gemxdnode3(31256:datastore)<v3>:34792, gemxdnode1(13348:datastore)<v1>:3799, gemxdnode1(13097:locator)<v0>:12067, gemxdnode2(31268:datastore)<v2>:17481

7, gemxdnode2(31268:datastore)<v2>:17481

 

Gemxd is now up and running in the cluster

Enable Isilon to integrate with Pivotal Hawq.

This is a continuing series on how to build a data lake. Welcome to part5

Part1

Part2

Part3

Part4

In this blog post I’ll show you how to enable Isilon to integrate with Pivotal Hawq.

In these previous posts I explained the architecture and install of our data lake. At this point we have installed and configured HDFS for EMC Isilon, deployed and configured Pivotal Hadoop, and installed Hawq. Now we need to initiate our hawq server and enable the Pivotal Framework extension to access Isilon.

To enable Hawq to access EMC Isilon HDFS we need to first initiate the Hawq service then install the PFX RPM. To initiate the Hawq server we do the following.

On the hawq server:

Confirm the Isilon setting

cat /etc/gphd/hawq/conf/gpinitsystem_config |grep Isilon

Your DFS_URL should be the FQDN of the Isilon cluster with the port number 8020 and the /hawq_data directory. Example for my environment is:

DFS_URL=isilon2.cto.emc.local:8020/hawq_data

To start the service:

su gpadmin

source /usr/local/hawq/greenplum_path.sh

./etc/init.d/hawq init

This initiates the Hawq environment

gpstart –a

This starts the hawq service

On the Onefs file system in the HDFS directory under the hawq_data diretroy we should see something like the following:

Enable the Pivotal Framework Extension

First you will need to download the Pivotal framework extension RPM (PFXD). It needs to be installed on to the hawq master and all hawq segment nodes. Ill use a script that calls on a hawq text file (contains the FQDN's of the hawq server and segment nodes) 

for dest in $(<hawq.txt); do

scp /tmp/PHD/pxfd-1.0-1.noarch.rpm ${dest}:/tmp

done

This next script does the install

for dest in $(<hawq.txt); do

ssh ${dest} rpm -ivh /tmp/pxfd-1.0-1.noarch.rpm

done

On Hawq server we need to enable the frame work

vim /data1/master/gpseg-1/postgresql.conf

add the following line

pfx_local_storage=false

Restart the service

su -gpadmin /etc/init.d restart

As root we need to share ssh keys with the hawq server and segment nodes and export our Java profile. We will use the “set gpssh-exkeys”

source /usr/local/hawq/greenplum_path.sh

cd /usr/local/hawq-1.1.4.0/bin/

./gpssh -f /tmp/hosts.txt "echo "export JAVA_HOME=/usr/java/latest" > /etc/profile.d/java.sh"

./gpssh -f /tmp/hosts.txt chmod a+x /etc/profile.d/java.sh

./gpssh -f /tmp/hosts.txt /usr/lib/gphd/pxfd/start-pxf.sh

Next we start the nodes:

su gpadmin

source /usr/local/hawq/greenplum_path.sh

cd /usr/local/hawq-1.1.4.0/bin/

gpssh -f /tmp/hosts.txt /usr/lib/gphd/pxfd/start-pxf.sh

Hawq and the PFX are now up and running and integrated with EMC Isilon for HDFS

Openstack Summit - Laying Cinder blocks Session

Openstack Summit Atlanta session

I was very fortunate to have the chance to present with Ken Hui and John Griffith in a session tilted Laying Cinder Blocks (Volumes) Use Cases and Reference Architectures this week at the Openstack summit. The session was standing room only and it was a great honor to share the stage with 2 guys I greatly admire and respect. The presentation can be found here:

  

There were 3 parts to the session. The first 2 were presented by Ken and John and it was about the pros and cons of using commodity or enterprise storage with examples of why you would choose one over the other or use a combination. The big take away for me was John’s assertion that 70% of workloads in an openstack environment would be well suited for commodity storage. Architecting a mixed environment would be standard for most deployments and Cinder is designed specifically to support this type of architecture.

The 3^rd section was about the role of the storage admin in openstack. Here is the talk track that I used.

In the olden days the storage admin was the master of his domain. Requirements for storage would come from an application architect and would drive the storage architecture. Things like IOPS, Capacity, and latency were know quantities and would be considered as gospel during the design phase. If you have ever done a storage design for Oracle you will know that most reference architectures have explicit instructions on how to layout the data, where to put the ASM disk, swap files, Raid groups types to use for data and logs, getting down to the nitty gritty of even giving spindle count for those Raid groups.

Today the Cloud architect is driving the storage architecture. He doesn’t have the details of IOPS or capacity, he just wants tiers of storage and a REST interface to program against. He wants a pool of storage that can be consumed in many different ways by his end users that can scale or shrink based on the consumption habits. This is the basis of how Cinder works. Take a pool of storage that cinder offers as a volume to openstack tenants. They will create and destroy LUNS on demand and need to dynamically map the relationship of those LUNS to the opnestack instances that are using them.

So how does the storage admin give the cloud architect what he wants while still being able to deliver the type of data services (backup,DR, dedupe) that he is required to? How does he remain master of his domain while avoiding “Shrinkage” that can sometimes happen in the pool? I believe that this is where SDS fits the model.

I then showed the demo below asking the crowd to look beyond the EMC equipment used and see the idea of SDS for Cinder. A central controller that can provide a single interface to offer Cinder block storage on Enterprise and commodity hardware and provide all your Object services for Swift, HDFS and s3.

It was a great session and I had many people approach me after and during the rest of the day asking about EMC and Openstack, and more specifically about VIPR and SDS. Thanks agin to John and Ken for inviting me to this talk. 

Long live openstack :)

Video from the session