Deploy Cloudera Manager with Puppet

 

In my last blog post I had started creating some puppet manifests to prepare an environment to deploy Cloudera Manager. I then discovered that Cloudera Manager does the creation of users, groups and permissions on cluster nodes. I still wanted to use puppet to prep the environment so I went to the puppet forge to retrieve a module for installing cloudera. Puppet forge is a central repository of puppet modules written and shared by the community. It’s an invaluable resource to help utilize puppet automation tasks, partially if you’re like me and not the world’s greatest coder. On the forge I found a great module written by Mike Arnold from razorsedge that installs and configures the Cloudera Manager server, and installs agents on cluster nodes. This module allows you to quickly spin up an environment, then use Cloudera manager to install Hadoop.

Mike built his module follwoing the instalation path in the Cloudera manager install guide. The guide list 3 different paths for install. A- automated, B- Your method, and C- Tar ball. The razoredge module follows path B. It installs the Cloudera server and requirements (JAVA, Repo’s, agents, and database), and configures them. It can then also deploy the Cloudera agents to nodes. The module is setup to support both rpm or Parcels  The rest of this blog will show how to setup both deployments types.

My previous blog post shows how to install and setup both the puppetmaster, and client nodes. I installed the puppet master on what would be my Cloudera manager server.

Prior to install

After installing the puppet moduel (below) you may need to edit the scm agent file. If you are only using hostnames on your linux boxes, not FQDN edit the following template so that listening ports are configured corrctly. If you are using FQDN then you do not need to edit this file. vim /etc/puppet/modules/cloudera/templates/scm-config.ini.erb Change the variable fqdn to hostname
The embedded database deployed with Cloudera Manager requires that SELinux is disabled.Turn off SELinux: sudo setenforce 0 Make sure it stays off on reboot vim /etc/selinux/config Change SELINUX = enforcing to SELINUX=disabled

Install

 Follow these steps to install the puppet module and edit your site.pp to begin deployment.

From the command line, install the puppet module puppet module install razorsedge/Cloudera After the install you can see the tree (above). I installed the module in the /etc/puppet directory. This is the defualt.
Create a site.pp manifest vim /etc/puppet/manifest/site.pp
To deploy the RPM install method define the classes as listed above with the correct variables for your cm server, and Cloudera version. Below shows the parcel method Run the puppet agent on the server. In my lab I am using the same server to be my puppet master and Cloudera Manager server. puppet agent –t –verbose NOTE: The first run through will have errors. This is because the Repos aren’t fully setup yet. This requires you to execute the puppet run a2 nd time. The 2nd run should be error free.
After the run completes you can check that the install ran correctly by listing the service and repo directories chkconfig –list |grep cloud (3 services should be installed) ls /etc/yum.rep (3 cloudera repos should be installed)
Verify the service are running service cloudera-scm-agent status service cloudera-scm-server status service cloudera-scm-server-db status
Logon to the manager using http://FQDN:7180 Username = admin Password = admin
On the cli of the puppet master vim the /etc/puppet/manifest/site.pp file and add the above for the node(s). You can add as many nodes as you want to use in your Hadoop cluster, just change the node name to match the cert name of registered puppet client. If you want to use parcels, just add the following line under the cm_server_host line for the node: use_parcels => true,
On the nodes run puppet –t –verbose After the install check that the agent installed chkconfig –list |grep cloud
Check on the Clouder manager web page that the node has been added and is ready to be configured.

Comments

[Rajasekhar]

Hi,
I followed the steps mentioned , but I am getting the following error related to Postgresql dependency for CM Database installation.
Could you please let me know how to avoid such errors...Thanks in advance.

Log:
Info: Applying configuration version '1392163961'
Notice: /Stage[main]/Cloudera::Cm::Repo/Yumrepo[cloudera-manager]/baseurl: baseurl changed 'http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.1.2/' to 'http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/'
Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install cloudera-manager-server-db' returned 1: Error: Package: cloudera-manager-server-db-4.8.1-1.cm481.p0.74.el6.x86_64 (cloudera-manager)
Requires: postgresql-server >= 8.0
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

Error: /Stage[main]/Cloudera::Cm::Server/Package[cloudera-manager-server-db]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install cloudera-manager-server-db' returned 1: Error: Package: cloudera-manager-server-db-4.8.1-1.cm481.p0.74.el6.x86_64 (cloudera-manager)
Requires: postgresql-server >= 8.0
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

Notice: /Stage[main]/Cloudera::Cm::Server/Exec[cloudera-manager-server-db]: Dependency Package[cloudera-manager-server-db] has failures: true
Warning: /Stage[main]/Cloudera::Cm::Server/Exec[cloudera-manager-server-db]: Skipping because of failed dependencies
Notice: /Stage[main]/Cloudera::Cm::Server/Service[cloudera-scm-server-db]: Dependency Package[cloudera-manager-server-db] has failures: true
Warning: /Stage[main]/Cloudera::Cm::Server/Service[cloudera-scm-server-db]: Skipping because of failed dependencies
Notice: /Stage[main]/Cloudera::Cm::Server/Service[cloudera-scm-server]: Dependency Package[cloudera-manager-server-db] has failures: true
Warning: /Stage[main]/Cloudera::Cm::Server/Service[cloudera-scm-server]: Skipping because of failed dependencies
Notice: /Stage[main]/Cloudera::Cm/File[scm-config.ini]/content:
--- /etc/cloudera-scm-agent/config.ini 2014-02-12 05:20:58.462716050 +0530
+++ /tmp/puppet-file20140212-18731-4v27lr 2014-02-12 05:45:59.830409544 +0530
@@ -3,7 +3,7 @@
###
[General]
# Hostname of Cloudera SCM Server
-server_host=rhel6vm3.example.com
+server_host=rhel6vm2.example.com

# Port that server is listening on
server_port=7182

Info: /Stage[main]/Cloudera::Cm/File[scm-config.ini]: Filebucketed /etc/cloudera-scm-agent/config.ini to main with sum 2460dd84fcaebc8e213df68397a6e3ce
Notice: /Stage[main]/Cloudera::Cm/File[scm-config.ini]/content: content changed '{md5}2460dd84fcaebc8e213df68397a6e3ce' to '{md5}568cd2c5550424b8eb49e6d96329ff92'
Info: /Stage[main]/Cloudera::Cm/File[scm-config.ini]: Scheduling refresh of Service[cloudera-scm-agent]
Notice: /Stage[main]/Cloudera::Cm/Service[cloudera-scm-agent]: Triggered 'refresh' from 1 events
Notice: Finished catalog run in 227.08 seconds
[root@RHEL6VM2 manifests]#
[root@RHEL6VM2 manifests]#
[root@RHEL6VM2 manifests]#
[root@RHEL6VM2 manifests]# chkconfig --list | grep cloud
cloudera-scm-agent 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cloudera-scm-server 0:off 1:off 2:off 3:on 4:on 5:on 6:off
[root@RHEL6VM2 manifests]#
[root@RHEL6VM2 manifests]# yum list | grep -i postgresql
pe-postgresql.x86_64 9.2.4-8.pe.el6 @puppet-enterprise-installer
pe-postgresql-server.x86_64 9.2.4-8.pe.el6 @puppet-enterprise-installer
[root@RHEL6VM2 manifests]#

[jim]

Raj,
I haven't seen this error before. Can you check that the correct repos are installed?
ls /etc/yum.repo/

Also post your site.pp
Thx
Jim

[Karthikasundari]

I followed automated Cloudera installation. It doesn't install the cloudera-manager-server-d package. i don't know why it is happening. can you please help me to solve this.

Log:

Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package cloudera-manager-server-db.x86_64 0:4.8.1-1.cm481.p0.74.el6 will be installed
--> Processing Dependency: postgresql-server >= 8.0 for package: cloudera-manager-server-db-4.8.1-1.cm481.p0.74.el6.x86_64
--> Finished Dependency Resolution
Error: Package: cloudera-manager-server-db-4.8.1-1.cm481.p0.74.el6.x86_64 (cloudera-manager)
Requires: postgresql-server >= 8.0
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest