{"id":1170,"date":"2012-03-09T16:56:00","date_gmt":"2012-03-09T21:56:00","guid":{"rendered":"http:\/\/mossiso.com\/?p=1170"},"modified":"2013-02-01T12:45:08","modified_gmt":"2013-02-01T17:45:08","slug":"setting-up-a-hosting-environment-part-2-connecting-the-storage-array","status":"publish","type":"post","link":"https:\/\/mossiso.com\/2012\/03\/09\/setting-up-a-hosting-environment-part-2-connecting-the-storage-array\/","title":{"rendered":"Setting up a Hosting Environment &#8211; Part 2: Connecting the Storage Array"},"content":{"rendered":"<p>[<a title=\"Setting up a Hosting Environment: Part 1 \u2013 The servers\" href=\"http:\/\/mossiso.com\/2012\/03\/01\/setting-up-a-hosting-environment-part-1-the-servers.html\">See Part 1: The Servers<\/a>]<\/p>\n<p>One of the most frustrating parts of this set up was getting the storage array talking to the servers. I finally got it figured out. I&#8217;m using a StorageTek 2530 to connect to two SunFire X2100 M2&#8217;s via SAS (Serial Attached SCSI) cables. I put in a dual port SAS HBA (Host Bus Adapter) in the X2100 M2&#8217;s, but for real redundancy, I should have used two single port HBA&#8217;s. The Sun\/Oracle documentation is pretty good about how to physically set up the servers and storage array, but are pretty lacking from there on.<\/p>\n<div id='gallery-1' class='gallery galleryid-1170 gallery-columns-2 gallery-size-thumbnail'><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon landscape'>\n\t\t\t\t<a href='https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/images-1.jpg'><img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/images-1-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail\" alt=\"\" srcset=\"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/images-1-150x150.jpg 150w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/images-1.jpg 225w\" sizes=\"(max-width: 150px) 100vw, 150px\" \/><\/a>\n\t\t\t<\/div><\/figure><figure class='gallery-item'>\n\t\t\t<div class='gallery-icon landscape'>\n\t\t\t\t<a href='https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/images.jpg'><img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/images-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail\" alt=\"\" \/><\/a>\n\t\t\t<\/div><\/figure>\n\t\t<\/div>\n\n<h2>StorageTek 2530 Set Up<\/h2>\n<p><span style=\"color: #339966;\"><strong>Replace the parts in squares brackets below with whatever you want.<\/strong><\/span><\/p>\n<ul>\n<li>Install the Sun CAM software.\n<ul>\n<li>Grab the latest version from <a href=\"http:\/\/support.oracle.com\/\">http:\/\/support.oracle.com<\/a>\n<ul>\n<li>You&#8217;ll need an active support contract and have an account.<\/li>\n<li>Go to the \u2018Patches and Updates\u2019 tab.<\/li>\n<li>Click on the \u2018Product or Family (Advanced)\u2019 link<\/li>\n<li>In the \u2018Product is\u2019 section start typing in \u2018Sun Storage Common Array Manager (CAM)\u2019 and select it from the list<\/li>\n<li>In the \u2018Release is\u2019 section select the most recent version<\/li>\n<li>For the last section, select \u2018Platform\u2019 and then select \u2018Linux x86-64\u2019<\/li>\n<li>Click \u2018Search\u2019<\/li>\n<li>Click the \u2018Download\u2019 link for the software.<\/li>\n<li>Upload the tar file to the server.<\/li>\n<\/ul>\n<\/li>\n<li>Pre-requisite software that needs to be installed.\n<ul>\n<li><code>yum install ksh bc \/lib\/ld-linux.so.2 libgcc.i686 libstdc++.i686 libzip.i686 gettext<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Once CAM software is downloaded, un-zipped, un-tarred or what have you, change directories to HostSoftwareCD_6.9.0.16\/components and install the jdk available there:\n<ul>\n<li><code>rpm -Uvh jdk-6u20-linux-i586.rpm<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Next run the RunMe.bin file in the HostSoftwareCD_6.9.0.16 folder\n<ul>\n<li><code>.\/RunMe.bin -c<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Agree to all License Agreement stuffs<\/li>\n<li>Select the Typical install.<\/li>\n<\/ul>\n<\/li>\n<li>Add the <code>\/opt\/sun\/cam\/bin<\/code>folder to path\n<ul>\n<li>With root using tcsh add this to .tcshrc\n<ul>\n<li><code>setenv PATH ${PATH}:\/opt\/sun\/cam\/bin<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Then do <code>source .tcshrc<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Make sure there is an IP on the same subnet as the array (192.168.128.xxx)\n<ul>\n<li>Make a <code>\/etc\/sysconfig\/network-scripts\/ifcfg-eth1:1<\/code>file and put this in there\n<ul>\n<li>\n<div>DEVICE=\u201ceth1:1\u201d<br \/>\nBOOTPROTO=static<br \/>\nHWADDR=\u201cxx:xx:xx:xx:xx:xx:xx\u201d<br \/>\nIPADDR=192.168.128.xxx<br \/>\nNM_CONTROLLED=\u201cno\u201d<br \/>\nONBOOT=\u201cyes\u201d<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<li>Install the RAID Proxy Agent package located in the Add_On\/RaidArrayProxy directory of the latest CAM software distribution. (I found this to be optional.)\n<ul>\n<li><code>rpm -ivh SMruntime.xx.xx.xx.xx-xxxx.rpm<\/code><\/li>\n<li><code>rpm -ivh SMagent-LINUX-xx.xx.xx.xx-xxxx.rpm<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Register the StorageTek with the host. Process can take several minutes.\n<ul>\n<li><code>sscs register -d storage-system<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Once registered, you can name the array anything you want. Note what the array is named from the previous step.<\/li>\n<\/ul>\n<ul>\n<li><code>sscs modify -T [Array-Name] array ARRAY1<\/code><\/li>\n<\/ul>\n<li>Set up the storage profile, pool, disk, volume, mapping. Use the command line commands below, or set it up via the web interface. NOTE: This part only needs to be done on one of the hosts.\n<ul>\n<li>If using the web interface, you have to use a windows laptop hooked up to the local network (192.168.128.xxx), or perhaps a server in the same local network that is not running CentOS 6, which has a known issue where the web interface does not work. For the web interface connect to https:\/\/localhost:6789 using the laptop or server Administrator\/root account information.<\/li>\n<li><code>sscs create -a knox pool [Pool-Name]<br \/>\n<\/code><\/li>\n<li><code>sscs create -a knox -p [Pool-Name] -n 11 vdisk [Vdisk-Name]<br \/>\n<\/code><\/li>\n<li><code>sscs create -a knox -p [Pool-Name] -s max -v [Vdisk-Name] volume [Volume-Name]<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Create the host group and apply to host.\n<ul>\n<li><code>sscs create -a knox hostgroup [ApacheHosts]<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Create hosts and assign to hostgroup\n<ul>\n<li><code>sscs create -a knox -g [ApacheHosts] host [Host-Name]<\/code> and repeat for other hosts.<\/li>\n<\/ul>\n<\/li>\n<li>Map volume to host group\n<ul>\n<li><code>sscs map -a knox -g ApacheHosts volume Volume-Name<br \/>\n<\/code><\/li>\n<\/ul>\n<\/li>\n<li>The array volume should now be available as \/dev\/sdb and \/dev\/sdc because the hosts are connected by two SAS cables each.<\/li>\n<p>It took me a while to grasp the meaning for the different terms: pool, volume, volume groups, disks, etc. I drew up a chart with the appropriate commands to create the different aspects.<\/p>\n<p><a href=\"http:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-large wp-image-1171\" title=\"StorageTek2530-DiskSetup\" src=\"http:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup-1024x441.jpg\" alt=\"\" width=\"620\" height=\"267\" srcset=\"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup-1024x441.jpg 1024w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup-300x129.jpg 300w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup-250x107.jpg 250w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup-150x64.jpg 150w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup-624x268.jpg 624w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup.jpg 1290w\" sizes=\"(max-width: 620px) 100vw, 620px\" \/><\/a><\/p>\n<p>To utilize both cables connecting the server to the storage array, the OS needs to use multi-pathing. I had lots of troubles trying to set this up after the OS was installed, so I just let it be done by the installer. Here&#8217;s what should happen if you find the OS already installed and need to set up multi-paths.<\/p>\n<p><a href=\"http:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/multipath-server3.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1183\" title=\"multipath-server3\" src=\"http:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/multipath-server3.png\" alt=\"\" width=\"306\" height=\"323\" srcset=\"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/multipath-server3.png 306w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/multipath-server3-284x300.png 284w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/multipath-server3-250x263.png 250w, https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/multipath-server3-142x150.png 142w\" sizes=\"(max-width: 306px) 100vw, 306px\" \/><\/a><\/p>\n<ul>\n<li>Set up DM-Multipath\n<ul>\n<li>NOTE: This is taken care of during the OS installation.<\/li>\n<li>Multipath allows both SAS connections to the storage array to appear as one connection to the server. This allows for data to pass through even if one cable suddenly stops working, it seamlessly fails to the other path. For example, taken the image above, if the connection between hba1-&gt;cntrlr1 goes down, you still have connection hba2-&gt;cntrlr2. The OS sees one connection, and just uses whichever path is working.<\/li>\n<li>After Multipath is set up, the storage array will be available as a device at <code>\/dev\/mapper\/mpatha<\/code>. This will be the device to partition, format, and throw LVM on.<\/li>\n<li>Install the multipath program and dependents\n<ul>\n<li><code>yum install device-mapper-multipath<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Run <code>mpathconf --enable<\/code> to create a default <code>\/etc\/multipath.conf<\/code>file or create one using the following:\n<ul>\n<li>\n<div>#\u00a0 multipath.conf written by anacondadefaults {<br \/>\nuser_friendly_names yes<br \/>\n}<br \/>\nblacklist {<br \/>\ndevnode \u201c^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*\u201d<br \/>\ndevnode \u201c^hd[a-z]\u201d<br \/>\ndevnode \u201c^dcssblk[0-9]*\u201d<br \/>\ndevice {<br \/>\nvendor \u201cDGC\u201d<br \/>\nproduct \u201cLUNZ\u201d<br \/>\n}<br \/>\ndevice {<br \/>\nvendor \u201cIBM\u201d<br \/>\nproduct \u201cS\/390.*\u201d<br \/>\n}<br \/>\n# don\u2019t count normal SATA devices as multipaths<br \/>\ndevice {<br \/>\nvendor \u201cATA\u201d<br \/>\n}<br \/>\n# don\u2019t count 3ware devices as multipaths<br \/>\ndevice {<br \/>\nvendor \u201c3ware\u201d<br \/>\n}<br \/>\ndevice {<br \/>\nvendor \u201cAMCC\u201d<br \/>\n}<br \/>\n# nor highpoint devices<br \/>\ndevice {<br \/>\nvendor \u201cHPT\u201d<br \/>\n}<br \/>\nwwid \u201c3600508e000000000c9c1189277b84b05\u201d<br \/>\ndevice {<br \/>\nvendor TEAC<br \/>\nproduct DV-28E-V<br \/>\n}<br \/>\nwwid \u201c*\u201d<br \/>\n}<br \/>\nblacklist_exceptions {<br \/>\nwwid \u201c3600a0b80003abca4000007284f33c167\u201d<br \/>\n}<br \/>\nmultipaths {<br \/>\nmultipath {<br \/>\nuid 0<br \/>\ngid 0<br \/>\nwwid \u201c3600a0b80003abca4000007284f33c167\u201d<br \/>\nmode 0600<br \/>\n}<br \/>\n}<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<li>Set multipathd to start on boot, and if not on, turn it on\n<ul>\n<li><code>chkconfig multipathd on<\/code><\/li>\n<li><code>service multipathd start<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>[See Part 1: The Servers] One of the most frustrating parts of this set up was getting the storage array talking to the servers. I finally got it figured out. I&#8217;m using a StorageTek 2530 to connect to two SunFire X2100 M2&#8217;s via SAS (Serial Attached SCSI) cables. I put in a dual port SAS &hellip; <a href=\"https:\/\/mossiso.com\/2012\/03\/09\/setting-up-a-hosting-environment-part-2-connecting-the-storage-array\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Setting up a Hosting Environment &#8211; Part 2: Connecting the Storage Array<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1171,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[249,167],"tags":[179],"class_list":["post-1170","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-setting-up-a-hosting-environment","category-technical","tag-servers"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/StorageTek2530-DiskSetup.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9wosP-iS","_links":{"self":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts\/1170"}],"collection":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/comments?post=1170"}],"version-history":[{"count":14,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts\/1170\/revisions"}],"predecessor-version":[{"id":1188,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts\/1170\/revisions\/1188"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/media\/1171"}],"wp:attachment":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/media?parent=1170"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/categories?post=1170"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/tags?post=1170"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}