{"id":1158,"date":"2012-03-01T16:00:57","date_gmt":"2012-03-01T21:00:57","guid":{"rendered":"http:\/\/mossiso.com\/?p=1158"},"modified":"2013-02-01T12:45:19","modified_gmt":"2013-02-01T17:45:19","slug":"setting-up-a-hosting-environment-part-1-the-servers","status":"publish","type":"post","link":"https:\/\/mossiso.com\/2012\/03\/01\/setting-up-a-hosting-environment-part-1-the-servers\/","title":{"rendered":"Setting up a Hosting Environment: Part 1 – The servers"},"content":{"rendered":"
I’ve spent a lot of time at work setting up a few servers to be our new production environment. Much of it was accomplished by reading the documentation over and over again. Not much out there on the Net, so I’m hoping this series of posts benefits someone else out there.<\/p>\n
First of all, I’ll cover what set up I would like to achieve and why.<\/p>\n
I’m using two Sun SunFire X2100 M2 connected to a StorageTek 2530 with 4.5TB of drive space. The servers attach to the storage array via SCSI cables for quick data transfer speeds. The array also has the ability to handle iSCSI connections. This will give me a decent base set up, with room to grow.<\/p>\n
I’ll put the two servers in a cluster and make the services available over the cluster. They will share the storage using GFS2. In the future, I’ll add a couple of load balancer\/proxy machines to farm out the Web traffic, and add a couple more SunFire X2100 M2’s to take that load. One of the main reasons to set up a new configuration with new servers is to provide a clean environment for the many WordPress and Omeka installations we host. We’ve had to hang on to some legacy services to support some older projects, so this will allow us to keep up to date. It will also allow me to set up Apache and PHP to run as a server user, locked down to it’s own directory. That way each of the 100+ sites won’t be able to access any other site’s content. I picked CentOS as the OS because it has cluster and GFS2 options of RedHat, but without the cost.<\/p>\n
One of the most important things to have with servers is some form of remote management. That way you don’t need to trek down to the data center each time the server hangs while testing (and it happens a lot). For Sun systems, that means setting up the ELOM (Embedded Lights Out Manager).<\/p>\n RedHat in EL 6, and thereby CentOS, moved to Upstart instead of Sysv, so we create a new serial-ttyS1.conf file instead of editing the \/etc\/inittab file.<\/p>\n Connect to the ELOM by ssh into the IP address. Next we secure the new servers with some software updates and a firewall.<\/p>\n setenv PATH ${PATH}:\/opt\/sun\/cam\/bin<\/p>\n #\u00a0 Make command completion (TAB key) cycle through all possible choices #\u00a0 Internal Services #LAST LINE \u2014 ADD YOUR ENTRIES BEFORE THIS ONE \u2014 DO NOT REMOVE<\/p>\n<\/div>\n<\/li>\n<\/ul>\n<\/li>\n The next part will be connecting the servers to the storage array<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" I’ve spent a lot of time at work setting up a few servers to be our new production environment. Much of it was accomplished by reading the documentation over and over again. Not much out there on the Net, so I’m hoping this series of posts benefits someone else out there. First of all, I’ll … Continue reading Setting up a Hosting Environment: Part 1 – The servers<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1167,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[249,167],"tags":[179],"class_list":["post-1158","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-setting-up-a-hosting-environment","category-technical","tag-servers"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/mossiso.com\/wp-content\/uploads\/2012\/03\/DSCN7152-copy.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9wosP-iG","_links":{"self":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts\/1158"}],"collection":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/comments?post=1158"}],"version-history":[{"count":11,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts\/1158\/revisions"}],"predecessor-version":[{"id":1432,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/posts\/1158\/revisions\/1432"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/media\/1167"}],"wp:attachment":[{"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/media?parent=1158"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/categories?post=1158"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mossiso.com\/wp-json\/wp\/v2\/tags?post=1158"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/a><\/p>\n
Sun X2100 M2 OS Install steps<\/h2>\n
\n
\n
\n
\n
\n
\/boot<\/code>, the File System Type as \u2018ext4\u2019 and the Size (MB) as 500, then click \u2018OK\u2019<\/li>\n<\/ol>\n<\/li>\n
\n
Steps to set up the Remote Console (Embedded Lights Out Manager \u2013 ELOM) for SunFire X2100 M2<\/h2>\n
Set the SP serial port rate to 115200.<\/h3>\n
\n
\n
Set BIOS<\/h3>\n
IPMI Config\r\n Set LAN Config\r\n Set PEF Config\r\n PEF Support ........ [Enabled]\r\n PEF Action Global\r\n All of them ..... [Enabled]\r\n Alert Startup Discover ..... [Disabled]\r\n Startup Delay .............. [Disabled]\r\n Event Message For PEF ...... [Disabled]\r\n BMC Watch Dog Timer Action ... [Disabled]\r\n External Com Port ............ [BMC]\r\nRemote Access\r\n Remote Access ................ [Serial]\r\n Serial Port Number ........... [Com2]\r\n Serial Port Mode ............. [115200 8,n,1]\r\n Flow Control ................. [Hardware]\r\n Post-Boot Support ............ [Always]\r\n Terminal Type ................ [VT100]\r\n VT-UTF8 Combo Key ............ [Enabled]<\/pre>\n
\n
Edit Linux Config Files<\/h3>\n
Add a \/etc\/init\/serial-ttyS1.conf file<\/h4>\n
# This service maintains a getty on \/dev\/ttyS1.\r\nstop on runlevel [016]\r\n\r\nrespawn\r\ninstance $TTY\r\nexec \/sbin\/mingetty $TTY<\/pre>\n
Change grub.conf<\/h4>\n
# grub.conf generated by anaconda\r\n#\r\n# Note that you do not have to rerun grub after making changes to this file\r\n# NOTICE: You have a \/boot partition. This means that\r\n# all kernel and initrd paths are relative to \/boot\/, eg.\r\n# root (hd0,0)\r\n# kernel \/vmlinuz-version ro root=\/dev\/Logical\/root\r\n# initrd \/initrd-version.img\r\n#boot=\/dev\/sda\r\ndefault=0\r\ntimeout=5\r\n#splashimage=(hd0,0)\/grub\/splash.xpm.gz\r\n#hiddenmenu\r\nserial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\r\nterminal --timeout=10 serial console\r\n\r\ntitle CentOS Linux (2.6.32-71.29.1.el6.x86_64)\r\n root (hd0,0)\r\n kernel \/vmlinuz-2.6.32-71.el6.x86_64 ro root=\/dev\/mapper\/Local-root \\\r\nrd_LVM_LV=Local\/root rd_LVM_LV=Local\/swap rd_NO_LUKS rd_NO_MD rd_NO_DM \\\r\nconsole=tty1 console=ttyS1,115200n8\r\n initrd \/initramfs-2.6.32-71.29.1.el6.x86_64.img<\/pre>\n
Add line to securetty<\/h4>\n
console\r\nvc\/1\r\nvc\/2\r\nvc\/3\r\nvc\/4\r\nvc\/5\r\nvc\/6\r\nvc\/7\r\nvc\/8\r\nvc\/9\r\nvc\/10\r\nvc\/11\r\ntty1\r\ntty2\r\ntty3\r\ntty4\r\ntty5\r\ntty6\r\ntty7\r\ntty8\r\ntty9\r\ntty10\r\ntty11\r\nttyS1<\/pre>\n
\nSUN SP Commands<\/h3>\n
\nssh root@192.168.xxx.xxx<\/code><\/p>\n
\n
\n
set \/SP\/SystemInfo\/CtrlInfo PowerCtrl=on<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
set \/SP\/SystemInfo\/CtrlInfo PowerCtrl=gracefuloff<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
set \/SP\/SystemInfo\/CtrlInfo PowerCtrl=forceoff<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
set \/SP\/SystemInfo\/CtrlInfo PowerCtrl=reset<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
set \/SP\/SystemInfo\/CtrlInfo BootCtrl=BIOSSetup<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
set \/SP\/AgentInfo IpAddress=xxx.xxx.xxx.xxx <\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
root<\/code>, and the default password is
changeme<\/code>.\n
\n
set \/SP\/User\/[username] Password=[password]<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
start \/SP\/AgentInfo\/console<\/code><\/li>\n
Esc-Shift-9<\/code> keys.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\n
\n
stop \/SP\/AgentInfo\/console<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Software Updates and installs:<\/h2>\n
\n
\/etc\/resolve.conf<\/code><\/li>\n
\noptions single-request-reopen<\/p>\n\n
options single-request-reopen<\/code> takes care of slow SSH logins. See here https:\/\/stomp.colorado.edu\/blog\/blog\/2011\/06\/29\/on-rhel-6-ssh-dns-firewalls-and-slow-logins\/<\/a> and here http:\/\/www.linuxquestions.org\/questions\/showthread.php?p=4399340#post4399340<\/a> for more info.<\/li>\n<\/ul>\n<\/div>\n<\/li>\n
yum install openssh-clients tcsh ksh bc rpm-build gcc gcc-c++ redhat-rpm-config acl gcc gnupg make vim-enhanced man wget which mlocate bzip2-devel libxml2-devel screen sudo parted gd-devel pam_passwdqc.x86_64 rsync zip xorg-x11-server-utils gettext<\/code><\/li>\n
\/etc\/sysconfig\/selinux<\/code> file and set
SELINUX=disabled<\/code>.\n
\n
\/etc\/vimrc<\/code>file:\n
\nset smartindent ” same
\nset shiftwidth=4 ” number of space characters inserted for indentation
\nset expandtab ” inserts spaces instead of tabs
\nset tabstop=4 ” number of spaces the tab is.
\nset pastetoggle=<C-P> ” Ctrl-P toggles paste mode<\/div>\n<\/li>\ntcsh<\/code>\n
\n
\/etc\/passwd<\/code>file to have root use tcsh
root:x:0:0:root:\/root:\/bin\/tcsh<\/code><\/li>\n
.tcshrc<\/code>file in root\u2019s home.\n
\nalias cp ‘cp -i’
\nalias mv ‘mv -i’set prompt='[%n@%m %c]# ‘<\/p>\n
\n#\u00a0 (The default is to simply display a list of all choices when more than one
\n#\u00a0 match is available.)
\nbindkey “^I” complete-word-fwd<\/p>\n<\/div>\n<\/li>\n\/etc\/hosts<\/code>. Add a line with IP and domain name.\n
\n#\u00a0 that require network functionality will fail.
\n127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
\n::1 localhost localhost.localdomain localhost6 localhost6.localdomain6#\u00a0 External IPs
\n192.168.1.1 bill.com
\n192.168.1.2 ted.com192.168.1.3 domain.com # this needs to be an IP that the cluster server can manage#\u00a0 Internal IPs
\n192.168.1.11 bill.localdomain bill # notice the .localdomain, this is necessary for mysql later
\n192.168.1.12 ted.localdomain ted othernode # this is bill’s hosts file. othernode would be on the bill line for ted’s hosts file.
\n#\u00a0 ServicePort IPs
\n192.168.1.21 billsp # I like to have a short name to use to connect to the service port (ELOM)
\n192.168.1.22 tedsp<\/p>\n
\n192.168.1.100 http.localdomain httpd.localdomain
\n192.168.1.101 mysql.localdomain
\n192.168.1.102 memcached.localdomain<\/p>\n<\/div>\n<\/li>\nupdatedb<\/code> to set up the
locate<\/code> database.<\/li>\n
\n
\/etc\/pam.d\/system-auth<\/code>file. Change the line
password requisite pam_cracklib.so try_first_pass retry=3<\/code>to this
password requisite pam_passwdqc.so min=disabled,disabled,16,12,8<\/code><\/li>\n<\/ul>\n<\/li>\n
yum update<\/code>, and then a
yum install firefox xorg-x11-xauth xorg-x11-fonts-Type1<\/code>There will be more you\u2019ll need too.\n
\n
process 702: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open \"\/var\/lib\/dbus\/machine-id\": No such file or directory<\/code>. Then run the following command as root.\n
\n
dbus-uuidgen > \/var\/lib\/dbus\/machine-id<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n
\n
ssh-keygen<\/code><\/li>\n
cat id_rsa.pub >> ~\/.ssh\/authorized_keys<\/code><\/li>\n
authorized_keys<\/code> and
id_rsa<\/code> both set to
rw-------<\/code><\/li>\n
<\/pre>\n
Shorewall<\/h2>\n
\n
\n
rpm -Uvh http:\/\/download.fedoraproject.org\/pub\/epel\/6\/i386\/epel-release-6-5.noarch.rpm<\/code>on the machine.<\/li>\n
\/etc\/yum.repos.d\/epel.repo<\/code> file and set the first \u201cenabled\u201d line to equal 0. That disables yum from using the EPEL repo by default.<\/li>\n
yum --enablerepo=epel install shorewall<\/code><\/li>\n<\/ul>\n<\/li>\n
\/etc\/shorewall\/shorewall.conf<\/code> file. Change the
STARTUP_ENABLED=NO<\/code>to
STARTUP_ENABLED=Yes<\/code><\/li>\n
\/etc\/shorewall\/zones<\/code>file:\n
\n
\n#\u00a0 Shorewall version 4 \u2013 Zones File
\n#
\n#\u00a0 For information about this file, type \u201cman shorewall-zones\u201d
\n#
\n#\u00a0 The manpage is also online at
\n#\u00a0 http:\/\/www.shorewall.net\/manpages\/shorewall-zones.html
\n#
\n###############################################################################
\n#ZONE TYPE OPTIONS IN OUT
\n#\u00a0 OPTIONS OPTIONSnet ipv4 # The big bad Internet
\nloc ipv4 # Internal LAN
\nfw firewall#LAST LINE \u2013 ADD YOUR ENTRIES ABOVE THIS ONE \u2013 DO NOT REMOVE#LAST LINE \u2013 ADD YOUR ENTRIES ABOVE THIS ONE \u2013 DO NOT REMOVE<\/div>\n<\/li>\n<\/ul>\n<\/li>\n\/etc\/shorewall\/interfaces<\/code>file:\n
\n
\n#\u00a0 Shorewall version 4 \u2013 Interfaces File
\n#
\n#\u00a0 For information about entries in this file, type \u201cman shorewall-interfaces\u201d
\n#
\n#\u00a0 The manpage is also online at
\n#\u00a0 http:\/\/www.shorewall.net\/manpages\/shorewall-interfaces.html
\n#
\n###############################################################################
\n#ZONE INTERFACE BROADCAST OPTIONS
\nnet eth2
\nloc eth1<\/div>\n<\/li>\n<\/ul>\n<\/li>\n\/etc\/shorewall\/policy<\/code>file:\n
\n
\n#SOURCE DEST POLICY LOG LIMIT: CONNLIMIT:
\n#\u00a0 LEVEL BURST MASK
\n#\u00a0 To\/from internal lan
\nfw loc ACCEPT
\nloc fw ACCEPT
\n#\u00a0 To\/from net
\nfw net ACCEPT
\nnet all DROP info
\n#
\n#\u00a0 THE FOLLOWING POLICY MUST BE LAST
\n#
\nall all REJECT info
\n#LAST LINE \u2014 DO NOT REMOVE<\/div>\n<\/li>\n<\/ul>\n<\/li>\n\/etc\/shorewall\/rules<\/code>file:\n
\n
\n#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL
\n#\u00a0 PORT PORT<\/acronym> DEST
\n#SECTION ESTABLISHED
\n#SECTION RELATED
\nSECTION NEWSECTION NEW#\u00a0 Standard services
\n#
\nACCEPT \u00a0net \u00a0\u00a0\u00a0\u00a0 fw \u00a0\u00a0\u00a0\u00a0 tcp \u00a0\u00a0\u00a0 ssh
\nACCEPT \u00a0net \u00a0\u00a0\u00a0\u00a0 fw \u00a0\u00a0\u00a0\u00a0 tcp \u00a0\u00a0\u00a0 80,443Ping\/ACCEPT \u00a0\u00a0\u00a0\u00a0 net \u00a0\u00a0\u00a0\u00a0 fw<\/p>\n\/etc\/shorewall\/routestopped<\/code>file:\n
\n
\n#\u00a0 Shorewall version 4 \u2013 Routestopped File
\n#
\n#\u00a0 For information about entries in this file, type \u201cman shorewall-routestopped\u201d
\n#
\n#\u00a0 The manpage is also online at
\n#\u00a0 http:\/\/www.shorewall.net\/manpages\/shorewall-routestopped.html
\n#
\n#\u00a0 See http:\/\/shorewall.net\/starting_and_stopping_shorewall.htm for additional
\n#\u00a0 information.
\n#
\n###############################################################################
\n#INTERFACE HOST<\/acronym> OPTIONS PROTO DEST SOURCE
\n#\u00a0 PORT<\/acronym> PORT<\/acronym>
\neth1\u00a0\u00a0\u00a0\u00a0 –
\neth2 \u00a0\u00a0\u00a0\u00a0–<\/div>\n<\/li>\n<\/ul>\n<\/li>\nchkconfig shorewall on<\/code><\/li>\n
service shorewall start<\/code><\/li>\n<\/ul>\n