Changes between Version 15 and Version 16 of CloudServer


Ignore:
Timestamp:
Nov 16, 2015, 11:24:00 PM (8 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • CloudServer

    v15 v16  
    1 = Deploying a BOINC server on the Amazon Elastic Computing Cloud =
    2 
    3 === What is Cloud Computing? ===
    4 
    5 Cloud Computing is a form of computation and storage that
    6 hides the complexity of hardware and software management
    7 from a user.  Typically, cloud computing services allow you
    8 to rent computing or storage services.  These services
    9 charge on a pay-per-use basis (roughly, about 0.10 per CPU
    10 hour or per GB/month).  Cloud computing services such as
    11 Amazon's [http://aws.amazon.com/ec2/ Elastic Compute Cloud (EC2)] also allow one to
    12 tailor a computing environment through the use of virtual
    13 machines.  In addition, Amazon offers a [http://aws.amazon.com/s3/ Simple Storage Service (S3)]
    14 for enabling remote storage.
     1= Deploying a BOINC server in the Amazon Elastic Computing Cloud =
    152
    163=== Why use a Cloud? === #why
    174
    18 Hosting a BOINC server on a cloud is useful for the following reasons:
    19  * It's cheaper for small to medium projects to rent time on a cloud versus paying for hardware, bandwidth, and electric power on one's own.
    20  * It's easier and faster to use an existing OS image with the BOINC server already installed than to compile and configure it on one's own.
     5Hosting a BOINC server on a cloud has two advantages:
     6 * It's cheaper for small projects to rent time on a cloud versus paying for hardware, bandwidth, and electric power.
     7 * It's easier and faster to use an existing OS image with the BOINC server already installed than to compile and configure it.
    218
    229For an in-depth cost-benefit analysis, please read:
    23 [http://mescal.imag.fr/membres/derrick.kondo/pubs/kondo_hcw09.pdf Cost-Benefit Analysis of Cloud Computing versus Desktop Grids].  Derrick Kondo, Bahman Javadi, Paul Malecot, Franck Cappello and David Anderson.  To appear in 18th International Heterogeneity in Computing Workshop, May 25 2009, Rome.
     10[http://mescal.imag.fr/membres/derrick.kondo/pubs/kondo_hcw09.pdf Cost-Benefit Analysis of Cloud Computing versus Desktop Grids].  Derrick Kondo, Bahman Javadi, Paul Malecot, Franck Cappello and David Anderson.  8th International Heterogeneity in Computing Workshop, May 25 2009, Rome.
    2411
    25 To calculate the cost-benefits for your own application, you can use this
    26 [http://mescal.imag.fr/membres/derrick.kondo/cloud_calc.xls Excel spreadsheet].
    2712
    28 === How to use a Cloud for a BOINC server? ===
     13You can set up a free BOINC server in a few minutes using a preconfigured server for Amazon's Elastic Compute Cloud.
    2914
    30 We have created a custom virtual machine with a BOINC server
    31 installed and configured.  This includes all server software
    32 prerequisites and the correct permissions and accounts
    33 already setup.  All you need to do is to deploy this virtual
    34 machine image on the cloud and your server should be up and
    35 running.  This guide describes how to deploy this Amazon
    36 Machine Image (AMI) over Amazon's EC2 cloud. This can be
    37 done in minutes.  Testing this image deployement will cost you less then 0.50 USD so give it a try.
    38 
    39 === Prerequisites ===
    40 
    41 [http://paulstamatiou.com/2008/04/05/how-to-getting-started-with-amazon-ec2 Create and setup an account with Amazon's EC2.]
    4215
    4316=== Deploying the BOINC Server Image ===
    4417
    45 Start the instance (ami-2290774b) with the BOINC server
    46 (stable 6.3.14 JAN 28 2009) installed on Debian Etch:
     18 
     19
     20Navigate to to aws.amazon.com and create an account if needed (requires a credit card, but it will not be charged for 12 months). Click “EC2: Virtual Servers in the Cloud” and “Launch Instance”. Click “Community AMIs” (left sidebar), search for “BOINCServerTemplate”, and click on the blue Launch Button. Choose type “t2.micro: Free tier eligible” and “Review and Launch”. Click “Edit security groups”, “Add Rule” and select type “HTTP”. Click “Review and Launch” and then “Launch”. If you do not have a key pair, choose “Create a new key pair” and give it a name like BOINCKeypair, click on “Download Key Pair”, and save BOINCKeypair.pem in a safe place. Click on “Launch instance” and  “View Instances”. Wait until your instance changes from Instance State “Pending” to “running” with a green circle.
     21
     22Paste the Public IP nnn.nnn.nnn.nnn (once it appears) into a web browser, and it should say “Apache2 Ubuntu Default Page”. You can assign a static IP address by choosing Elastic IP in the left side bar, allocate IP, and associate the IP with your instance. You can then buy a domain name and point to that static IP.
     23
     24 
     25
     26Then, connect to Public IP nnn.nnn.nnn.nnn by ssh. For instance, download PuTTY, use PuTTYgen to import the .pem file and save a .ppk file, and choose public ip address nnn.nnn.nnn.nnn, port 22, connection type SSH, choose Connection, SSH, Auth, and browse to find the .ppk file, click Open, click Yes at the “PuTTY Security Alert”. At the "login:" prompt, enter "ubuntu".
     27
     28 
    4729
    4830{{{
    49 ec2-run-instances ami-2290774b -k [key-pair]
     31
     32$su boincadm
     33
     34password: boincadm
     35
     36$cd /home/boincadm/boinc/tools
     37
     38$hostname
     39
     40ip-nnn-nnn-nnn-nnn  (note: this is the internal IP address, with dashes, differennt from the public IP)
     41
     42$./make_project --url_base http://nnn.nnn.nnn.nnn/ --project_host ip-nnn-nnn-nnn-nnn --test_app uppercase
     43
     44...
     45
     46[Y/n] (hit enter)
     47
     48$cd /home/boincadm/projects/uppercase/
     49
     50$sudo -- sh -c 'cat /home/boincadm/projects/uppercase/uppercase.httpd.conf >> /etc/apache2/httpd.conf'
     51
     52 
     53
    5054}}}
    5155
    52 Show the instance deployment status:
     56 
     57
     58Check in a web browser that the page "http://nnn.nnn.nnn.nnn/uppercase" works and has the title "REPLACE WITH PROJECT NAME"
    5359
    5460{{{
    55 ec2-describe-instances
    56 }}}
    5761
    58 After the instance status goes from pending to running
    59 (takes a few minutes), you
    60 should see the hostname of the form
    61 ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com.
     62$bin/xadd
    6263
    63 If you haven't already, open ports 22 and 80:
     64$bin/update_versions
     65
     66}}}
     67
     68Choose "y" repeatedly
    6469
    6570{{{
    66 ec2-authorize default -p 22
    67 ec2-authorize default -p 80
     71
     72$bin/start
     73
    6874}}}
    6975
     76In a web browser, using the Public IP, navigate to "http://nnn.nnn.nnn.nnn/uppercase/server_status.php". Status should be "Running" green now for all Programs on the left.
    7077
    71 At this point, you should be able to access the web server
    72 at the URL:
    73 {{{http://ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com/}}}
     78 
    7479
    75 Then login to the instance:
     80Using a BOINC client, add the project "nnn.nnn.nnn.nnn/uppercase" and the client should start downloading and doing work. Congratulations you have a working BOINC project!
     81
     82 
     83
     84To protect your administrator page:
    7685
    7786{{{
    78 ssh -i [private_key corresponding to key-pair]  root@ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com
     87
     88$cd /home/boincadm/projects/uppercase/html/ops
     89
     90$htpasswd -c .htpasswd boincadm
     91
     92password: boincadm (choose your own password)
     93
    7994}}}
    8095
    81 As root,
     96In the web browser, navigate to your project administrator web interface at "http://nnn.nnn.nnn.nnn/uppercase_ops". Click on "Summary" to see work already done.
     97
     98 
     99
     100=== Creating Your Own Application ===
     101
     102 
     103
     104To create your application, you need a work generator, a client program in Windows/unix, and an assimilator to combine the work that comes back. To modify the work generator and sample assimilator:
    82105
    83106{{{
    84 su boincadm
    85 cd
     107
     108$cd /home/boincadm/boinc/sched/
     109
     110$emacs sample_work_generator.cpp
     111
     112$emacs sample_assimilator.cpp
     113
     114$make
     115
     116$cp sample_work_generator /home/boinc/projects/uppercase/bin
     117
     118$cp sample_assimilator /home/boinc/projects/uppercase/bin
     119
     120$cd /home/boincadm/projects/uppercase
     121
     122$bin/stop
     123
     124$bin/start
     125
    86126}}}
    87127
    88 In the home directory is a README.txt file with important
    89 information.
     128 
    90129
    91 === Setting Up a Test Project ===
    92 
    93 Here we describe how to create a test project using the
    94 uppercase application.
    95 
    96 Create a test project uppercase:
     130To modify the Unix client program in Unix,
    97131
    98132{{{
    99 cd ~/boinc/tools
    100 ./make_project --url_base http://ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com --db_passwd boincadmpw --test_app cplan
     133
     134$emacs /home/boincadm/boinc/samples/example_app/uc2.cpp
     135
     136$make
     137
     138$cd /home/boincadm/apps/example_app
     139
     140$cp -r 22489 30000 (to create the directories for the new versions)
     141
     142$cd 30000/x86_64-pc-linux-gnu
     143
     144$cp /home/boinc/boinc/samples/example_app/uc2 /home/boinc/projects/apps/example_app/30000/x86_64-pc-linux-gnu/example_app_30000_x86_64-pc-linux-gnu
     145
    101146}}}
    102147
    103 Answer yes to all questions.
     148and delete the other files in that directory. To create updated client program for Windows, download Microsoft C++ Express Version 10, download the boinc source, Build All, in project "example_app" of the "boinc" solution, edit "uc2.cpp", Build All, run the compiled program win_build\Build\Win32\Release with input file "in", and review the output file "out" and the "stderr.txt". When the program is ready, use FileZilla to transfer the file example_app.exe to http://nnn.nnn.nnn.nnn/, SFTP, username "ubuntu", select the private .ppk key under Settings, SFTP, and click on the example_app.exe file in the left hand tree. Connect to your server, and look for the file in "/home/ubuntu". Then
    104149
    105150{{{
    106 cd ~/projects/cplan
    107 ./bin/xadd
    108 ./bin/update_versions
    109 ./bin/start
     151
     152$cp /home/ubuntu/example_app.exe /home/boincadm/projects/uppercase/apps/example_app/30000/windows_intelx86/example_app_30000_windows_intelx86.exe
     153
    110154}}}
    111155
    112 To check that all daemons are running:
     156and delete the older versions in that directory". Delete other architectures for which you do not have clients in the 30000 directory. Navigate in the browser to "http://nnn.nnn.nnn.nnn/uppercase_ops/", choose Manage application versions" and deprecated the old versions.
    113157
    114 {{{
    115 ./bin/status
    116 }}}
     158 
    117159
    118 Now any [http://boinc.berkeley.edu/download.php BOINC client] should be able to attach to the project and download
    119 workunits. For example, go to the project page (http://ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com/cplan) and create a user account.
     160To fix the appearance of the page "http://nnn.nnn.nnn.nnn/uppercase", edit /home/boincadm/projects/uppercase/html/project.in especially to enter the project name in place of "REPLACE WITH PROJECT NAME".the "COPYRIGHT_HOLDER", the "SYS_ADMIN_EMAIL", and the smtp settings. To limit SPAM users, add akismet and recaptcha keys to config.xml.
    120161
    121 Then with a Linux command-line client in ~boincadm/BOINC, run
    122 {{{
    123 cd ~/BOINC
    124 ./boinc_client  -no_gui_rpc -attach_project http://ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com/cplan [account_key]
    125 }}}
     162 
    126163
    127 At this point, the client should attached to the project and begin downloading work.
     164=== Notes ===
    128165
    129 === Technical Notes ===
     166 
    130167
    131 Below is not required reading, but we describe how and why
    132 the image was built.
     168Change the following default passwords: for the Unix user boincadm password boincadm, and for the mysql root user password boincadm. The mysql user boincadm has not password; add one and update config.xml. Edit /etc/apache2/apache2.config and /etc/apache2/httpd.config to use the "Require" directory in both places.
    133169
    134 ==== BOINC Configuration ====
    135 
    136 You'll find in the ~boincadm directory :
    137 
    138  * BOINC: directory with client for testing purposes
    139  * projects:  directory where BOINC projects are installed
    140  * boinc_server_stable_r16906_6.3.14: directory with server source code
    141 
    142 
    143 
    144 ==== Why Xen? ====
    145 
    146 
    147 The general approach was to build a Debian EC2 AMI from a
    148 Xen image.  (The alternative approach was to use an existing
    149 Debian AMI.)  However, we chose the Xen approach primarily
    150 to be able to install and test things without having to
    151 worry about EC2 hourly costs. As a side-effect, one can use
    152 the Xen image either as a standalone server or over EC2 (as
    153 it's Xen based).
    154 
    155 These links describe the setup involved:
    156 [http://jimmyg.org/2007/09/01/custom-debian-ec2-amis-from-xen-images/ Debian from Xen Images],
    157 [http://www.linuxconfig.org/Howto_CREATE_BUNDLE_UPLOAD_and_ACCESS_custom_Debian_AMI_using_ubuntu Bundle Debian AMI using Ubuntu]
    158 
    159 
    160 If you would like to use the Xen image directly, you can
    161 download the Xen image
    162 [http://mescal.imag.fr/membres/derrick.kondo/ec2/disk.img.tgz disk.img.tgz (1.5GB)] and
    163 [http://mescal.imag.fr/membres/derrick.kondo/ec2/swap.img.tgz swap.img.tgz (.5MB)],
    164 and config file
    165 [http://mescal.imag.fr/membres/derrick.kondo/ec2/boinc_debian_etch.cfg boinc_debian_etch.cfg]
    166  and go from there.
    167 
    168 The Xen image above was configured to use a bridged Ethernet connection and DHCP.
    169 
    170 ==== Hostname Configuration ====
    171 
    172 The hostname can change each time an instance is started.
    173 As such, the start-up script {{{/etc/rc.local}}} was modified to query
    174 Amazon for the hostname and to set it at each boot.
    175 
    176 
    177 ==== IP Address ====
    178 
    179 The IP address could change each time the instance is
    180 created.  To associate a fixed address (regardless of the
    181 instance), one should use
    182 [http://aws.amazon.com/ec2/  Amazon's Elastic IP Addresses].
    183 
    184 In addition, one can configure the image to use dynamic DNS
    185 to associate a domain name with the instance. See the following links.
    186 [http://www.ioncannon.net/system-administration/120/dynamic-dns-with-ec2-and-zoneedit/ Dynamic DNS with ZoneEdit]
    187 [http://blog.spaceprogram.com/2008/03/how-to-set-up-dynamic-dns-for-your.html Dynamic DNS Setup]
    188 
    189 ==== SSH ====
    190 
    191 SSH server host keys are (re)generated each time the instance is booted in {{{/etc/rc.local}}}.  This is to help prevent man-in-the-middle-attacks.
    192 
    193 {{{rc.local}}} is configured to randomly change the root
    194 password at each boot, and to download your ssh public key
    195 and cat it to the authorized_keys file to allow remote passwordless access.
    196 
    197 To verify the identity of the host before logging in, you can run {{{ec2-get-console-output}}} to get the key fingerprint written to the console during the bootup process.  The key fingerprints will look something like:
    198 {{{
    199 Restarting OpenBSD Secure Shell server: sshd.
    200 ------- KEY FINGERPRINTS -------
    201 2048 28:f1:0c:51:95:82:c6:a1:a2:58:a3:37:7f:39:ef:f9 /etc/ssh/ssh_host_rsa_key.pub
    202 1024 90:f2:98:4b:e6:c5:c7:7a:23:6e:ef:67:d0:be:72:a9 /etc/ssh/ssh_host_dsa_key.pub
    203 --------------------------------
    204 }}}
    205 during bootup.
    206 Then you can compare this key fingerprint with the one reported by ssh when you log in to verify the host identity.
    207 
    208 === To do ===
    209 
    210  * Consider setting up dynamic DNS by default in the bundle
     170 
    211171
    212172=== Questions? ===
    213173
    214 Email [http://mescal.imag.fr/membres/derrick.kondo Derrick Kondo] (derrick.kondo :: inria fr)
     174 
     175
     176Email: daniel-monroe :: verizon net.
     177