Development workstation in the Cloud

I have a dream that one day I will not need to have a dedicated machine to do software development. That I will be free from the shackles of physical hardware and do all my development in the cloud.

I work as a consultant, so I change places quite often. It should not matter whether I’m at work, at home or at a client site. When virtualization first came along, I was pleased. My prayers were heard. In practice, the virtual machines required serious physical hardware to run at any decent speed and that still chained me to a big machine.

Then came the cloud. Hmm, not so much different from running my own machine… on the surface. In practice, it gives me a lot of freedom. I just rent my powerful machine instead of buying it.

I work with java, grails and .NET and for my purposes I use Windows 7 64bit most of the time. Sure, Linux is great too, but not for .NET work. Many of the cloud hosts do not support running windows so those are out of the question. I need a very responsive system, so I can’t feel the difference between running a remote desktop on my LAN or on the cloud.

That sets a very high requirement for the 4 big factors of cloud performance:

                  Network latency, Disk IOPS, RAM and CPU

Network latency is the most important. I need the host to register my mouse movement and my typing quickly. Cable quality, router speed and distance all matter. For my purposes I need to run it really close to home. I live in Copenhagen, Denmark and the network latency to Rackspace and Amazon is too great. Even their european servers are too far away in ping-time. I’ve tried host with 50ms ping time or less, and it’s still too much. I tried a great cloudhost called Cloudsigma in Zürich, but even though their performance was great, their latency was not good enough for me.

Up until now there was nothing to do. Host a server at home maybe…? Nah, I don’t want to run a powerful server at home. Not because I can’t, but because it’ll be a dinosaur in two years anyway. And what if i need 10 machines for a projekt? Or another 32 GB of ram? Nope, not for me. Cloud it is.

Cloud.dk is a new danish provider of cloud computers. Their datacenter is around 3 ms away from me, and their disk/ram/cpu are pretty good too. I’ll show some benchmarks later.
Here’s a little how-to of how to get a Windows 7 running on their infrastructure. I’ll assume that you have an account and know how to fire up a virtual machine from one of their existing images.

Step 1
Download the ISO image of your operating system. I used my MSDN account to download Windows 7 Enterprise with SP1 in the 64 bit version

Step 2
Make the ISO available via HTTP. In my case i uploaded it to an S3 bucket and made it public.

Step 3
Go into the cloud.dk control panel and go to “images” and then “fetch image” http://i.imgur.com/NkhRk.jpg

Step 4
Instruct cloud.dk to fecth the image from your httpserver. Take a look at the following image http://i.imgur.com/Y8LGG.jpg. Be sure to change the URL and be sure to add a default user.
That will start a job that fetches the image. Be sure to have a fast HTTP server, as the job will time out if it runs too long. You can go to the jobs tab to see the status. http://i.imgur.com/YHytl.jpg

Step 5
When the download is complete, you’ll have a new image in the list: http://i.imgur.com/Z8BsX.jpg. If you don’t see this, your download has failed for some reason.

Step 6
Make a new machine. Use the new image as the primary disk and add another 50GB or 100 GB disk if you need the extra space. I recommend at least 4 GB/ 2 vCPU if you plan on running Visual Studio 2010. http://i.imgur.com/jRpR0.jpg
Switch the primary and secondary disks: http://i.imgur.com/ht0gQ.jpg. Think of this as putting the harddisk as the primary and the uploaded iso image as the DVD-ROM. IF YOU FORGET THIS STEP, THE INSTALL WILL FAIL as it will be unable to partition correctly.
Start the server: http://i.imgur.com/6A921.jpg. Let the machine run for a couple of minutes and then start the console: http://i.imgur.com/oziFm.jpg
If you have java installed, it’ll start a program (webstart) that acts as a vnc console. You should see the windows install screen like this: http://i.imgur.com/p9Lrp.jpg

Step 7
Install windows as you normally would.

Step 8
Enable Remote Desktop in windows 7
Close the console
Login with remote desktop
Your Win 7 should be ready for you to setup as you like.
My install (from boot to remote desktop login) took around 12 minutes! That’s fast! And that time even included waiting for the human (me) to react

Step 9
Shutdown the machine. If you logged on with remote desktop you can shutdown with this command:
shutdown -s -f -t 3
After that, you wait for the VM to be completely off. When it’s off, remove the ISO image from the server: http://i.imgur.com/8W6KM.jpg
After that, delete the disk image. http://i.imgur.com/cn77v.jpg. It’s a copy of the uploaded image anyway, so no need to waste cloud storage on it. You can still make new machines with the uploaded image.

Step 10
That’s it. No more required steps. I do have a few suggestions for activities though:
- Get all windows updates
- Set up the cloud.dk firewall. RDP uses port 3389
- Enable power management and set to hibernate after 1 hour without use. Execute the command “powercfg -H ON” in an elevated prompt to enable hibernation feature of of Win7 if it’s not enabled already.
- Figure out how to start and stop your machine from the commandline API
- Disable the pesky UAC feature.

Oh, I almost forgot I promised some benchmarks. Remember this is the 4 GB / 2 vCPU machine configuration.

• Geekbench: 4580 – A decent score, as expected.
• Crystal Disk Mark 4K QD32: 43.77 MB/s read and 11.84 MB/s write. An impressive score for a cloud machine. They’re using a TSM system, where their SAN has a mix og ssd’s, harddrives, large ram caches and migrates automatically. Seems to work out nicely for them
• Speedtest.net: Ping 5ms, download 186.64 Mbps, uploda 180.86 Mbps
• Pingtest.net: Packet loss 0%, PIng 4 ms, jitter 1 ms.
• Details here: http://i.imgur.com/qb9Jr.jpg

So: in summary, a decent RAM/CPU score, a good network score and a disk score that’s similar to a low end SSD. That’s just numbers, and good ones too, but…
What’s my “feel” for it? Well, it feels very snappy and fast. Complete windows install in 12 minutes. Windows updates download and install quickly. Any sort of manipulation of the user interface feels almost as if it was a local machine. Everything loads quickly and programs start up fast. It feels almost like it’s a SSD

Enjoy your new workstation in the cloud!

Java JDK silent install on windows

I hate next-next-next installers. Up until now, I had to endure the horrors of that, while installing the Java JDK on windows.
Thank to my colleague Erik, I don’t have to do that anymore. With his help, I discovered how to do a silent command line install of the entire JDK:

UPDATE: this article turned obsolete. Instead I’ll provide the link to the “Installation notes” for the windows JDK 7:

http://www.oracle.com/technetwork/java/javase/documentation/install-windows-152927.html

I had succes with this command line:

 

jdk-7u2-windows-i586.exe /s ADDLOCAL="ToolsFeature,SourceFeature" INSTALLDIR="%CD%\jdk7u2"

 

Compiling and installing Apache webserver from source on redhat linux

Just a quick post about something i had to to do today. It’s all in the Apache docs, but bits of it was hard to find. I was working on a redhat 4 64-bit

• Grab the source from http://httpd.apache.org/download.cgi

  • I picked the Unix Source: httpd-2.2.17.tar.gz

• Unpack with tar xzf httpd-2.2.17.tar.gz

• Cd httpd-2.2.17

• ./configure --prefix /path/to/apache --enable-mods-shared=all --enable-proxy --enable-ssl

  • If you leave out "--prefix ..." then the installation directory is set to /usr/local/apache2

• make

  • Does all the compiling

• make install

  • Installs into the directory that you wrote in --prefix

• Edit the /conf/httpd.conf if you want to set a another port or change some other setting

Controlling the Apache server:

• /bin/apachectl -k start
• /bin/apachectl -k stop
• /bin/apachectl -k restart
• /bin/apachectl -k graceful-stop

Congratulations are in order :)

I am now a ScrumMaster with certificates and everything.

We did a two day ScrumMaster certification training course with our entire company (around 40 or 50 people). It was a very good course and our teachers were both knowledgeable, experienced and very inspiring. I can recommend them if you’in the market for some training.

Our trainers were by Gertrud Bjørnvig and Jim Coplien and their website is at: https://sites.google.com/a/gertrudandcope.com/www/

And here’s my proof and pretty certificate

All in all a very good experience, and I feel much more enthusiastic about Scrum than I did before.

 

 

 

Deploying from grails to tomcat

I’ve set up my grails build on Hudson, and now I want to deploy to my Tomcat server. Every time my tests complete, I want a new deploy. And I want to be completely automatic too.

First step is to navigate to your grails app and try:

grails install-plugin tomcat

It’s probably already installed, as it’s the default for grails these days.

Then we need to install Tomcat if you don’t have it already.

First, make sure it’s the Sun JDK, argh, I mean Oracle, but the Ubuntu packages still say Sun. The OpenJDK doesn’t really work well with grails and tomcat.

sudo aptitude install sun-java6-jdk

Then install tomcat:

sudo aptitude install tomcat6

Then edit the tomcat user file to enable some users:

sudo nano /etc/tomcat6/tomcat-users.xml

I suggest you have the following lines active, but please change the passwords:

<role rolename=”manager”/>

<user username=”manager” password=”manager” roles=”standard,manager”/>

<user username=”grails” password=”grails” roles=”standard,manager”/>

Restart tomcat to activate the changes.

sudo /etc/init.d/tomcat6 restart

Test by going to:

http://<hostname>:8080/manager

You should now specify the username/password and url for the to) in your grails project. Open the  grails-app/conf/Config.groovy file and add this at the bottom of the file:

tomcat.deploy.username=”grails”

tomcat.deploy.password=”grails”

tomcat.deploy.url=”http://<hostname>:8080/manager”

That should set up easy deployment directly from grails. We’ll simply be using grails’ own plugin for deployment, so we’ll have no real need to understand how ant or tomcat handles it. It’s all nicely camouflaged by grails.

The deployment sequence is like this: First you need to undeploy the app from tomcat (not needed the first time)

grails tomcat undeploy

And then deploy the app

grails tomcat deploy

You should now be able to use your app on your testserver.

To add them to your Hudson project just add those two lines as a post-build shell script (or batch command if you’re on windows) and you’re good to go.

If undeployment fails, there may be some file locks. I experienced that on windows, but not on the Ubuntu 10.10 server. If (and only if) you have this problem, you can edit the

sudo nano /etc/tomcat6/context.xml

And change the <context> line to:

<Context antiJARLocking=”true” antiResourceLocking=”true” >

That fixed the issue for me.

PS: To control tomcat use these commands:

sudo /etc/init.d/tomcat6 stop

sudo /etc/init.d/tomcat6 start

sudo /etc/init.d/tomcat6 restart

Building a grails project in hudson

I’ve learned how to set up setup grails and how to install hudson on Ubuntu 10.10 Server. Now I want to use hudson to build my grails project.

Before we set up the project, we need to install some plugins. Access your hudson admin interface, probably at http://<hostname>:9090

Go to Hudson -> Manage Hudson -> Manage plugins -> Available (it’s a tab) and get these required plugins:

Hudson Grails plugin

Hudson Mercurial plugin or get the one for git/svn/cvs depending on what you use. I’m a mercurial fan.

If you like, you can also install some of these, but they’re not required:

Radiator View Plugin

Green Balls

ChuckNorris Plugin

Monitoring

Build-timeout Plugin

Hudson Tray Application

 

After that you need start a new project by going to the link called Hudson -> New project

Choose a job name, and the Build a free-style software project. After you click OK, you get a lot of options. Only some of them are needed:

Discard Old Builds -> Days to keep: 3, Max # of builds to keep: 10

In the Source Code Management part you select Mercurial and then enter the Repository URL. For a private bitbucket it’s like this:

https://<username>:<password>@bitbucket.org/<username>/<project>

Make sure to include the username:password if it’s a private project. For a public one, it’s not needed.

In the Build Triggers section, choose Poll SCM and enter this into the schedule field (it’s std cron notation):

# every minute

* * * * *

In the Build section, choose Add Build Step and Build With Grails. You’ll se a lot of grails related fields but you only need to enter the following into the Targets field:

test-app

Eventually, you’ll probably want to run some other targets, but this one is a good starter.

In the Post-build Actions section, select the Publish JUnit test result report. Enter the following into the Test report XMLs field:

target/test-reports/*.xml

You’re almost done, just click Save in the bottom.

Now run the project and check the outcome. After two runs, you should start to see a unittest graph.

Make a change to your code, push to the repo and see if hudson picks it up. It should poll once every minute and pick up the code change and start a build automatically

Install the Hudson build server on Ubuntu 10.10 Server

First off, after trying out most of the free build servers on the market, I’ve found that Hudson ( http://www.hudson-ci.org/ ) is the best for me. It’s by far the easiest to configure and it runs pretty smoothly. The plugin ecosystem is vibrant and there are constantly new and interesting plugins. I’ve used it for java, grails and even msbuild projects with success.

I’m learning about headless linux servers and hosting on them, so I’m using this an opportunity to document the sometimes tricky install procedures. I’m using Ubuntu 10.10 Server in a 32-bit edition. I must admit that I’m impressed by the performance of the headless linux server. It’s running in VMWare Player 3 with 512mb ram, on a USB drive, on my underpowered Lenovo T500 laptop… and it feels snappy!

I’ll assume that you’ve already done the steps for installing grails. If not, you at least need to do a succesful:

 sudo aptitude install sun-java6-jdk

I got some of this installation info from http://hudson-ci.org/debian/. First we need to install a prerequisite:

 sudo aptitude install daemon

Then we need to fetch a key (and install it) and a deb package (and install it). After that we update the apt repo list and do a “normal” install

 wget -O /tmp/key http://hudson-ci.org/debian/hudson-ci.org.key

 sudo apt-key add /tmp/key

 wget -O /tmp/hudson.deb http://hudson-ci.org/latest/debian/hudson.deb

 sudo dpkg --install /tmp/hudson.deb

 sudo apt-get update

 sudo apt-get install hudson

Once the package is installed, normal updating should work. Check that the install is okay by visiting:

 http://<hostname>:8080

Running hudson on port 8080 is not cool, since so many other servers want to run there (Tomcat for instance). So let’s change the port number:

First, stop the hudson server:

 sudo /etc/init.d/hudson stop

To change change port number for hudson to 9090, run this line:

sudo perl -pi -e 's/HTTP_PORT=(.*)\n/HTTP_PORT=9090\n/g' /etc/default/hudson

It does a search and replace in the /etc/default/hudson and replaces the port number. Call this to verify the new port number:

cat /etc/default/hudson |grep HTTP_PORT=

Then you can start Hudson again:

 sudo /etc/init.d/hudson start

In case you need to debug something or if you’re a log geek you can find the hudson log is in /var/log/hudson/hudson.log.  You may want to run

 tail -f /var/log/hudson/hudson.log

to keep viewing it

PS: Thanks to Theis Borg for helping me with the linux scripting!

Continuous testing revisited: http://rapid-testing.com/

I have a little update about continuous testing for .Net. I’ve used the ones I mentioned earlier, but they all have the problem that they run all tests on each compile. That’s simply to slow for me. I’d rather run the test suite manually, when it suits my coding rhythm.

Infinitest for the java platform (eclipse and intellij IDE’s) is great and does close-to-instant testing, even on fairly large projects. However that type of instant testing has not been available for Visual Studio yet… until now:

http://rapid-testing.com/

They are in the alpha stages, but keep an eye on them. They show great promise, and I hope they can deliver the instant action I desire. Check their blog here for updates:

http://blog.rapid-testing.com/

Ubuntu 10.10 Server: From Clean install to Grails for all users

I’m playing around with Grails at the moment. Grails is a make-websites-really-fast framework, that is based on the Groovy language. It’s inspired in spirit by Ruby-on-Rails, but it’s based on solid java technology instead.

In my opinion, this the best framework for java developers to switch to. Learning curve for a java developer is a lot easier than ruby/python/scala/clojure etc. and the groovy language is very powerful and easy. Ont eh other hand… it doesn’t teach you how to think differently, and you may be tempted to keep your bad java habits. Anyway, I’m getting derailed. What I really wanted to show you today was how to get grails working on a linux server.

I’ll assume that this is a totally clean install of Ubuntu 10.10 server, I used the 32-bit version, but I think the 64-bit installation will be identical. The first thing we do is to install ssh, so we can log in with putty ( http://www.putty.org/ ) afterwards. Then we upgrade the system, then we reboot.

sudo aptitude install ssh

sudo aptitude update

sudo aptitude upgrade

sudo reboot

Login with putty when the reboot is done. Then install the C++ compiler and the linux headers. It may not be needed, but I tend to do it anyway, as they cover a lot of weird dependencies when installing various development tools. We also install sun’s java (as the openjdk fails to run grails) and groovy (yay, it’s a std package).

sudo aptitude install build-essential linux-headers-$(uname -r)

In order to install sun’s jdk we need to enable some more repositories. Start by running

sudo nano /etc/apt/sources.list

Then uncomment the lines for the partner repos and save the file. Partner repos look like this:

## Uncomment the following two lines to add software from Canonical's

## 'partner' repository.

## This software is not part of Ubuntu, but is offered by Canonical and the

## respective vendors as a service to Ubuntu users.

deb http://archive.canonical.com/ubuntu maverick partner

deb-src http://archive.canonical.com/ubuntu maverick partner

Update apt with:

sudo aptitude update

Once that is done we can install the sun jdk and groovy

sudo aptitude install sun-java6-jdk

sudo aptitude install groovy unzip

After that we fetch the grails binary, unpack and move it.

wget http://dist.codehaus.org/grails/grails-1.3.5.zip

unzip grails-*.zip

mv grails-1.3.5 grails

sudo rm -r /usr/lib/grails

sudo mv grails /usr/lib/grails

Then we need to edit the environment file, so we can access the JAVA_HOME and GRAILS_HOME. We also add the Grails bin directory to the path. Run these commands:

sudo cp /etc/environment /etc/environment_backup

sudo sed -i 's/\(^PATH=".*\)"/\1:\/usr\/lib\/grails\/bin"\nJAVA_HOME=\/usr\/lib\/jvm\/java-6-sun\nGRAILS_HOME=\/usr\/lib\/grails\n/g' /etc/environment

After the commands, you can verify ( cat /etc/environment ) that the  environment file should look a lot like this:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/lib/grails/bin"

JAVA_HOME=/usr/lib/jvm/java-6-sun

GRAILS_HOME=/usr/lib/grails

Once that’s done we need to reboot the system, in order to read the new environment:

sudo reboot

Log in again and test by calling:

grails help

It should show you the version of grails and the help info. To test it for another user, create a user ( sudo adduser <username>) , log in as that user and run “grails help”. If it does that correctly, then this guide worked

Continuous Testing with IDE integration

What is Continuous Testing?

Continuous Testing (CT) is a way to get almost instant feedback while doing TDD (Test Driven Development). However, it’s not like a CI (Continuous Integration) server that runs your build and unit tests on commit of new code. Ben Rady, the creator of Inifinitest, tries to explain it here as well: http://blog.objectmentor.com/articles/2007/09/20/continuous-testing-explained

Continuous Testing runs your tests as soon as the code has been built, on your own developer machine. The tests should be run automatically, so you don’t need to remember to run your tests. The CT tool simply monitors your compiler artifacts and fires up the necessary tests. If there are test failures, then the developer gets alerted. The really good CT tools only run the tests for the classes that have changed, thus making the tests extremely fast.

Personally, I love to use TDD. I’ve almost forgotten how to do it the old fashioned way. But if I didn’t have a CT tool, then I wouldn’t run the tests very often. The clicks/keys/time necessary for to start the test-runner, wait for the results, view the results and react to them are simply too long. I like to work in very very short cycles and get instant feedback, without taking any special action to fire up my tests. By working this way I can “work in the zone” / “achieve flow”, the state of mind that makes me the most focused and productive. Without a CT tool, I probably wouldn’t run TDD at all. I’d still use unit tests, but probably in a more traditional way.

In some CT tools, test failures appear as if they are compiler errors. Others show a red/green light, while some just show a command line with coloured text. All are fine, as long as they give feedback without any action from me. I have two favourite CT tools, one for the Java platform and one for the .NET platform:

Infinitest

In a word: Awesome!
I cannot express how cool I find this tool. If I had to bring a single plugin with me to an isolated island, this would be it.

Eclipse automatically compiles on save of a file, since the java/eclipse
compiler works fine on single file.  Intellij does not compile by
default, but there is an “Eclipse mode” plugin that will do “build on
save” for intellij as well – a must if using infinitest in my opinion.

It works great in Eclipse and Intellij, and simply shows test failures as compiler warnings. Infinitest reacts to newly compiled test classes and normal
classes in your classpath. Tests are lightning fast, since Infinitest is very clever and only runs the changed tests, not the whole test suite everytime. Usually response time from “save file” to “test complete” is less than 100 ms. It feels like it is instant. It’s what makes TDD worth doing.

It works great on tests and classes written in Groovy and Scala too!
It even has a filter file, so you can exclude certain file types or names. For example, you can exclude files containing “integration” in their name, and thus use JUnit to run your long running integration tests.

License: Free license available for personal use, $29 for prof, $149 for a transferable corporate license -> http://improvingworks.com/products/infinitest/pricing/
Source: Closed

Continuous Testing for Visual Studio 2008 and 2010

Yay! Finally a CT tool for .NET. I’ve been asking around for a tool like that, and recently found it
Håvard Stranden has recently released it, and I’ve been in a dialog with the author to iron out a few bugs. I now have it working great in both VS2008 and VS2010. After using Infinitest on Java, I’ve been sorely missing a CT plugin for .NET. I’m so used to instant feedback TDD now, that I find it frustrating to work with a normal unit test runner.

It works very much like Infinitest by showing test failures as compiler errors. It runs on compile, not on save. Fortunately ctrl-shift-b saves all open files and builds, so it’s just as good as the “build on save” of eclipse. It runs the test a little slower than infinitest, since it runs all the tests every time. There’s also a small overhead of starting up the commandline runner. However, that happens in the background and you can keep coding while your tests are running – so it doesn’t feel slow. It feels like feedback is coming fast, since I can keep working and thus stay in “flow”
It doesn’t have a filter like Infinitest does yet, but I know the author already has that on his “wanted features” list.

It’s a great tool and I’m never going to uninstall it I’m so happy that we have this kind of tool for .NET now. I’ve used it on a small test project, and look forward to trying it on something bigger.

License: Free
Source: Closed

 

How does that makeitfaster?

A CT tool reduces the constant overhead of starting up a test runner, and I believe that tools like this will become default part of future IDEs. It’s a powerful way to do TDD, and in my eyes, it’s the only way to do TDD. After having tried these tools, I’m never going back to regular test runners.
It’s simply decreasing my edit-compile-test cycle time, by cutting out the test step. It also increases the likelyhood that I can reach “flow”, which in turns increases my productivity.

If there are other great CT tools for these platforms or maybe for other platforms, I’d love to hear about them in the comments.