Comments on: Some thoughs on grid computing…

By: Thomas Mailund

Thomas Mailund — Sat, 11 Oct 2008 12:21:46 +0000

In that case, I guess it is vaporware … Of the project I’ve suggested that Brian has put students on, there is a success rate of 0%, so I am not holding my breath on this one ;-)

By: Martin Rehr

Martin Rehr — Sat, 11 Oct 2008 11:04:43 +0000

Well, it’s still in the student project state, so I think you will have to ask Brian :)

By: Thomas Mailund

Thomas Mailund — Fri, 10 Oct 2008 13:25:29 +0000

How far have you gotten in this, Martin? Is it something I can try out? How much work is it in setting up?

By: Martin Rehr

Martin Rehr — Fri, 10 Oct 2008 13:22:08 +0000

Configuring and packing a Grid Job as a virtual machine image is something we are currently working on in the Minimum intrusion Grid (MiG) Grid infrastructure.
As a part of that project we are looking at diff mechanisms to ensure that only changed data are actually transmitted when transferring the images.

By: Thomas Mailund

Thomas Mailund — Wed, 08 Oct 2008 22:21:01 +0000

In that case, sending an entire OS along with job might work, but it is a large IO overhead for most of my jobs…

By: Mikaelhc

Mikaelhc — Wed, 08 Oct 2008 21:12:02 +0000

Oh, but the VM’s should not be considered to be environments which would be kept on the server – an VM image would correspond to one job, so you would just submit a complete VM image for processing to the grid hypervisors who would discard the image after use.

The grid should have no pre-existing environments of any kind. This way you would be sure to get exactly the setup you want.

By: Thomas Mailund

Thomas Mailund — Wed, 08 Oct 2008 20:40:05 +0000

Dependencies like those you mention would get caught by the installation script / testing of the package.

Installing virtual machines instead of packages is not a bad idea, but you would need to install lots of them to support more than one environment, and then they start to fill up the resources…

As for the compilation time: if the environments can have dependencies, then most of them won’t take that long to compile, and they don’t have to be for each job — only each time a new environment needs to be installed.

By: Mikaelhc

Mikaelhc — Wed, 08 Oct 2008 20:31:55 +0000

But what if your software depends on the newest version of the Linux kernel? Or depends on a commercial or closed-source package? Or if your software runs on Windows/Mac/Whatever?

Another idea would be to create a working minimum setup including the operating system, required packages (commercial, open source, …) and data, and pack it as a virtual machine. The grids should then be running thin (native) virtual machine hypervisors, who would fetch and execute pending vm images. For distributing and parallelizing, any existing protocol could be used, since it could just be distributed as part of the VM. The overhead of downloading a complete VM would not be large (a complete Ubuntu server fits in a 250MB VMWare image), and probably faster than trying to compile an application and its dependencies from source.