Planet HantsLUG

UDS Karmic Videos and HTML5 Goodness

2009-07-02T23:25:31+00:00

I noticed that the videos from the most recent Ubuntu Developer Summit are now online, and thought I’d have a play with the new embedded HTML5 video stuff in Firefox 3.5.

Rather than view all the videos by downloading them individually I thought I’d make a page where I can view them all sequentially.

Here is the html I threw together. Guess it will look rubbish in anything but Firefox 3.5. Of course that’s no guarantee it will look any good in Firefox 3.5. Just, y’know, you’ll see the videos

Bog Roll: Mr Miot's Rhubarb Jam

2009-07-01T21:46:00+00:00

Yesterday I made a batch of jam using Mr Miot's method. It's based on his standard method which is different from the method I've used myself previously.

1 Kg Rhubarb (frozen then defrosted)
0.8 Kg Sugar
½ a lemon (frozen then defrosted)
200 ml rhubarb juice (from the defrosted 1 Kg)
250 g crystallised ginger (my addition, not in the French original)

First you freeze the lemon and the chopped and cleaned rhubarb. Freezing and defrosting the lemon should ease the extraction of pectin for setting the jam. Freezing and defrosting the rhubarb should extract water juices from it, keep just 200 ml.

Heat the sugar, juice from the lemon and the lemon along with the rhubarb juice up to boiling point (121°C). Once it's rolling along add the chopped rhubarb and return to the boiling point. Boil hard for a further 15 minutes (give or take) and then add the ginger. After removing any scum and a a few more minutes it should be ready to pot.

I jammed 2.2 Kg of rhubarb with 1.76 Kg sugar, two small lemons and 0.5 Kg of chopped crystalised ginger. Tasted okay on the night, but rhubarb and ginger takes a few days to reach full flavour.

Using the latest Gwibber on openSUSE

2009-07-01T20:46:16+00:00

We're in the process of getting the relatively stable 1.0 branch of Gwibber into Contrib. So, I decided to test out the latest trunk to see how it's looking. originally my plan was not to publish the packages, assuming they could be ropey. What I've found is that they are more stable for me than the stable ones and have a lot of the functionality that one now expects of a Twitter client.

So, I'm publishing them. They're in the home:Riggwelter:GNOME_Contrib repository for 11.1 and Factory. Feel free to test them but the usual warnings about non-stable and non-official packages.

Cowon iAudio S9 with Banshee on Linux (3)

2009-06-30T10:39:41+00:00

Previously, I've blogged about my adventures with the gorgeous Cowon S9 and Banshee/Linux. In the intervening time, I've been working towards getting it working with libmtp.

While I've been doing that, Cowon have released an updated firmware that adds M3U playlist support. This is a real result for Linux users. The S9 doesn't actually seem to be able to read the M3U files correctly as yet but I've opened this as an issue with Cowon and hopefully they'll fix it in the next release. Remember, this latest firmware is only a beta and may eat your children or your data - install with care although I've had no problems other that then non-reading M3Us.

Just have to get the HAL information integrated upstream so that people don't have to download my .fdi file for it.

Track my thoughts on this via Twitter.

Bog Roll: To Bing For

2009-06-29T19:49:00+00:00

Recently Microsoft replaced their also-ran web search engine MSN Live Search with an all new Google beating search engine called "Bing". Their old search engine wasn't actually that bad, it's just that no body used it, so along with a redesign they came up with a new name that they thought would be more catchy.

Just like Google they want Bing to enter normal language and for people to use it by default - gradually pushing Google into the same obscurity as Netscape, Stac, AOL, Yahoo!, Real and countless other companies that MS decided to destroy.

So here we go with some examples of how to use "Bing".

To Bing for - to look in vain
I Binged it - I looked and couldn't find it
I've been Binged - I've been swamped with irrelevant commercial data

and so on... The old MSN Live engine wasn't too bad, sometimes it was even better than Google but no one used it. Considering this is Microsoft's nth go at search it's sad that it's actual worse than it's predecessor...

Bog Roll: Strawberry Jam

2009-06-28T21:56:00+00:00

Yesterday we went to a local PYO farm to collect fruit for jamming. As much as we love redcurrant jelly we decided to skip it for this year and try something new - so I spent £15 on mostly strawberries.

Strawberries are terrible to jam, they are low in pectin, high in water and (in the shops in the UK) low in flavour. If it were not for the national obsession with them, no one in their right mind would bother with them...

I decided to use the recipe of Francis Miot, who is some top French jam maker:

700 g strawberries (fraises)
350 g redcurrants (groseilles) - you are supposed to remove the pips with a goose quill but we skipped this...
200 g water (d'eau) - it seemed an awful lot but it did turn out okay
1/2 lemon (citron décongelé) - ideally frozen and defrosted
800 g sugar (sucre blanc) - does not need added pectin
30 ml red wine (vin rouge) - we skipped this as we don't drink wine

The method is his standard method. First heat the sugar, water and lemon (squeezed juice and whole fruit) up to a full boil (121°C), then you add your topped and halved strawberries and (deseeded) redcurrants and bring back to a full boil. You then boil on full heat for 20 minutes before potting into hot cleaned jars as normal.

For best flavour do not add butter, remove the scum with a slotted jam spoon instead. Don't soak your fruit overnight in sugar as it draws out too much water - or so Mr Miot says.

We started with 2.1 Kg strawberries and 1.05 Kg redcurrants and yielded 13 (full) 370 g Bonne Maman jars. This morning we opened a jar to test - VERY GOOD!

Teeching Me A Lesson

2009-06-28T21:33:21+00:00

Yesterday I went to see the second of two performances of “Teechers” by John Godber at the Oasthouse Theatre in Rainham. The play was being performed to raise money for Jenny’s trip to Ecuador. She assures me that this isn’t just a holiday, but that she’s going to help teach young children. More importantly, it was an opportunity to see three very good friends of mine acting. It’s been a long time since I’ve seen Chris performing, I’ve never seen Jenny in as big a big part and I’ve never seen Heidi perform in a play at all! It was all very enjoyable and hopefully raised some cash to help swell the charitable coffers. Particularly impressive was that the cast of three brought to life about twenty different characters, which was an excuse for some particularly tongue-in-cheek performances which I’m sure would be recognisable by anyone who has worked in education. Not, I should add, just characterisations of students, either. It was certainly fun to see some classic silly voices be wheeled out for some of the smaller characters.

The Teechers cast

The programme for the play was also special, as I had taken the cast photos for it. It was, pretty much, my first commission, albeit not a paid one! The session, which was a couple of months ago now, was fast and fun, but I learnt the same lessons as Graham “codedragon” Binns did recently shooting outside in strong direct sunlight. Although I did have a reflector, there wasn’t time to use it as effectively as it could have been. This was because I was rushing. (To be fair, there wasn’t much time, we all had other appointments to make.) I shot lots and hoped they would be OK, rather than taking a bit longer to ensure the shots were set up properly. So I learnt some important lessons and am looking forward to the next time a similar opportunity arises.

I gave a CD with the JPEG versions of all the image to the cast with a list of of images I recommended, although I was doing so without knowing the context of the play. It was therefore quite interesting to see the ones that they selected for use; on the programme covers, a centre-page scrapbook montage and for each character (not cast) biography. (The image above is the one they picked for the poster.) It was surprisingly gratifiying to see photos I had taken all over the programme.

I also won a bottle of wine in the raffle. Having previously scoffed at the auction of cakes and comestibles at other AmDram productions, I am now convinced by this activity and will petition the National Theatre to follow suit. I want to see Trevor Nunn giving away Blue Nun forthwith!

Sermon: Matthew 6.19-24 - Where are we banking?

2009-06-28T17:46:43+00:00

Recorded today at St Paul's. 28 minutes long. Available as Ogg Vorbis or MP3:

PDF of the accompanying presentation.

My hovercraft is full of eels.

2009-06-28T13:01:46+00:00

Recently I've been seeing an awful lot more bounced mail addressed to my domains, to the extent that I now wonder whether they are deliberate "attacks".

Over the past four or five years I'd expect to receive one joe-job attack every six months. Over the past two that's risen to once every two months. For the past two months its been once a week.

I run several domains on my Xen guest, and most of those domains rarely have mail received, so there are only a few localparts. (A "localpart" is the bit before the @ sign in an email address.)

My main domain is steve.org.uk and unfortunately this was historically setup with "catchall" behaviour. I used that wildcard expansion pretty seriously so I had localparts such as "slashdot.org", "lwn.net", etc. Over time I've stopped making up new addresses and just stuck with "steve".

Still I'd never quite gotten round to enumerating all valid localparts, instead I tried to mitigate against these rare bounce storms with various simple hacks. For example the following procmail recipe to file away bounces:

#  Bounces
#
:0:
*(Return-Path:).*(<>)
.Automated.bounces/

However this doesn't work as well as it used to - too many ~~idiots~~ people are using challenge/response systems so I'll receive a reply to a mail I didn't send which doesn't look like a bounce (ie. There is a real envelope sender.)

In short blocking bounces by detecting an empty envelope sender is not a complete strategy these days. I started down the heuristic path blocking mail to "unlikely" localparts via patterns such as:

[0-9]@        DENY  Localparts never end in digits
,             DENY  Localparts never contain a comma
|             DENY  Localparts never contain PIPES.
^([^a-zA-Z])  DENY  Localparts start with a-z/A-Z
"             DENY  Quotes are never used in accounts on this system:
'             DENY  Quotes are never used in accounts on this system:

That was actually a simple change to make, via the addition of a new QPSMTPD plugin and it managed to block a lot of the bounceback spam - regardless of the envelope sender. For example:

IP:84.45.254.18    sender:<> Recipient:treacherously9@steve.org.uk
IP:203.202.253.252 sender:<> Recipient:envoyz0@steve.org.uk

Blocking "unlikely" localparts wasn't perfect, but without implementing BATV or enumerating valid localparts there wasn't too much else that I could do. In terms of numbers yesterday I blocked just over 18,500 messages with these six rules.

I also wrote a couple of cronjobs to look at the contents of the Automated.bonces folder so that I could add per-user rejections on the specific addresses being received - with some whitelisting.

(For example if I received 20+ bounces to fluffy32qp@steve.org.uk within the space of ten minutes I'd drop further mails to that address automatically.)

Anyway enough is enough. Today I woke up to just over 40,000 replies to mails I didn't send. I've now scanned my mail directories for all the email addresses I've ever used and will now only accept mail destined to those localparts.

Thankfully it turned out that since 1999 (when steve.org.uk was registered) I've only used about 150 distinct localparts, and many of those are now obsolete. So hopefully I'll now have less of a problem.

It seems to be paying off already:

62.193.234.95   wpc0505.host7x24.com  <>  virtual_rcpt_ok
    901     mail to subtotalingxa@steve.org.uk not accepted here (#5.1.1)

65.99.223.234   cobra.compukey.net    <>  virtual_rcpt_ok
     901     mail to suctionsw@steve.org.uk not accepted here (#5.1.1)

207.44.156.81   box19.fuitadnet.com   <>   virtual_rcpt_ok
     901     mail to reappearcum@steve.org.uk not accepted here (#5.1.1)

In the future this means I could still get flooded with bounces, but there will be two outcomes:

The bounces will not hit valid localparts and will be dropped easily, quickly, and cheaply.
The bounces will hit valid localparts:
- Real bounces will end up in Automated.bounces/
- Challenge/Response things will still reach me. Sigh.

Still this is progress and I can steal some ideas from this great spam filtering service (ahem) to improve the handling of those! (I explicitly chose to use a similar but different system for my personal mails. Even though my support system is on another box I want to avoid problems where failures requiring human intervention are swallowed in the same way that the original one was. Those kind of reasons mandate a similar system but different implementation.)

I guess I could publish some of the qpsmtpd plugins I use locally virtual_rcpt_ok, virtual_badusers, rcpt_pattern_test, etc. Then again most people who do funky things with qpsmtpd will have plenty of choice already.

ObFilm: Monty Python's Flying Circus. (OK technically not a film. Sums up my mood though.)

Links for 2009-06-27

2009-06-27T22:00:20+00:00

Ladies and Gentlemen, [jwz has] ported Dali Clock to the Palm Pre.
Phil Hands explains how to do passphrase-less ssh properly.

File Uploads in Firefox 3.5

2009-06-27T13:26:05+00:00

Apparently, improved developer control of browser-native file uploads, something I wishlisted back in 2007, is going to be available in the upcoming Firefox 3.5.

Perl is Alive: UK Train Running Times

2009-06-27T10:59:00+00:00

A while ago Greg McCarroll gave an interesting talk at the London Perl Workshop on the topic of "JFDI". His example lead to the creation of WWW::LiveDepartureBoards.

I later used this as an example topic for a Perl talk to my LUG: Achieving the Impossible with Perl. At the same time I also supplied Greg with a patch.

Since then Network Rail have started to provide a SOAP interface to their running time data, and there is now a new interface Net::NationalRail::LiveDepartureBoards. This has the advantage that it's not a screen scrape, instead it's a SOAP monster...

What I want is a trivial interface, a LiveDepartureBoards::Simple... I suppose now is the time to have a look at them both in detail and see if I can abstract over them to create a drop-dead simple interface that uses either as a back end.

Nobody touches the second shelf but me.

2009-06-27T10:42:22+00:00

It seems the IMAP client crash I accidentally discovered in Thunderbird/Icedove was already known.

My report is a duplicate of a bug which was previously reported in 2007. Oops.

ObFilm: The Lost Boys

Web developers who don’t understand how email works

2009-06-26T23:53:53+00:00

More annoying that companies who don’t believe ‘+’ in an email address is valid, are ones that did until their site got redesigned, leaving you unable to log in and having to explain the problem to people that can only read scripts. I’m looking at you, MBNA.

Update:

  CustomerService@MBNA.co.uk
    SMTP error from remote mail server after RCPT TO:
    <CustomerService@MBNA.co.uk>:
    host mbna.co.uk.s7a1.psmtp.com [64.18.6.14]: 554 No
    relaying allowed - psmtp

Update:

From: autoreply@customerservice.mbna.co.uk
Subject: Customer Service Reply

Thank you for replying to this email.

Any emails sent to this inbox are not
responded to. If you have a query about
your account please check your account
details online.

Do you REALLY appreciate free software?

2009-06-26T18:05:56+00:00

I’ve enjoyed freedom for a long time.

Not only is Linux software generally open source, it’s also given away. So not only do we not have to pay for it, but we can also see and develop on top of other people’s previous work. Lucky old us, free software and money we can keep because these Linux developer types work for free, outstanding.

That is until….

Over this last month or so I’ve been working on a package called byobu. Now, when I say working on I mean that I’ve been essentially working on the man pages and coming up with a few ideas. The trouble is that life and work really do get in the way of helping out. After a long day at work, I’m often lacking motivation to sit at the computer and work, even though I have a head full of things that I need/want to do. But it takes commitment, and time; time that could be spent doing many many other things.

So, next time you’re using a piece of software that you love, consider finding that project’s donation button and congratulate them on their fine product.

I’m not suggesting that everyone should be paid to develop software or maintain it, and I’m certainly not suggesting I should be getting fists full of dollars for the teeny tiny amount of work I’ve done (I’d only forward it to Debian). But using that donation button might allow project leads to thank their team members for their work over the last release cycle in some way other than words.

Go on.. it’s a great way to say thanks.

Caption Competition 3

2009-06-25T22:36:00+00:00

Security and stupidity

2009-06-25T21:17:07+00:00

Okay, I’ll rise to the bait.

You can’t be serious. There are often people seeing what you type, besides which, if this was adopted then people would _start_ looking. When I’m assisting someone on a computer and they are at a password prompt, I even go to the lengths of looking away - very pointedly moving my whole head not just my eyes so that they know I’m not looking.

Of course, I’m a sysadmin and a geek - if I _really_ wanted their password I probably could.

We should eat lots of pasta before recording

2009-06-24T21:00:01+00:00

Despite my best intentions, it’s been a while since I posted here. Last time it was to shamelessly shill the latest editions the Ubuntu podcast from the UK LoCo team. This post may not be significantly different as it seems most of the trivia of my day is increasingly dissected and distributed on twitter and identi.ca. Not that I ever intended this blog to be a log of thoughts of the calibre frequently shared via twitter and the like, but it seems I don’t feel the need to write long missives any more. (Although there may well be one about ISPs on the way. Watch this, erm, site.)

Episode 7 of Season 2 of our little podcast has hit the (community donated) mirrors this evening and is already sneaking its way onto all manner of computers and portable media playing devices around the world.This episode features an interview with the executive director of the Open Rights Group, an organisation of which I am a supporter, Jim Killock. Unfortunately the output from the phone interface was very low during the interview, which I didn’t really notice at the time. (We use a digital output from the desk into the laptop which records the show, so I should have just brought everyone else down to the same level then boosted the whole lot in the mix.) But despite a shed-load of compression, I wasn’t able to iron out the difference satisfatorily. At least for me. Technicalities aside it was good to catch up with all the latest campaigns that ORG is working on.

It is always exciting when we release an episode to see the first few hundred downloads hit the logs in a couple of hours. It feels to me that we’re hitting our stride with the new series and format now. We regularly record over an hour of material in two hours. The secret is in the preparation. It’s also in the concentration; our biggest slips have happened when someone has drifted off for a bit. The downside of hitting some kind of stride is that is feels like we’ve been doing it for a while. I feel like we’re half way through the season already, when realistically we’re only one third of the way in. Podcasting is a marathon, not a sprint, maintaining pace without burning out is the key. That’s one of the good things about doing a fornightly show; you get almost an entire week off between episodes. (We keep and eye on the website and news stories betweentimes of course.) At the moment we’re using that “time off” to tweak some of the systems behind the scenes. This has involved upgrading Wordpress and various plugins, patching podcoder and so on.

So, please download the show and listen. A lot of work goes into it. If you like it, or dislike it, please send us feedback through the various routes given on the website.

I'm gonna forget this conversation ever took place.

2009-06-24T20:12:12+00:00

Recently I mentioned I'd been hacking about with a simple IMAP server.

Yesterday I was working on it some more, because the message store I've been testing against contains about 8 million messages and the damn thing is too slow.

During the course of some tweaking I discovered something interesting, every time a specific IMAP client connected to my server it crashed...

I spent a while fiddling around with backtraces and suchlike, but the upshot is I'm still not sure where the client crashes, but I've mailed some details to a few people to see if we can get it narrowed down.

I guess this counts as an accidental security issue. I wonder if I'll be able to collect a bounty? (Not that I'm bitter about past bounty-worthy reports being ignored ;)

Anyway interesting times, when I least expected them.

Mostly this post is being made to test a new release of the chronicle blog compiler - which now allows gravitars and has improved display of comments as demonstrated here.

ObFilm: Rambo First Blood Part II

RSS: Error-prone

2009-06-24T11:19:51+00:00

I subscribe to only about a dozen RSS or Atom feeds, but more than half of them suffer from one problem or another.

Intermittently dumping a dozen duplicate posts.
Dumping a dozen duplicate posts on every refresh.
Duplicating the most recent post on every refresh.
Double-escaping HTML entities, so I see “, ”, … and such like in post names.
XML syntax errors causing total feed outage until some improperly encoded post drops off the feed.
<pre> code snippets that have lost their formatting.
And, of course, the occasional snippet of HTML that doesn't work as intended when removed from the context of the original HTML document and embedded in RSS.

I often have to search for Pipes to get a useful feed, which is a consequence of the way RSS specifies only a data format, not an obligation on producers, an architectural flaw I've discussed before.

But quite aside from this, it seems that a significant proportion of feeds aren't implemented properly.

Obviously we can blame developers for bugs, but the design of RSS may well be a contributing factor. The process of encapsulating HTML fragments in XML is not as straightforward as it looks. The requirement for a unique ID for each post at first glance does not look onerous. But does the ID correspond to the specific version of a post? Or does it correspond to the current version, however it may have changed since it was first published?

RSS may be useful, but it should also just work, and it doesn't. Developers and standardistas alike should start thinking why.

Seen on a comments page today

2009-06-23T21:22:00+00:00

"When choosing a new speaker, members of the house of commons have stuck two fingers up at the electorate and gone for an MP."

Um...?

Sermon: 1 Timothy 6.3-10 - Purses with holes

2009-06-23T09:45:57+00:00

Recorded Sunday at St Paul's. 19 minutes long. Available as Ogg Vorbis or MP3:

PDF of the accompanying presentation.

Gallery 3 Beta 1

2009-06-22T22:05:50+00:00

So, a bit back, I wrote a post about how Gallery have been focusing on making Gallery 3 easier to use. So when Beta 1 came out, I gave it a go on my laptop.

Uploading images (first things first!)

After installation (which was a reasonably slick experience, although I was slightly confused by having to install apache2, php, and mysql first), Gallery neatly leads me through installing the first photos to my gallery. The browse dialog in which you select the photos you want to upload is slightly odd because as soon as you’ve selected the photos in the window, they start uploading. There is a ‘Done’ button but that seems to refer to having ‘done’ the upload, as opposed to having ‘done’ the selection of photos ready to upload – which is what I expected and was slightly surprised by the uploading starting before I expected it to. Wonder if this is intentional…cos it’s a little bit weird?

When the images have uploaded, they’re displayed in a tiled layout on the page (although, at the moment, the photos themselves don’t display – I guess that’s the joy of betas ). The cool thing is that when you hover over a thumbnail, a small toolbar containing the most common tasks (edit, move to another album, set the photo as the highlight photo for the album, delete) appears over it. I did just try to take a screenshot but sadly I’ve forgotten my password so I can’t log back in…and the password reset function hasn’t been implemented yet…

Update: turns out I took some screenshots when I was playing:

Album permissions

And then we get to album permissions. On Gallery 1, the permissions were slightly clunky but most could cope with them. On Gallery 2, the permissions were incomprehensible and when I googled for help I found other people who were similarly baffled and no actual answer to my problems. On Gallery 3, they’ve rightly got rid of the obviously UNIX-style permissions.

You can create different users and groups for your gallery. A reason for creating other users (who aren’t administrators) is so that you can section off albums so they can be selectively seen, for example, by family members, by friends, by work colleagues). When you create a user, you get the option to check the box ‘admin’ which presumably gives the user administrative access to the gallery. The users makes sense but I’m slightly confused as to the groups. I’ve nothing against the groups per se (I can see they might be useful for administrators of massive gallery sites) but I think groups should be an ‘advanced’ option that is not required for use by most people.

I can’t quite work out the ‘Registered Users’ group – it seems to get everything added to it apart from ‘guest’. I added TestGroup group and created two users (TestUser and test2) which I dragged and dropped to the TestGroup group. Worked nicely.

You set who can access each album by clicking Options > Permissions when that album is open, which opens the Edit Permissions dialog box. You then indicate the permissions that each group has on the current album. I like that you work by album but I’m not so sure about dealing with groups. I feel that it’s a bit of a ‘power user’ task to be working with groups – you have to have planned and organised your groups to be able to use this dialog effectively; it also adds a layer of complexity to understanding what permissions an individual user has.

Thinking of my friend who uses a gallery we host, she (and I) would find it a lot easier to work with the users themselves – maybe with a power user tab option to switch to working with groups. I’d much rather say that user ‘family’ can access this album, rather than set up a group called ‘family’ with a user called ‘family’ in it (there’d be little point, typically, to separate out different parts of the family to be multiple different users within the group). I agree that groups can be useful but I just don’t think they should be the default.

Slideshow

And finally, the slideshow facility (for viewers of your gallery rather than for you a gallery owner/administrator) is provided by a third-party Gallery plug-in which is a little slow to load but you get the option to install a browser plug-in that gives you some client-side loveliness.

Overall impressions

Looking good.

Bog Roll: Poppy Fields

2009-06-21T22:00:00+00:00

We went for a bike ride this morning. One of the fields we went past was a lovely shade of lilac and full of Opium poppies, (presumably) grown under license for British opiate production.

If the Daily Mail knew this kind of thing went on in rural Hampshire they'd be up in arms, worried about the youth of today getting high and ending civilisation was we know it. Apparently you don't get much opium from the poppies so you'd need to inject an awful lot to get any effect - hence the industrial scale of the production.

Some harsh realities

2009-06-21T13:00:05+00:00

Recently BitFolk has been accused of overcharging for disk space.

In general I don’t try to defend BitFolk’s price-point - the unmanaged VPS hosting market is flooded and it is very easy to find stuff hosted out of the US or continental Europe for just a couple of pounds per month. Clearly I am not going to try to compete on price alone, yet BitFolk does sit firmly towards the cheap end which I feel is fair given that there isn’t a 24-hour team of support persons in nice business premises.

This particular complaint however seems to stem from the perception that “disk is cheap.” Well, yes, it is fairly cheap. That’s why we sell it at the “fairly cheap” price of £6/5GiB/year (10p/GiB/month), with no VAT added on top. Just because you can buy a 1.5T consumer hard drive for about 9p a gigabyte doesn’t mean that you should expect to find 1GiB of usable disk space on a server in a decent datacentre for anywhere close to that figure!

I try to keep costs down by using a configuration based around 4×7.2kRPM 3.5″ SATA disks with hardware RAID. I would dearly love to have a nice shared storage solution with 10 or 15kRPM 2.5″ SAS disks, or even to use them as local storage. Lack of disk I/O is the limiting factor for how many customers I can put on one machine. The problem is that the storage costs would be around 10 times as much and the target market (mostly people looking for cheap personal hosting) will not pay for it. They don’t understand why it would be desirable; for many of them it may not even be necessary since if they do only a little I/O they get the same performance either way.

So okay, if we resign ourselves to 4×7.2kRPM SATA disks and a RAID card as local storage, the next way to keep the price down would be to buy the disks with the sweet spot for price per gigabyte. At the moment that would be 1T. The problem now is that I’d end up with roughly twice as much disk space as I could ever sell on each server. I don’t get to keep adding customers until the disk space runs out — the I/O operations per second run out first. At the moment I can sell around 700GiB per server.

I thought I would not need to explain that 2×500G in a stripe with no redundancy would be insane, but apparently not, because I am told that some people “don’t need RAID.” I have to disagree, and I feel the ~49 or so other people on the server would also disagree when the first disk failure sees their service down and all their data lost (apart from the ones who have a backup strategy, right? No, really, why are you laughing?). Let’s not go there.

If you recall, I/O is what runs out first. So any sort of RAID-5 configuration is a bad idea because of the read-modify-write problem. The minimum number of disks and the most sensible RAID level then is a 4-disk RAID-10. Four 500G Western Digital Green Power drives will set me back around £165+VAT. You’re looking at around a further £225+VAT for a 3ware 9650 RAID controller. After the manufacturer lies are accounted for and an operating system is installed, there’s going to be about 930GiB of usable space left. We’re now at £390 for the lot, or 41p/GiB of usable space. Excluding VAT.

By the way, I am repeatedly told that Linux software RAID is good enough and I needn’t bother with hardware RAID (even a cheapy one like 3ware). I started off using Linux software RAID and still have one server using it, but that’s due for decommissioning next month. In general it does perform well enough. Unfortunately, hard drives accumulate errors and the only way to find them is to read the disks looking for them. The code for doing so on software RAID needs to be in the main operating system and the Linux mdadm package in Debian (and presumably elsewhere) handles it by means of a cron job that runs once a month to verify all the disks. Because it’s running on the host all the data has to go through the OS and while the machines are under moderate write load I have found that this verify process will take several days to complete and will impact I/O performance. In short it’s actually more cost effective to spend more on a RAID controller and put more customers on one machine.

Now consider the power usage. More than 60% of BitFolk’s recurring hosting costs are directly related to power. Disks aren’t huge power draws when compared to the CPU or chipset, but it’s not an inconsiderable extra cost and it’s often overlooked.

We’re already up to 41p/GiB cost price, but you may be thinking that this is no problem since at 10p/GiB/month, 700GiB sold brings in £70 a month, paying for all the disks and RAID controller after about 6 months. The reality is nothing like this. The full price has to be paid up front to get the hardware into service, but it’s going to be months before the server is full of paying customers. And if those customers don’t happen to want any extra disk space, then still around 50% of this capacity will remain unsold. The remaining capacity is not usable when the IOP/s have run out, but it has to be there from the start just in case there is demand. Does 10p/GiB/month start to look more reasonable yet?

If not, maybe you would be better off going to a really big cloud computing vendor who can take advantage of massive economies of scale to really drive the price down for you. Like say, Amazon S3 who will charge you $0.18/GiB/month for storing stuff in Europe. Plus $0.10/GiB/month more to write it and $0.14/GiB/month to read it.

Finally, the entire point of paying for a virtual server is that you don’t need to worry about the hardware. If it breaks, it’s BitFolk that replaces it, hopefully without you even noticing. If you are sitting there thinking “I could buy a 1.5T hard disk for 9p a GB, screw this!” then you just don’t get it. If from the outset you are prepared to manage your own hardware, and your needs justify purchasing an entire machine, then guess what? Don’t buy a virtual server on someone else’s hardware! Buy your own hardware that is set up exactly how you want (and please feel free to have no RAID and host it under your bed). With this mindset, pretty much every “* as a Service” product is going to look expensive to you because you have missed the point.

Summer Solstice at Stonehenge

2009-06-21T06:40:43+00:00

First off.. Happy Solstice, I hope the day finds you well and brings you the things you need.

This morning I went to Stonehenge to see the sunrise with a couple of running friends, yeah, we ran there and it was a great run. But the sights at Stonehenge were utterly utterly horrible. Seriously. These free festival types need to find somewhere else to go at this time of year, they’re not there to celebrate the solstice and see the sunrise, they turn up for a party, get pissed/wasted, throw their rubbish all over the floor, urinate pretty much everywhere. Why are English Heritage allowing this to happen? There is no justification for a free festival of this scale at any historical site, especially Stonehenge.

I’m not at all impressed, and letters will flow.

If you want to celebrate the solstice then stay at home, find a local place that you find beautiful and see the sunrise there. Don’t come to Stonehenge, please, stay away. That doesn’t just apply to the free festival types, that applies to everyone, the Druids, the Witches, the Heathens, if you live more than 10 miles away from Amesbury then don’t come, go somewhere else, find your own local spot and leave this monument alone.

Yeah I’m grumpy, you would be too.

I go down with one helluva bang.

2009-06-20T15:44:48+00:00

Right now I have a lot of music, and I primarily interact with it via playlists.

I have a cronjob that generates, and populates, ~/Playlists/ every night. I generate playlists on multiple criterion:

~/Playlists/Artist/
~/Playlists/Albums/
~/Playlists/Titles/
~/Playlists/Keywords/

Playlists for specific artists & albums are probably self-explanatory, but the others might be interesting.

For every unique songtitle I have a playlist. In most cases that means there is a playlist called "Song Title" having one entry. But, as an explicit example, I have a playlist called "Under The Bridge" with two entries:

All Saints/Under The Bridge.mp3
Red Hot Chili Peppers/Under The Bridge.mp3

Similarly I break each song title into words, and generate one playlist for each distinct word discovered.

As a matter of randomness I have:

Term	Count
Girl	83
Boy	31

(e.g. Songs containing "girl" in their title: "Madonna:Material Girl", "Amy Whitehouse:Hey Little Rich Girl", "Garbage:Stupid Girl"..)

There are times when I want something specific and my playlist approach doesn't work. For example "All songs which are 2 minutes long, and happy". I guess the problem is working out which meta-data is worth searching/storing, and then working out how to jump from that data to a playlist.

Today, whilst walking into town to buy some new pies, I wondered "How many songs do I have that end in a chuckle, or laughter?"

If I wanted an "ends in laughter" playlist right now I'm screwed. Yet no system I've ever seen allows you to add that level of detail. (To be honest I'd probably give up even entering it.)

In conclusion, my music collection is vast and various, and dealing with it is sometimes harder than I'd like.

How do you handle the music on your computer(s)? (When it comes to mobile-music I just use an ipod telling it to play all, randomly. If a song comes on I don't like I just skip it.)

ObFilm: Lolita

Yellow Liquid and Jesus

2009-06-20T15:44:00+00:00

Jesus walked on water.

What a miracle!

But to be honest there's a question in my mind: Would it have been more or less of a miracle, if he had turned the whole of the sea of Galilee into Custard first. After all we know our Lord is not above a spot of water based transformation.

Admittedly that would have made the actual liquid based promenation significantly less miraculous. It could, and indeed should, be pointed out that the walking upon custard in fact does not violate the laws of nature in any way. Thus pushing the event below the threshold of the miraculous (at least given the medieval scholastic definition).

But this would demonstrate the existence of this sweetened dessert sauce a full fourteen hundred years before its invention. Not only that, but this would give demonstration to a non-newtonian liquid, at a time when fluid mechanics was in it's infancy. Thus giving an irrefutable divine signature to the event and subsequent validation of the Bible as scripture.

It's a difficult one to work through, isn't it?

I suspect the reason our Lord chose to act in the manner he did, is due to the envirmomental impact that the other course of action would have caused. For example I suspect suspention within custard would render fish gills non functional. Not to mention those who relied upon the water drinking and irregation. Finally the sheer quantity of bananas that amount of custard necessitates is beyond the abilities of humankind.

Once again Jesus, well ahead of his time.

I've been discussing this with the CU, anyone who thinks they're just a bunch of unreflective fundamentalists is clearly nuts.

Bog Roll: Perl Blogs

2009-06-20T10:27:00+00:00

For historical reasons I maintain two separate Perl blogs as well as Perl content on this blog. As I've been doing a lot more Perl at home and work of late I thought it best to update things.

use Perl; - my original Perl blog.
Perl is Alive - my newer Perl blog.

I've blogged on use Perl; for a long time, but frustrations with the interface and the development of better syndication technology makes it less important to me and many of the people I follow.

The Perl is Alive site is a newer site, which I'm blogging on at the moment to help it grow. I need to add some more articles to it as well.