2010-02-02
the hose drawer | tecznotes
2010-01-31
Let the sunlight in on climate change | New Scientist
Ed Miliband declares war on climate sceptics | The Observer
2010-01-29
Lighthouses could be switched off | BBC News
2010-01-28
Introducing docent
Flickr and galleries
It's now a little over four months since Flickr launched their galleries feature. I liked it as soon as I saw it: it's taken a frequent request ("how can I have sets of favourites?") and delivered something that does the same job, but in a different way. I know some people quibble about some of the constraints, but I like the limited number of photos you're allowed, and generally I've enjoyed creating and browsing them.
Unfortunately, there's a problem: discovering other people's galleries. Aaron Straup Cope is good at bookmarking them on delicious, and there's an Explore page, but neither of those necessarily find things I'd like to see.
The gist of it
Just over three weeks ago, Kellan announced the first API support for galleries, and I quickly created a Python script that would go through all my contacts and fetch their galleries. It was useful, and it turned up a lot of galleries I hadn't seen, but it had two big flaws: nobody else would use it, and it wasn't pretty.
App Engines and data models
I've used App Engine in the past, but that was before the advent of their experimental Task Queue API, and I didn't use the datastore. Using Aaron's gae-flickrapp as a core, I spent about a week's worth of evenings on and off learning how to use both, ending up with the core of docent¹, a small web app.
There are only four kinds of object: dbFlickrUser, from gae-flickrapp, which handles logged in users; Contacts, which have a one-to-one relationship with dbFlickrUsers; FlickrUser, which is an object for a user docent knows about but who isn't necessarily logged in; and Gallery, which stores information about the gallery itself.
What it does
When you first log in, a task is added to a high-priority queue to fetch your contacts from Flickr. The NSIDs² from this call are stored in a single ListProperty in the Contacts object, and then a new task is added to a lower-priority queue. This goes through the IDs one by one, fetching the galleries Atom feed³ and creating the relevant objects (if necessary). This, and the various tasks to update galleries for older users, make up the bulk of the CPU load of the app, and almost all of the Datastore writes.
The big difference between traditional ORMs and the way I'm using the App Engine datastore comes into play here. In an ORM such as Django, a dbFlickrUser would have a many-to-many relationship with FlickrUsers, which would then have a one-to-many relationship with Galleries. The former would require a join table between them. The query to fetch all galleries from a single user would look something like galleries = Gallery.objects.filter(owner__contact_of__nsid=nsid)
By contrast, in the datastore, Both FlickrUser and Gallery objects have a contact_of ListProperty. As a new user's contact list is examined, their NSID is added to the contact_of list. This is how the pages showing galleries for a contact are built: it's a simple equality test, which is translated behind the scenes to a list-membership test:
galleries = Gallery.all().filter('contact_of =', nsid).fetch(256)
It took a lot of fiddling to break out of the ORM/SQL mindset, based on joins, but I think I'm happier now I have. On the other hand, keeping the contact_of lists on all the objects in sync is something of an overhead, and the query code isn't significantly easier. There's also a rather severe limitation I only ran across later.
Onto the Flickr blog
This was all well and good as I let a few other people at the site; initially close friends, then via a couple of screenshots on Flickr, before inviting a bit more of a burst of users via Twitter. The site seemed to be scaling fine; there was a lot of CPU used fetching contacts, which eventually I managed to optimise by being more selective about updating from the gallery feeds.⁴ In fact, the FlickrUser object is currently pretty much a stub, although I'm thinking of changing that.
However, when docent made the Flickr blog, it hit a serious issue: exploding indexes. The version of the app that was live was doing this query:
galleries = Gallery.all().filter('contact_of =', nsid)
.order_by('-published')
.fetch(offset, per_page)
That extra "order_by" criteria required an additional index, and because it's combined with a ListProperty (namely contact_of), it hit the problem documented in the Queries and Indexes page:
Custom indexes that refer to multiple properties with multiple values can get very large with only a few values. To completely record such properties, the index table must include a row for every permutation of the values of every property for the index.
When I last looked, docent knew about 14,000 or so galleries. While most had small contact_of lists, some no doubt expanded to dozens of people, and so the index was too large to store. As a workaround, I eventually realised I had to abandon sorting in the query and instead use Python, at which point the app started being responsive again. Lesson learnt, the hard way.
Moving On
So, what now? The app is up, and although there are a few errors still happening, they're mainly in background tasks that can be optimised and retried without any impact on users. Personally, it's been a fairly good, if occasionally intense, introduction to App Engine's unique features.
Would I do things this way again in future? I'm not sure. Turning the relationship model on its head hasn't led to an obvious improvement over the ORM+SQL methods I'd use in, say, Django, and while the Task Queue API is very easy to use, it's hard to develop with (since it has to be fired manually locally) and there are other job queue solutions (such as Delayed Job, for Rails, as used on Heroku). On the other hand, even with the heavy load, and not the best of optimisations, docent almost stayed within the App Engine free quota CPU limits⁵, and didn't approach any of the others.
In any case, I'm happy to have produced something so useful, and hope that anyone who tried using it yesterday only to run into errors feels willing to try again. In the meantime, I'm sure there'll be more scaling roadbumps as the site gains users and more galleries are added, but I'm looking forward to fixing them too.
(I know comments aren't enabled on this site at the moment. Feel free to add them on docent's page on Flickr's App Garden.)
¹ Why "docent"? Originlly it was the unwieldy gallery-attendant, but Chris suggested the name, based on a term more common in the US than here for the guide to a museum or gallery. ↩
² NSIDs feel like the primary key for Flickr users: in methods like flickr.people.getInfo, it's one of the key pieces of returned information, and it can be used in feeds to fetch information as well as URLs to photos and profile pages.↩
³ Using feeds rather than API calls can be handy. For one thing, they don't count against your API queries-per-second count; hopefully they're cached more aggressively, both via the feedparser library and on Flickr's side so they take less resources.↩
⁴ One nice thing about getting more users is that the likelihood of finding a contact's galleries in the data store already goes up. When I was developing, I had to fetch everything; for the second user, there was some overlap, saving calls. As the site gets bigger, the number of fresh gallery fetches should keep fairly low.↩
⁵ Since I last wrote about App Engine, it's grown the ability for users to pay for resources beyond the free quota levels. I decided to do this when I hit about 55% of my CPU quota, and the app did indeed reach about 120% yesterday. I don't have a bill yet but I expect it to be under $0.50, which is fine.↩
City Planning with the Bressey Report | Ptak Science Books
"The Days Between" by Allen Steele | Asimov's
New content for a new device | Snarkmarket
2010-01-26
Looking for news | Phil Gyford
Micropatronage and the virtuous paywall | Lee Maguire
Guardian editor hits back at paywalls | The Guardian
2010-01-25
home | things I can't fave
2010-01-24
London 2030: our expert predictions | The Observer
London in 2010 – as predicted in 1990 | The Observer
2010-01-22
LT Museum Launch Film Archive | Londonist
Automatic fare collection and you | Film collection, LTM
2010-01-03
On New Year
I’d forgotten - until yesterday - that the epic post on calendars and blue moons, on the Panic blog, had made me think about doing a post about the changes in New Years. So, before 2010 properly gets going (with most people going back to work tomorrow), I thought I’d try and get this out while it’s still topical.
You’d think the concept of a new year was straightforward. After all, it’s right there: the date is 1/1 (whether you’re European or American), and given we don’t use 0 for dates¹, that’s the first day of the year, right? Well, yes, it is now, for a good chunk of the world’s population. It wasn’t always.
Readers of Pepy’s Diary will know that; indeed, the entry for 1st January 1666/1667 bears two dates. Until the UK changed to the Julian calendar in 1752, the first day of the year was on the Feast of the Annunciation, Lady Day, marking the occasion of Mary’s meeting with the Angel Gabriel. Before then, dates for the first third of the year carry both the date of the Julian and Gregorian year. The British tax year still starts on this date, with (complicated) adjustments for the days lost when the calendar changed.
That’s not the only “new year”, though. Parliamentary years start with the State Opening, in November (or, occasionally) December; the Catholic and Anglican liturgical year also starts in December. Meanwhile the academic year starts after harvest in September. (Australia’s also starts in late summer.) Admittedly, none of those has as much legal force as the calendar or tax year, but still, I thought them worth mentioning.
That’s just in the UK, of course. There are two other obvious major world calendars, both lunar. The Chinese new year (also celebrated in Korea and Vietnam, but not Japan, which swapped to the Western calendar in 1873) is based on a lunar-solar calendar, so it moves around, but not much: it’s defined as the second new moon after the winter solstice, fixing it to a date between 21 January and 21 February (with thanks to this PDF, which did all the sums for me).
Meanwhile, the Hijri calendar, used by Muslims, is a pure lunar calendar, with nothing fixing it to the solar year. As a result, the Islamic new year shifts by either 11 or 12 days a year, moving through the Western calendar every 30 years or so. Even more alarmingly for those used to the rigid certainty of solar reckoning, the first day doesn’t happen until the new moon is officially sighted: this can shift the start of the year back a day, in theory at least. In 2009, the first day of Muharram, the first month, was on 18 December.
I’m not even going to try and explain the various Indian new year’s days, except to note that most of them seem to be around the northern hemisphere spring equinox.
So, happy new year, unless you’re Islamic, in which case, belated happy new year, or Asian, in which case, it’ll soon by new year, unless you’re Japanese, in which case: happy new year.
¹ There’s an exception: astronomers have a year 0, and their convention has been adopted by ISO 8601.
2009-12-20
The Visibility of Skill
Last week, I visited Decode at the V&A, a collection of digital art, and there’s a quote from it in the Times review (as mentioned by Chris Heathcote) that I wanted to examine:
There is no denying the technological craft behind the work in Decode. However, unlike physical craft of the kind that fills the rest of the V&A, you cannot actually see the skill behind digital art. You cannot see the intricate computer codes and algorithms. … All you can judge it on is the “object” itself. And that, while undeniably pretty, is too often underwhelming.
Maybe that’s true for Tom Dyckhoff, but for me, it’s easier for me to conceive how to make something like Chris O’Shea’s Audience (part of Decode, but outside the paywall, to repurpose an internet term) than it is to understand the process behind, say, sculpture or ironwork. Obviously I’m a bit of an outlier, but I wouldn’t be surprised if, in a decade or three, the idea that “analogue” art is understandable and “digital” isn’t falls by the wayside.
Oh, and the exhibition? It’s not the best in the world but if you’re at all interested in digital art or interactivity, it’s worth the £5 to see it. You’ve got until April 2010.
2009-12-14
iPhone, video, colour profiles
I use a lightly modified version of FlickrTouchr.py to back up my photos from Flickr, and also to push them to the iPhone (in case I get terminally bored on the Tube and fancy paging through my favourites). Unfortunately, there are two issues with it.
Firstly, until I made a patch this evening, videos were downloaded with a .jpg extension. I fixed that (note the ‘&extras=media’ argument I add to the URL when fetching photos) and then deleted the old “pictures” manually. (I suppose I could have checked for a matching ID, but… exercise for the reader, sorry.) Sadly, iTunes won’t sync videos mixed with photos: if I want them on my phone, I’d have to add them explicitly as video, and I really can’t be bothered. Still, at least they now don’t cause errors, and I can look at them on my hard drive.
Secondly, and more mysteriously, some of my favourites won’t sync. The ones I’ve checked have a ColorSync profile embedded in them, for a “LaCie 321”, which I assume is a specific (fancy) monitor. Quite why iTunes won’t just ship them over and sod the colour matching, I have no idea, but I wondered if anyone else had come across this or knew why it was enforced?
2009-11-30
King's Cross Northern Ticket Hall
Yesterday, the new ticket hall at King’s Cross Underground station opened. The official unveiling had been on Friday, with the Mayor and Minister for London, but it was on Sunday that regular commuters got their own chance to have a look around.
So, first things first: the station works. It’s big - surprisingly big, in fact, given how much of it is in deep tunnels. It’s shiny enough (although I’m not sure how long that will last). Generally, it’s well signed. The new ticket hall is well located for St Pancras and the new high-speed domestic services due to start properly in December, and the whole thing has to pretty much double the capacity of the (incredibly busy) station.
The station is so big, in fact, that the lifts have their own map. There are nine of them (although one isn’t open yet and, alarmingly, when I popped in today one was closed), six of which belong to the extended station. Oddly, the lifts don’t go up or down automatically; once called, they wait for a button press to ascend or descend. Generally they’re well signed, but the Piccadilly’s Lift J is hidden between the platforms- I figured out where it was by descending from the interchange subway (pictured above).
The opposite end of the subway to the Victoria line sees the Northern line’s new concourse hosting an artwork by Knut Herik Henriksen, as discussed at Building Design, and photographed by Londonist (cheekily reused above). I suspect the art is so subtle as to go unnoticed by many, but I quite like it (although perhaps more so in photographs and diagrams than in reality). Londonist’s pictures really do a good job of capturing the way a 2009-era Tube station looks when freshly uncovered, too; the shot up at the ticket hall ceiling from the bank of four escalators down to the subway level is lovely.
Speaking of the interchange subways, they’re very, very long. The map above shows how the three deep lines (the Northern, Piccadilly and Victoria) more or less meet at a point a the bottom of the current Tube hall’s escalators. The new hall feeds down to a much longer set of passages (in peach), especially for the Victoria line (where they connect with the existing, now barely used, subway to what was once the Thameslink station, now maintained as an exit to Pentonville Road).
There’s nothing really wrong with that. What is somewhat offputting is that the signage at the new entrance to the Underground from the main-line station’s concourse is that it suggests any deep-level passenger should head via the Northern hall. This turns a one and a half minute journey into a four-minute one.
Of course, the signs are there for the confused, and there’s probably merit in sending people down the wider, shinier new corridors. If I get the choice, though, the older entrance is far more likely to be the one I use.
Still, it’s good to see a project that nearly didn’t happen take a massive step forwards.
2009-06-29
Olympus PEN E-P1: A Hands On Review
Introduction
The big digital photography news this month has undoubtedly been the launch of the Olympus PEN E-P1. If you're not the sort of person who checks sites like DPReview (where the PEN has taken top place in "most popular cameras"), the E-P1 is a rather strange, interesting new camera. Using a system called Micro Four Thirds, it offers interchangable lenses, a relatively large sensor, and by abandoning the mirror and pentaprism of a true SLR in favour of "live view" technology, a compact body. I was lucky enough to have the chance to spend a few hours playing with one over the weekend, and here's what I thought.
There's been a small but vocal section of the photography community wanting a small, well-specified, prime-lensed camera for years. Mike Johnston's classic Decisive Moment Digital post set out what he wanted in a digital camera, and why the traditional compacts and SLRs failed to satisfy. For the last few years, people have tried the Ricoh GR-D cameras, the Panasonic LX-2 and LX-3 (and their Leica rebadgings), Sigma's DP1 and DP2, and all had been found lacking. When Olympus unveiled their prototype last year, people hoped their desires might soon be met.
So, how does the E-P1 actually hold up? Sadly, the final design isn't quite as nice as the prototype, but the addition of the grip bulge on the right hand side works well. Physically, it's about half the volume of my Canon 450D (XSi), and about 50% larger than my Fuji F-30 (although the prime lens, as you'd expect, protrudes less). The silver colouring makes it look somewhat consumer-centric; an all-black version would definitely be nice.
Olympus have delivered both "pancake"¹ (17mm f/2.8, 34mm equivalent) and zoom (14-45mm) lenses at launch, with adaptors for both Leica M and full-size Four Thirds mounts (which I didn't get to play with). I'm pleased to see the choice of a wide lens, since the 2x crop factor² means what are standard lenses for film are zoom lenses for the Pen. The UK has bundles with body-only, either kit lens, and finally one with both lenses, although bafflingly, this mixes the colours (what on earth is the thinking there?).
In use
In the hand, the camera is nice and dense; apparently it's made of metal, and certainly feels that way. The one I was using had a neck strap which was quite thin, hung too low, and is apparently a fairly expensive extra; perhaps Olympus have taken the retro thing too far. Of course, as there's no mirror, you compose your image on a screen. The lack of mirror also makes the shutter silent, which is a nice change from an SLR.
The screen was certainly bright enough on a cloudy London evening, but I'm not sure how it would be in direct summer daylight. There is an optional viewfinder, but that's fixed for the 17mm, and I didn't have it during my walkabout. I did try it quickly in Jessops on Thursday, and framing seemed correct, but of course it won't show depth of field or a focussing preview.
Speaking of focussing, it's handled nicely, considering the lack of a direct light path; using the focus ring on the lens causes the live view to flip to a 1:1 pixel view of the centre, which seemed to be perfectly usable when I tried it. Generally autofocus was reliable, but towards the end of the walk (at past 9pm, under cloud) there was a bit of focus seeking with the zoom lens. Speaking of that lens, it has a neat feature, letting it collapse up for storage. I think that both lenses have the same lens cap diameter, but differing filter threads: the prime is 37mm, the zoom 40.5mm.
Naturally, the Pen has an orientation sensor (one of those features that tends to be forgotten by reviewers, but which can be annoying when absent), but it also has a view mode where there are two on-screen level meters, which is handy for architectural shots. In fact, there are a wide range of display overlays, including a grid, a rule of thirds view, a live histogram, and a multiple-shot view. Unfortunately I forgot to take a picture of this in action, but it seemed to work well.
The screen doesn't fold out (something that until recently was confined to compacts, but which has spread to SLRs), but it is visible from fairly wide angles (I held it above my head and was able to make out enough to frame pictures). There's the now-obligatory video mode, which offers HD (at 1280x720, not the larger 1080 size), and it seemed fine, even in lowish light. (I didn't get to take one of my usual "train entering the station" videos, but I did get some on an escalator.)
Speaking of low light, the Pen can be pushed to ISO 6400. I tried this on a couple of shots inside a restaurant (with dim, reddish lighting), and while the grain is very noticeable at full size, scaled for the web (or even full screen), the image is completely usable, with a grain-like quality, if anything. Combined with the relatively wide f/2.8 on the prime lens, I'd say low light performance would be considered good by anyone who's not used to a modern DSLR.
A drawback that's plagued digital cameras since their invention, shutter lag, is unfortunately also a problem for the E-P1. I first noticed this when trying to take video; I'd see a cyclist under the Queen Elizabeth Hall start moving, and click the shutter, but it would take a good second or two to start up, missing the action. I tried a few other times to take still photos while walking and the problem was the same in that mode. Perhaps it was due to focussing time, since it was perfectly fast in continuous shooting, and I could have been asking too much, but I'm sure it's slower than I'm used to from my Canon SLR.
More minor, but perhaps also noteworthy, is the fact the combination of the prime lens's maximum aperture of f/2.8 and the 2x crop factor mean that getting narrow depth of field is a little trickier than it would be at full-frame (or even a 1.6x crop with a f/1.4 lens). Having said that, there's some nice depth of field on this portrait (which I didn't take). One final niggle: the picture review seemed a bit slow to come up. If you don't chimp, you won't care.
To be fair, neither of these were serious issues in most of the shots I was taking, since I usually shoot buildings, details, signs, and other things that don't move, and I suspect a bit more time on my part to think about how to roll with the camera would have made even more difference. Still, they definitely need to be mentioned.
I've posted a set of images and videos, with original size available, on Flickr. I'm not a pixel-peeper, and at web resolution they seem nice; definitely better than I'd expect from the F30, and probably about on a par with the 450D.
Conclusions
I'm very happy to see this camera come to market. There's definitely room for a compact yet professional-quality camera, and this is probably the closest I've seen to being the DMD. Unfortunately, the last few years have seen the prices on digital SLRs built around the mirror/pentaprism drop so far as to squeeze this part of the market; who'll pay £700 for a body and kit lens when you can buy a Sony or Nikon for half that?³ On the other hand, it's a heck of a lot cheaper than digital rangefinders, which it can also claim to compete against.
Hopefully some people will look past the sticker and realise that this is something interesting. Would I recommend it? If you're looking for something pocketable but powerful, don't mind not running with the mainstream, and can justify the expense, then I'd say its unique abilities definitely make it worth serious consideration. That said, I doubt I'll get one myself. Maybe the next version?
The Good:
- Solid, sturdy feel
- Picture quality at SLR standard
- Sane menu structure
- Low-light performance seemed good (although some focus seeking)
- Olympus are genuinely trying something new
The Bad:
- Shutter lag
- Price - it'll have its work cut out making room in the market
- Limited choice of lenses (but then, it is just launched)
The Ugly:
- No black model - choice of white+cream or silver+black
- The bundling of randomly coloured lenses
Thanks to Ghene Snowdon for fixing it for me to have the chance to play with the camera. (Ghene's pretty active on London Flickr Meetups; it's worth paying attention there, as I know friends who've got to try out cameras there for various reasons before.)
¹ A pancake lens is a very thin prime (ie fixed focal length) lens, named for the fact it's as thin as a pancake.↩
² The "crop factor" is the ratio between the focal length needed for the sensor and the focal length for 35mm film. This means that, to get the same field of view as a 50mm film SLR lens would provide, the Olympus needs a 25mm lens, while my Canon SLR requires 32mm.↩
³ Hopefully the next few weeks will see the street price drop a little. Canon's 450D dropped from £650 to £500 over the first three months it was on the market.↩
2009-06-09
Quick thoughts on iPhone 3GS
Well, I'm sold.
I've owned an iPod touch for eighteen months. At the time I didn't want to take a punt on the just-released iPhone, but in the intervening time the launch of the 3G hardware made me consider buying one. I'm disorganised, though, and when it got to April I decided to hold off, suspecting the hardware would be refreshed in June.
Of course, it has, and I'll soon be getting in touch with O2 to pre-order My First iPhone. I'm undecided on whether it's worth spending the extra for the 32GB model, but I probably will. The camera improvements (autofocus, slightly higher resolution, and video) are nice; I'm enough of a fanboy to cheer the compass (Google's Sky Map would be lovely), and of course it'll be nice to have a faster device. (Will games throttle their speed on the new hardware, I wonder?)
Existing owners of iPhones are a bit peeved, though. Unlike the last time there was an upgrade, O2 aren't doing anything to let people upgrade early, and operators everywhere seem keen to annoy people who want tethering, either by not offering it or overpricing it. Personally, I'm not that bothered. I know going in there's almost certainly going to be six months in late 2010 when I don't have the latest and greatest, and I dare say I'll cope. (As Matt Jones put it on Twitter, "if you like the shiny, don't be whiny.") A price cut in the UK would have been nice, but I suppose O2 don't feel they need it. Maybe if exclusivity ends?
(I also wonder if the loudest complainers are the same people who are used to upgrading their laptops with every speed bump? That's not a group I've ever been part of; instead, I aim to make my machines last at least their three years of AppleCare. Perhaps the first group are just more vocal, or more used to being able to buy what they want? Of course, iPhones aren't computers, but I assume people think of them as more like computers than phones.)
There is a subset of those vocal complainers who may have a point- developers. The iPhone platform now has devices that run the gamut from the first generation touch, which has no camera, Bluetooth, or support for microphones, to the iPhone 3GS, which has al of the above built in, plus the improvements noted above. The speed range is getting quite large too, and I can understand the desire of devs to get cheaper access to various bits of hardware.
For now, the best bet - outside of large companies - seems to be to find people to test things, but that's hardly the best approach. On the other hand, expecting Apple to duplicate Google's I/O stunt - handing out free phones to every attendee - wasn't likely either. I also wonder if Apple are expecting that developers will just use the emulator?
Still, for all the complaints - largely unjustified, as we all know telcos are like that - this is a perfectly good incremental update. As Steven Levy says, "It's not a game changer." It doesn't need to be, though, and I'm sure it'll do well.
