Lord Rees on the risks of connectedness

On Friday I was lucky enough to see Martin, Lord Rees, Baron of Ludlow and Astronomer Royal, speak at Sydney University. I asked a question after the lecture, and Rees’s response was unexpected and interesting.

Rees had talked about how he is a technological optimist, but political pessimist. That is, he’s optimistic that science and technology will answer questions and produce solutions to the critical problems humanity will face over the next century as we overreach our planet’s resources. But he’s pessimistic that large groups of people and their representatives will act appropriately to implement the solutions, or even just refrain from directly causing a massive disaster.

So I asked what humans could to today to help ensure that societies and politicians in the future will take the right actions.

I’ll confess I had already thought about the answer I was looking for while listening to his lecture, a product of both what he said and my own recent musings. I was expecting something about widening the availability and understanding of the knowledge that scientific endeavour is providing – informing the masses – and perhaps a dash of increasing connectedness between humans over the globe: improving the cohesion in our global village, our sense of shared fate, even increasing empathy for our fellows.

Instead, after noting that he was far from alone in his political pessimism, Lord Rees suggested acting to minimise the risks of a highly networked society. When humans are so tightly connected in such complex, fragile systems, the impact of small mistakes or malicious deeds can be great. What will happen, for example, when biological viruses are as easy to create as their computer kin? Many humans showed a great desire to unleash computer viruses just because they could. In the global village, the village idiot has global reach. Our tight interconnectedness is dangerous.

The global economy and resource networks are similarly complex, fragile systems that we are far from fully understanding (and probably never will). So while I was going for greater interconnectedness as a gain, Rees is more keenly aware of the risks. Though I’m not sure he would actually disagree with the view I had formed: he happened to mention Facebook and thinks that it’s beneficial, privacy erosion and all. This social connectedness means that we are watching each other, humans paying attention to our friends’ and relatives’ behaviour. It’s harder to go rogue when your social network will notice and, likely, try to pull you back. And, I think, less likely, when such easy social connections increase our sense of belonging to a community, of being one of a tribe.

UPDATE: a video of Rees’s lecture is now available on the Sydney Ideas site.


Party spherification

After an enlightening Smoke & Mirrors cooking course with James Viles at Signorelli Gastronomia, Angry and I decided to attempt spherification of liqueurs. Shots without the need for a shot glass. It came off really well. We made spheres of Southern Comfort and Coke, and of straight Frangelico.

Here’s how we did it.

  1. Dissolve 30g of Calcium Chloride (CaCl2) in a 2L bath of cold water. We used tap water with about 2 cups of ice. The Calcium Chloride warms up the water as it dissolves, so the ice is mainly to keep it at room temperature. Be aware that Calcium Chloride is mildly toxic and may irritate or burn: don’t ingest it, and wash your hands after touching.
  2. Fill another shallow bowl to about 10cm deep with tap water. This is for rinsing the Calcium Chloride off later.
  3. Measure 500g of spirit into a jug or cup with a spout (note: 500g is not the same as 500ml). Slowly and carefully dissolve 2.5g of Sodium Alginate (NaC6H7O6) in the spirit, stirring constantly. It won’t dissolve fully right away, so warm the liquid in a microwave in 10 second bursts, stirring in between, until it’s warm to touch.
  4. Bring the liquid back down to room temperature in the freezer. Alternatively, do step 2 ahead of time to avoid this delay. By now we were drawing a crowd, but if you’re prepared earlier it will look less like chemistry (but correspondingly more like magic).
  5. Fill a tablespoon with spirit, pouring into the spoon. Rest the back of the spoon on the surface of the Calcium Chloride bath, then the slowly but smoothly twist the spoon over, pouring out the liquid. The surface of the spirit will gel immediately as the Alginate reacts with the Calcium Chloride and the blob will draw itself into an approximate sphere.
  6. Leave the sphere in the bath for about 5 minutes, then spoon it out, draining excess Calcium Chloride solution, and drop it in the rinsing bowl. Ready to serve!

You can pick up a sphere from the water bath with your fingers, carefully, and pop it in your mouth. Then squeeze until it bursts with a shock of flavour and alcohol. Unlike a traditional shot, this one will cover your tongue so you’ll really taste it.

The pouring technique can take a little practice in order to avoid odd shapes and “tails”. The quantities above are the result of some experimentation. Too much Alginate and the spirit won’t pull itself into a sphere before gelling, too little and it won’t hold together at all. Use a digital scale to measure, preferably with 1/10 gram precision. We also tried straight Tequila, without practicing beforehand, only to discover that it didn’t hold together at all. Possibly it’s too acidic.

You can buy the ingredients at The Essential Ingredient (Rozelle, and elsewhere) or Chef’s Armoury (Rosebery).

Railcorp, you have a problem

Well, duh.

I think everyone knows you have a problem. Possibly even yourselves. Trains continually run late or are cancelled. The network clogs up causing follow-on delays. Even after degrading performance with a lower-throughput timetable you struggle to meet performance objectives (weak ones, too: eleven out of twelve trains arriving within 5 minutes of schedule). Even this reported performance is artificially high since large segments of the infrastructure are out of service every weekend and not counted. Trackwork is such a likely event that it’s the third most prominent link on the CityRail website, trailing only Home and Timetables.

But these aren’t the problem. These are merely symptoms of your problem. Your problem is best illustrated by your careers page. Or what’s not on your careers page. Apparently you have no need for anyone to work on scheduling, programming and optimisation of the CityRail network. This is clearly a ridiculously hard problem but it seems you don’t want anyone to help you solve it. I can see a couple of possible reasons for this, but they all mean you’re doing something wrong.

Perhaps you think the problem is solved, that what you have is good enough and the problems lie elsewhere. Um, no. I really hope you’re not that blind. You have scheduling problems.

Possibly you already have people employed in these roles. They’ve probably been at CityRail for a long time having moved up from being a station master or locomotive engineer or something equally irrelevant but are deeply familiar with the network etc. etc. Well, it appears that they suck, or have been promoted outside their competence, or just don’t care any more.

Maybe you bought some software from a better rail corporation elsewhere in the world, and you plugged in your network and requirements and out popped the current plan. Wait, so you’re telling me that no-one at RailCorp actually knows how the schedule is created? Please, please let this not be the case. You need to have this knowledge yourselves.

More likely you outsourced the planning and scheduling to some other company. If it was a big consulting firm, we are doomed. But in any case, you probably outsourced it to a team with no idea about rail networks, particularly the myriad complexities of Sydney’s. I’m sure you provided very detailed requirements and they might even have given you exactly what you asked for, but it still sucks. They don’t understand rail. It’s hard! And now whenever you want something changed you need to go back and write a detailed specification and open up the taxpayers’ chequebook and you already forked out millions up front and you’re running a tight budget so really it’s easier just to let it keep sucking.

RailCorp, you need to solve this problem yourselves, and you need to do it with the right people. You need to solve this problem yourselves, build an in-house rail planning and scheduling system, because the problem is really hard and depends crucially on details of Sydney’s particular infrastructure and requirements. You can’t just buy software from someone else. It will have been built with an entirely different set of assumptions, and it won’t work. You can’t outsource it. You already have the complex, detailed domain knowledge from running (barely) the network. You can’t effectively transfer that to another organisation. And the situation will change continually and you need to be able to react, fast.

You need to keep the knowledge of how the network is programmed, and how to create that program, internal. It needs to reside in the collective mind of your employees. The people with the responsibility for an efficient train network need to also have the knowledge and authority required to make it efficient. You can’t pass the buck — it’ll just end up on the floor and you’ll be bending down to pick it up when something … unfortunate happens.

But I have some good news. The people you need exist. In Sydney (and everywhere else). You need people who enjoy solving problems like this. People who get more enthusiastic the harder the problem is, the more complexities and special cases there are to be encompassed by an elusive elegant solution. People who will think about the problem constantly, never resting until they reach a perfect solution (which they never will), but who will reach a really really good solution with imperfections you won’t even notice.

These people are commonly known as computer programmers. Programmers have wet dreams about problems like this. They are hard-problem otaku. There is nothing to match the challenge of a problem like this, the complexities, the endless possibilities. Along with routing road traffic, programming a rail network is one of those problems that many programmers secretly long to attack but don’t have the physical network to try it out on. Good programmers dig a problem like this, and will work tirelessly to create an efficient, elegant and practical solution to it for the sheer joy.

Not just any programmers, of course. Many programmers suck. They’re probably the ones employed by the consulting firms you outsource to. That’s right, you pay a premium to use the crappy programmers who went to work at PWC or IBM because they couldn’t get a better job. You have a better job, potentially, and you need the best programmers to work on it. Perhaps the top 3% of computer science grads would be suitable. Maybe even the top 1%. But they would nail this problem.

You’ll have to do things a little differently though. For a start you need to employ only the very best, which seems counter to the way most government-related organisations work. It might take a while to find them, but you can’t just hire people who are capable, you need people who are brilliant. Luckily for you, they would go crazy for a chance to solve your problems. You’ll need to pay them too. What they’re worth, which is a lot. There are other jobs with challenging problems too, so you need to pay them commensurately or you won’t be taken seriously.

You won’t need many of them though. A team of five, with a great leader and great equipment, should set you back less than $1m a year. Give them what they want (responsibility, authority, freedom, time) and they will solve your problem. Not only solve it, they’ll give you the tools to solve it repeatedly and better. And the knowledge of how to solve it — why the solution works, on what assumptions it rests, its robustness and sensitivities — will be within RailCorp. The experts will be right there with you, and they will take responsibility for what they create.


I’m not particularly optimistic that you’ll ever do something like this. It’s too different to how you’ve worked for the past couple of decades. It might actually work. It probably requires some massive change at the top for you to grow the balls required to do this. No doubt the incumbent management would see all sorts of problems with a plan like this, and it’s not like I personally have deep rail experience. The NSW RTA have the right idea though. Their locally-developed SCATS traffic control system is effective and has been sold to some 80 cities worldwide (brochure). Take a leaf from their book.

In any case, this idea has one definite thing going for it – you’re not doing it now, and what you’re doing now sucks.

There will never be another Beatles

There will never be another Beatles … and that’s a good thing.

Let me explain what I mean (don’t worry, I think the Beatles are great). I don’t mean that there will never be another band as “good” as the Beatles, for whatever subjective evaluation of “good” that you might use. Well, there might never be, if you choose “good” to mean “sounds like the Beatles”, but most of us have a more complex and encompassing taste in music than just one band.

I also don’t mean that there will never be another band who sell as many records or who clock up as many weeks in the world charts (although the Beatles aren’t winning there anyway, it’s probably Queen).

By “there will never be another Beatles”, I mean that there will never be another group who completely capture the world of music like they did. There will never be another super-group as super as the Beatles. Or, for that matter, as super as Queen, the Rolling Stones, or INXS. And yes, that’s a good thing.

With the advent of the Internet, digital media, and file sharing, the music industry has become a different world. It used to be the case that to get your music anywhere you needed to have a contract with a record “label”, a publisher, marketer and distributor. You needed a distributor to get your music into shops where people could buy it. You needed a marketer so that people knew about your music, to get promotion and airplay so that people would want to buy it. And you needed a publisher to put up the cash for all that expensive distribution, marketing and studio magic. It was natural for successful labels to combine all three, and hence the most successful labels (EMI, Sony BMG, Universal and Warner) came to have tremendous power and influence in the music industry. To the musicians, the labels presented a scarcity of available contracts. There were lots of musicians so labels could afford to be choosy, contracting only the most profitable. To consumers, the labels were able to present a scarcity of music and the media on which it was distributed, a scarcity which they had created. And they flourished.

Now the game has changed. The development of affordable digital recording technology means it no longer costs hundreds of thousands of dollars to record an album. If you have some friends with the right equipment and experience you can probably record your first album for practically nothing. So we don’t need record labels to front up the cash for recording any more. With far greater impact, however, the advent of the Internet and file sharing technologies has dropped the cost of music distribution to almost zero. The scarcity of media (e.g. CDs) that the record labels could maintain has disappeared for good. Consumers can now get music for free, and artists can distribute it for nearly free (and with any kind of popularity, fans will distribute it for them). So we don’t need record distributors to print discs or get shelf space in music shops any more.

Fast internet connections and services like (the original) Napster spelt the beginning of the end for the music distribution monopoly. But they still had one very important function: marketing. Unless people know about your music it doesn’t matter that you can record and distribute it for free. With no-one listening there’s no point (and no money). But then came blogs, then myspace, and now services like Pandora and Last.fm, social networks revolving around music and music recommendation. Consumers have begun cutting out the record labels from their last bastion of influence. The speed, breadth and richness of communication offered by the Internet, especially by social networking services, means that people don’t need to listen to the radio to know what new music is out. We’re no longer limited to the record labels’ channels of music discovery. Social networking allows people with diverse tastes, who might otherwise never meet, to discover each other and share music. They also help the artists to reach out the their listeners, to promote their music directly. Such technologies provide an audience for even the most diverse, eclectic music, music that could never before have found its listeners. In short, it opens up the long tail of music distribution. Services like Pandora and Last.FM take it even further, automatically matching listeners up with other people with similar tastes, suggesting new music they’re likely to enjoy, cross-pollenating playlists everywhere. An artist’s reach becomes much more closely tied to how much their music is liked, rather than how heavily it is promoted. A nobody can become huge if they manage to speak to our heart.

So with cheap production, no false scarcity in distribution, and music that markets itself, it looks like the record labels are out of cards. But how does this relate to the Beatles? Well, as great as they truly were, the Beatles’ and every other super-group’s success, owes something to the mindshare monopoly that the record labels could create and to the existence of a mould which they could break. People had very few ways to discover music, and the labels controlled most of them and thought they knew what people wanted. Now, people discover their own music, listen by choice to whatever they want to, rather than whatever’s on air. With the great democratisation of music that the Internet has brung, no group will ever control that much mindshare again; if anything, tall-poppy syndrome will more actively select against it. Diverse and eclectic music can reach its listeners right away. Without labels constraining published music to successful formulae there’s no build-up of pressure for change. And that’s a good thing, because we’ll all be listening to what we really like, not what the labels want us to like.

So there will never be another Beatles, but there will be many more groups who’s music suits your tastes, however mainstream or eclectic they might be. And like never before, you’ll be able to discover these groups and hear their music.

(More? Chris Anderson was way ahead of me)