muesli's Blog - Development

Updating Web 2.0 Profiles Really Really Sucks!

nospam@example.com (muesli) — Wed, 04 Feb 2009 08:11:41 -0500

Don't you hate it? You've decided to take a new snapshot of yourself and update your avatar on... oh... Last.fm, Digg.com, Facebook, Twitter, MeinVZ.net, Lokalisten.de, everywhere. You have to go to every single website and change it manually. Changed your relationship status, phone number or discovered you can't stand your favorite movie anymore? There you go again. I don't even know how many websites I've signed up for now offer a twitter-like status update that I can't be bothered to use.

I'm kinda fed up with it.

I wish I could publish a standardized XML file on my webspace which contains all the data (like my Avatar, Current Status etc.), so all those Web 2.0 services could parse it and update my profiles automatically. There could even be a ping-back like mechanism where I tell the services when to check my XML for updates to avoid unnecessary traffic.

Now it would be easy to spec out such an XML and offer APIs to parse it in various languages. But I guess it would be daft to do so, since existing services don't even start to pick up on OpenID. After all, collecting this precious data is all those Web 2.0 services do and you don't want anyone to interfere with your core business, right?

Yet at the same time I feel like all those services would be able to gather even more data about me if they were able to import and sync my data with their profile.

(I'm aware that one would probably be able to write a Facebook application which could import the XML and update your profile. As long as Facebook is the only service to support this import mechanism, it wouldn't really help anyone, though.)

Update: It seems OpenID itself already supports avatar syndication. Read more about it here. (Thanks to Julian Stahnke for the link)

libketama - a consistent hashing algo for memcache clients

nospam@example.com (muesli) — Wed, 11 Apr 2007 05:39:59 -0400

We wrote ketama to replace how our memcached clients mapped keys to servers. Previously, clients mapped keys->servers like this:

server = serverlist[hash(key)%serverlist.length];

This meant that whenever we added or removed servers from the pool, everything hashed to different servers, which effectively wiped the entire cache. We add (and sometimes remove) servers from the memcached pool often enough to warrant writing this - if your memcached pool never changes, you can probably stop reading now

Ketama is an implementation of a consistent hashing algorithm, meaning you can add or remove servers from the memcached pool without causing a complete remap of all keys.

Here's how it works:

- Take your list of servers (eg: 1.2.3.4:11211, 5.6.7.8:11211, 9.8.7.6:11211)
- Hash each server string to several (100-200) unsigned ints
- Conceptually, these numbers are placed on a circle called the continuum. (imagine a clock face that goes from 0 to 2^32)
- Each number links to the server it was hashed from, so servers appear at several points on the continuum, by each of the numbers they hashed to.
- To map a key->server, hash your key to a single unsigned int, and find the next biggest number on the continuum. The server linked to that number is the correct server for that key.
- If you hash your key to a value near 2^32 and there are no points on the continuum greater than your hash, return the first server in the continuum.

If you then add or remove a server from the list, only a small proportion of keys end up mapping to different servers.

The majority of the code is a C library (libketama) and a PHP4 extension that wraps it. I've also included a class from our Java client. (Java Collections makes it rather easy). We use a single-server memcache client wrapped with a native php class to make it multi-server capable, so we just replaced the hashing method with a ketama_find_server call. (should be easy enough to plug this into libmemcache if need be)

http://static.last.fm/ketama/ketama-0.1.1.tar.bz2

We've been using this in production for all our PHP installs and Java services at Last.fm for around 10 days now. We deployed it just in time to smooth over moving loads of webservers between datacenters.

For further information, please refer to the README inside the tarball or these threads on the memcached mailing list:
http://lists.danga.com/pipermail/memcached/2007-April/003853.html
http://lists.danga.com/pipermail/memcached/2007-April/003834.html

Know C++, Maybe Even Love Qt4?

nospam@example.com (muesli) — Tue, 14 Nov 2006 17:38:12 -0500

Someone out there looking for a new job opportunity? Enjoy accessing a huge music-library? Even more important: always wanted a red user-icon on Last.fm?

There you go, start working at Last.fm as a C++ Developer. You will be working on the Last.fm Client, a Qt4-driven piece of software for Windows, Linux and OS X. Want more info? Click the link. It's worth it. Oh, and it's fun.

Now I wanna see you guys applying

cheers,
muesli