Alex Rea

#dbmillion

2012-01-09T16:16:22Z

Note: I wrote this article using up-to-date data on the Monday evening, shortly before the competition closed on the Tuesday.

So, Derren Brown has decided to run a competition where you guess a number between one and a million, and the person with the closest number to the number that he has (allegedly…) written down beforehand gets taken to dinner. All you have to do is tweet your entry with the hashtag #dbmillion.

I’m not one to let a nice dataset like this go past without doing something with it - while also increasing my chances of winning, assuming that it is random.

Code

The code took about an hour to write. It is very simple, and comes in two halves; the first half saves the tweets with the hashtag in real time, and the second half works backwards and saves old tweets with the hashtag, until it reaches the first occurrance. Because this is all public timeline stuff, no authentication was required - just liberal use of the max_id search parameter to walk backwards in time.

Once the tweets were saved, some more code was needed to munge the data. Again, fairly straightforward to write, though I had to make some assumptions. As a result, the data below comes with some caveats:

Anyone who posted more than one number in their tweet isn’t included in my dataset. This omits some people who guessed in the form of “I’m 18! My guess is 23455”, but there aren’t many of those.
Anyone who guessed in the form of “Six thousand, one hundred and two” has been omitted. I didn’t particularly want to get a natural language parser involved, regardless of how straightforward it would have been.

If anyone wants to repeat the analysis, then CakeMonitor has archived the tweets in question in a .csv which he has hosted on MediaFire

Digits

So, firstly a basic test of randomness. If Derren’s Twitter followers are truly pulling from a uniform distribution, then we expect the digit ‘1’ to occur the same number of times as the digit ‘9’. Clearly, this is not the case.

Looking at the graph, we see that the digit ‘1’ appears nearly twice as often (6059 times) as the digit ‘9’ (3135 times). I have no real explanation for why, in general, higher digits are less popular. The digits ‘3’ and ‘5’ seem less popular than they should be, compared to their neighbors if there is some trend here, but it’s certainly not Bedford’s Law.

Popular Numbers

Okay, so it’s clear that we’re not pulling from a random distribution, but we don’t expect that to be the case - humans are notoriously bad at choosing a random number. Asked to pick a number between 1 and 100, ‘37’ occurs a disproportionate amount, we are assured. If anyone knows an actual study where this is shown, however, I’d like to see it - I’ve never been able to track one down, and it’s always passed around as common fact.

We have a million numbers (ignoring the fact that Derren asked for a number ‘between’ one and a million, as much of the internet seems to have done) that we could be picking from. As a result, we surely expect very few collisions. We have around 8000 entries for the competition, and so using the solution for a Birthday Problem where we have a year that lasts a million days and 8000 guests, for a truly random distribution there is a 99.9999999999% chance that we will have a single collision, so we are not surprised to see some. In fact, the expected number of collisions is 32.

We see significantly more than this number of collisions. I present the top ten numbers, in reverse order, with explanations.

270271, with 50 guesses. This is Derren Brown’s birthday.
42, with 52 guesses. The answer to life, the universe, and everything.
666666, with 60 guesses. It’s the devil’s number, except twice so it’s near the middle of the range of 1 to a million - it’s more likely to be close to Derren’s number! I assume that’s the thought process.
7, with 62 guesses. Oh, trusty, ‘random’ 7. Glad you’re able to join us.
400, with 64 guesses. ‘4’ and some zeros - see below for a brief discussion of why the number 4 is making a strong showing in this competition.
2, with 64 guesses. Strictly, the smallest eligible number. Possible that there is some crosstalk here, as #dbmillion started trending, and getting a lot of spam. As a result, things like “U hav 2 see this! #dbmillion” with a link would be counted; I’ve not seen any examples of this though, and 2 does seem to be guessed a lot from looking at the timeline.
123456, with 70 guesses. Digits in order, up to a number that is between 1 and a million. 12345, interestingly, has far fewer guesses - only six.
100112, with 100 guesses. The closing date of the competition. I thought I was accidentally catching people tweeting the detail of the competition, but no, lots of people really are guessing this number. Fair play.
1, with 170 guesses. Staggering. Despite the fact that they were asked for a number between 1 and a million, 106 people have guessed ‘1’. Read (or listen to, in this case), the question, people. I guess a lot of people got excited at the prospect of dinner with Derren. And who can blame them.
4, with 176 guesses. Again, see below for a discussion of ‘4’.

The number 4

I think a lot of people have been trying to second-guess Derren, which has lead them to the number ‘4’. In the video, he describes that the card is attached to his forehead, before pausing and saying ‘or, if you prefer, forr-ed’. I think a lot of people have been taking that as their cue to guess ‘4’. Later on in the video, he uses the fraction ‘four-fifths’ which a lot of people have been clutching onto as verification that ‘4’ is related to the answer.

I’d be dismissing the ‘4’ business as a red herring, if it wasn’t for the restaurant that he was taking the winner to, which is the Ivy in London. That is to say, the IV. I’ll be interested to see if there’s a reveal at the end of this. If not, isn’t it a shame that Derren can’t just take a fan out to dinner without suspicion clouding his every move?

Early on in the proceedings, ‘4’ was second to ‘1’, but then Derren tweeted that lots of people were guessing ‘4’. Weirdly, this cause ‘4’ to surge into the lead (and yes, my data does strip retweets before doing the analysis).

Other interesting data

Off the top of my head, some other interesting factoids from the dataset.

The lowest unguessed number was 53.
This is a representation of all the guesses. White means no guess, and dark blue indicates one guess. The scale goes all the way up to bright red, which is the number of guesses ‘4’ got. The pixel in the top left of the graph is ‘0’, and the pixel in the bottom right is ‘999999’. Each row contains 1000 numbers. The line across the top corresponds to people guessing low numbers (under 1000), and the line along the left hand side corresponds to people guessing round numbers (i.e. ending in ‘000’). It is interesting - but expected, I would argue - that the right hand and bottom edges are considerably less well defined. If anyone has any suggestions for how to present this data in a more intuitive way, I’d be very pleased to hear it.

My guess

What did I guess? Monday evening, the day before the competition closed, I guessed the number in the middle of the biggest unguessed gap, which at the time had a size of 4800. If we assume fair play all around, and that by guessing randomly we would we have a 1-in-N chance of winning, where N is the number of entries, if Derren’s number truly is random then this will have increased my chances of winning by around 25. 25 times a small number is still very small, so I don’t see myself having dinner with Derren, but I’ve given it a good shot nonetheless.

The answer

The number turned out to be 758031, from the man himself. I wasn’t close! Congratulations to the winner, and hope they enjoy their dinner with Derren. Even without such a payoff, I’m still pretty happy with what I’ve achieved here, just out of curiousity’s sake.

Earned

2011-12-31T21:26:55Z

This holiday season is the first time I’ve had some time off since last Easter - one way and another, I just never got around to taking time off. This seemed like a reasonable idea at the time, but towards the end of this year it revealed itself to be a terrible mistake. Still, I made it through with most of my sanity in check, so that’s an achievement, at least.

Dreamcast

My annual summer project is to try and get my Dreamcast online. Without a summer break this year, it never happened. I’ve devoted a small amount of time to it over the Christmas holiday, and - as always - not made much progress. My inching-forward this year was only to confirm that my modem requires a Line-Voltage inducer or equivalent for the DC-PC server to work.

What I did discover during my yearly trawl of haunts-past was that the Dreamcast can now read SD cards and someone is making enormous strides in getting rid of the GDROM drive entirely. Both of these projects interest me, as transporting the thick jewel Dreamcast cases is a huge pain if you don’t want them getting damaged. Sadly, many of mine already are, and I want to prevent further heartbreak. I’ll be watching these projects with interest, and they could compel me to get a VGA cable and move my Dreamcast to Cambridge in the near future…

Last FM

I decided I needed (yet another) project to tinker with, and so started improving a proof of concept HTML5 last.fm player that got posted at the Last.fm forums. I forked it after the original author put it on GitHub and made it scrobble and login, which were the two big things that the proof-of-concept didn’t do. I got a gift of a month’s premium use of Last.fm from the original creator, pleasingly. I’ve agreed to collaborate on further improvements, but I can see time slipping away from me in the near future as the screws really tighten during the third year of my Ph.D….

Another project that was really just an extension of an existing project was a Tumblr to automatically post CAD comics with the CAD Rule applied. It was only an hour or two of coding, and now that I’ve made it a cronjob it should just post away whenever a new CAD is posted. I fully expect to have to debug it a couple of times now that it is ‘in the field’, of course.

Books.fm

Christmas spawned a Kindle, which I have been enjoying using since I opened it under the tree. I was struck almost immediately by the idea that there should be a last.fm-esqe service for books. Naturally, I am not even close to the first to realise that this should be a service, but it is interesting to note that there still doesn’t seem to be such a product. Even something simple like integration with Goodreads seems like it’s a long way away. This boggles the mind.

Amazon is collecting (we presume) all this useful data (i.e. their Whispersync logs), which no-one else has collected before. The fact that they are not doing anything with it seems neglectful at best and offensive at worst. They even already have the ‘other people bought’ algorithm. How hard can it be?

Nizlopi

Nizlopi reunited for a four song set for charity. Naturally, upon receiving the email from the old fan mailing list, I had bought a ticket within minutes. I resolved when Nizlopi was still together to never turn down a chance to see them, and it looked like it was going to be a thoroughly enjoyable evening with a large variety of performers on show.

When the day got there rather early, and was greeted by three other people who had started the queue - all to see Nizlopi. Coincidentally, two were from Cambridge - a ten-year-old and his mother. The former was a huge Nizlopi fan, but had never seen them live and was, understandably, excited. I’d been to the Union Chapel before, so I knew what to expect from the venue, but the lineup blew me away. I encountered three or four new artists that I will be keeping an eye out for (one of which was Jamie Lawson, who memorably offered no banter before delivering Wasn’t Expecting That in a heartfelt manner.

Nizlopi were everything I had remembered, and more. It is a real shame that they’re probably not getting back together, but they hung around after the show so that people could talk to them; some had come from as far as Boston and Montreal to see them! The ten-year-old Cambridge-based friend I had made could barely talk to begin with, and his mother showed Nizlopi his book of Nizlopi songs, where he had written out the lyrics to the songs he liked. JP and Luke both signed it, and spoke to him about music; when he mentioned that he was learning the guitar, Luke handed him his and asked him to play them something. He fumbled around for a bit, lost on the frets in all his excitement, before playing ‘Girls’. I honestly think it made his year.

Hardware

I’ve had some reasonable successes with fixing things in the last few months - of note, I permanently (I hope) repaired a free flatscreen monitor I acquired for free using only a couple of capacitors. Secondly, I repaired an Xbox 360 that was suffering from the dreaded RROD - but sadly this was only a temporary fix, and it has since started freezing. This is a problem that seems to require significant work to fix. Somewhat frustrating after spending money on VGA cables and so forth, but part of the risk associated with such project, admittedly.

We have also made some further progress on the ever-ongoing Dalek, which looks better and better every time we go to see it. My work on it goes in fits and starts, mostly as we keep changing bits around (oh yes, we’ll use the Velleman K8055. Oh, no, an Arduino might be better), which is entirely our own fault - but it’s always frustrating to sideline code that will probably never be used.

Roll on 2012…

Once more, with feeling

2011-08-24T21:14:15Z

It has been quite some time since an entry, and I’ve been nursing this one for a little while now, one way and another. Forgive the disjointed nature, but there isn’t really a narrative thread to link this all together - or at least one worth getting into in a public forum. And so on with the show…

Bitcoin

I’ve been exploring the novel - and on occasion, wonderful - world of bitcoin of late. I have been looking for an excuse to buy a new machine to operate as a server for a little while, as my laptop which has been serving such duty for the last few years is starting to be unable to lift the computational weight that I ask of it - and has an annoying habit of wanting to be shut down on occasion. The confluence of several factors spurred me to action. It has not been purchased with the intention of using mining to make the money back, but getting some of the money back in a form that is currently intrinsically interesting certainly holds a lot of appeal. The fact that my rent includes all bills also helps.

The fact that the community comes up with documents that explain pool hopping, what it is, and prove the optimal strategy is extremely attractive to me. There has been massive media interest of late in bitcoin, which has caused a somewhat amusing series of events, including a boom, crash and a hacking scandal at the largest exchange. This has all been fairly entertaining to watch from outside - it remains to be seen whether it will be as amusing from behind the curtain. Regardless of whether Bitcoin truly takes off, I am happy to have now been at least a small part of it, and it’s exactly the sort of thing I enjoy tinkering with in the meantime.

Last FM

Last FM has sadly moved to a model where to stream on your iPod / iPhone you must pay a subscription. I used my iPod to stream Last.fm radio while I was at work where my PC (due to a unfortunate mixture of SunRay workstations and Linux) does not supply sound, and the result is that I am mourning the loss of what has accompanied my PhD for the last twenty months or so - as is my officemate who did exactly the same thing.

It’s somewhat annoying that all of these music services are pulling the rug out from under us of late. I remember using Pandora when it wasn’t just for those in the US of A. It wasn’t that long ago that Spotify allowed unlimited streaming for free accounts - and I didn’t even mind the ads that much. Now Pandora finds itself in a walled garden, Spotify only allows free accounts a limited amount of streaming a month, and Last.FM won’t let me stream to my iPod. I can’t imagine it’ll be long before my current haven, mflow, ends up in a similar boat, sadly.

Google Plus

A challenger approaches. I snuck in under the rope cordon, past the bouncer, fairly early on through the ‘share-something-with-an-email-address’ workaround, and quickly found the same thing everyone else did: even if the animation to drop people in circles was incredibly satisfying, there was very little to do with no friends on the network.

This has become less of a problem in recent weeks, as Google have accelerated to an alleged 20 million users in less than a month. My immediate impression was that it was Skype, and not Facebook, that would have to worry, due to the very impressive webchat capability integrated to Google Plus. Of course, shortly afterwards, Skype announced integration with Facebook which leaves the fight wide open. I think the privacy features of Google Plus will help it win a lot of converts - while essentially all of the same options have been on Facebook for quite some time now, they are all squirrelled away; the same options on Google Plus are in full view, and almost more of an effort to not use. Easily delimiting where friends end and where family begins is an attractive feature for many people, including myself who, up to this point, has been operating on a no cross-generation Facebooking basis. Regardless of the outcome, it will be interesting to watch how the battle pans out. Google’s fights now span a wide range of frontiers, but they still seem confident that they have not yet bitten off more than they can chew.

Git

I’ve been using git for a while now. Indeed, it’s the only version control I’ve ever really used, but I was properly sold on it recently when I used it in a way that was completely novel for me, but extremely useful. Indeed, not even a way I had ever considered that it might be useful.

I was debugging some code at work, when I ran across what was the magic number 173.3 hard-coded with no associated comment (I know, I know). Naturally, I needed to know what this number was and why it was there - and if it was correct. So a quick use of git grep "173.3" $(git log -g --pretty=format:%h) gave me a list of revisions that the number appeared in. Looking up the commit date for the earliest revision using git log allowed me to quickly flick to the appropriately dated page in my lab book - where I found a derivation of the number in question.

It was correct, but saved me a large amount of time flicking through old lab books and wondering where it had come from.

Treasure Hunt

At short notice, for a friend’s birthday, I set up a treasure hunt around Cambridge. Sadly, I am not going to give details here, even though I’m incredibly proud of how good a treasure hunt I turned around in about 24 hours. This is because I will probably use many of its elements again in another treasure hunt for some other friends, who may all too easily stumble across this post.

It was universally enjoyed, however, to the extent that I got a round of applause from those involved at the end.

Less Computation, More Contemplation

The last couple of months have been fairly strenuous for me in most domains of life, but I think I’ve found my mojo again in the last week or two. This is something which I - and frankly, others - have sorely missed. My projects are interesting again, and I’ve reverted to my traditional volume of reading. Work is hard, although satisfying when it works - the trick is getting it to work often enough to not go mad, which I don’t think I’ve managed yet. Unfortunately, my work at the moment revolves around measuring a 30 nanometer movement on a mount for a five-inch optic. That’s a distance less than one-twentieth the wavelength of visible light on something that is comfortably larger than my cereal bowl.

It’s non-trivial.

I reckon the third year of a PhD must get easier though. Right…?

To date

2011-05-27T18:36:10Z

Another new site

I’ve maintained my website to some extent now for quite some time - even longer if you take my blog into account. I decided it was time for a change, and a little while ago I fell in love with Steve Losh’s website. This was for a number of reasons; the first is the sheer simplicity - it’s super easy to read. The second was for the cool header to the left of the text you see here. Have a little scroll up and down the page. Go ahead, I’ll wait.

Pretty great, huh? That said, I did change a few things to make it feel more like my own, like the splash page and the colours (see below).

A third reason I liked it is that it’s based on hyde and this allows me to easily unify the appearance of my site. For too long, cool projects like [the Lottery Frustrator][frustrator] or my Qwantz-Twitter mash-up have had inelegant and ugly HTML wrapping them, even thought they deserve much better. Something like Wordpress doesn’t really allow for easy unification of such pages, but hyde does, allowing you to build up a page out of building blocks that you already have, compiling it into HTML as a one-time event. My PHP files now just have these blocks in them, and are rendered by hyde upon deployment. I hope that all my old

The colours

I wasn’t a huge fan of Steve’s colour scheme - I thought that it was a bit pink.I think mine might be a touch too green, but it’s good enough to get the site up. For too long I’ve had it sitting on my hard drive, gently being tweaked while I tried small changes here and there. I’ve now got it up, facing the world, and I have to deal with it like that. I found Kuler very helpful on the final push to find a colour scheme that I liked.

Top and Tail

I like having defined headers and footers, which Steve didn’t have quite so much. My CSS was extremely rusty, so the basic layout you see before you is a) The product of many more hours of work than it should have been and b) not implemented in the most elegant fashion. If you are of a nervous disposition, I don’t recommend that you look at the nested elements I’ve invoked to get it to work.

Splash Page

Upon seeing my first take on the site, my girlfriend announced that it looked ‘boring’, and I can see where she was coming from. So I’ve removed the text and thrown in some images of things that I have done or am doing. Hopefully that’s much more enticing. I certainly can’t see anyone not being intrigued by the Old Spice Guy.

Writing

Part of the reason I was so enthralled by the scrolling header to the left is that it lends itself to a style of writing I’m trying to adopt on the encouragement of my PhD supervisor. Namely, build up from a framework of headers. Pad each of these out slowly, allowing the structure of your writing to build up as you get a better idea for what you want to tell your reader.

I gave it a shot for this post, and I think it worked out okay - but such a sprawling post is not really the best test case. More focussed posts in the future will form a better test, for sure.

Magic

About eighteen months ago, I made a cube out of commons and uncommons I had mostly lying around. My friends and I would occasionally draft from boosters we bought, and I thought this would be a good way to get more use out of the cards we had lying around, and sate our desire for a draft for free when we got the urge.

Unfortunately, this has spectacularly backfired if we are using ‘free’ as our metric of success. We’ve started going to prereleases again, and even building constructed decks. Happily, I’ve held off going completely mad on singles; this Standard season I’ve been running mono white Signal Quest, which was pleasantly cheap to build.

When the rotation happens, I’ll evaluate where I stand on the constructed front. I’ve always been a fan of Legacy, as it lets me play Enchantress (my all-time favourite constructed deck - drawing cards in green never gets old), which I already had mostly built from back in the day. My main issue with Standard is the cost of chase singles, but I could see myself keeping a cheap Standard deck to hand to be able to go with friends to game days and so forth. The recent announcement of modern has certainly captured my interest though, I must say…

PhD

The PhD can be honestly described as going. Going where, I’m not sure - but going it is. My second year report is due soon, which must include a thesis plan. To say that this is concentrating my mind at the moment is somewhat of an understatement.