Cryptocurrency: Yesterday, the bitcoin exchange MtGox – riddled by problems – issued a press release saying the bitcoin protocol was to blame for its ongoing problems. That statement, which caused the markets to nosedive temporarily, is outright false. The problem is, and was, bad code hygiene in the MtGox exchange itself. Here are the details.
Yesterday, when MtGox blamed “transaction malleability” as the cause of MtGox’ problems, implying that the problems at MtGox affected all exchanges and everything bitcoin, that was a sign of a very elastic relationship with facts. It’s true that transaction malleability was a factor, but not nearly in the way that MtGox implied. (We’ll be returning to what the “malleability” is.)
Here’s the real problem: MtGox is running its own homebuilt bitcoin software, and has not cared to update and upgrade that software along with the developments of the bitcoin protocol. Recently, after a very long grace period, the bitcoin protocol tightened slightly in order to disallow unnecessary information in transaction records, and did this to fix the malleability problem that MtGox blamed.
So the problem of malleability remained at MtGox, while having been fixed in the rest of the world. This – the discrepancy itself – was the root cause of the problem, because it meant that MtGox started issuing invalid transaction records for bitcoin withdrawals. Obviously, they were rejected by the bitcoin network.
Let me explain in a bit more detail.
When you write an amount of money, say twenty-three thousand four hundred and twenty-two dollars and fifty-four cents, you typically write that as $23,422.54. But it would also be valid to write it as $0,023,422.54. Or $0,000,023,422.54. This fact – that one number can be written in many ways, all valid – is the malleability. (For the sake of completeness, it wasn’t the amount, but another number in the transaction record that was concerned.)
This was tightened in the bitcoin protocol to only allow the shortest version of writing a number, $23,422.54, in this specific code change, which happened a whole year ago.
This change was ignored by MtGox, if I may speculate, probably because “it kept working anyway”. That is, until bitcoin 0.8, when the core developers decided to enforce this change across the protocol, having had the tightening published for over a year. The moment bitcoin 0.8+ gained majority deployment on the network, such invalid transactions started getting rejected.
In other words, MtGox’ lack of code hygiene and lack of very basic IT release processes led to the MtGox code getting out of sync with the bitcoin protocol itself. It kept writing numbers in a way that wasn’t always the shortest possible way in some of its transaction records, and therefore, the inevitable happened: those transaction records were rejected by the bitcoin network.
As a complete side note, this situation is well described by a saying in Sweden that we use to honor our neighboring Finns and their gung-ho attitude toward life, the universe, and everything. The saying is supposed to be pronounced slowly with a slight sauna-induced slur and a strong Finnish accent, like such:
http://falkvinge.net/files/
Now, let’s return to MtGox’ press release. There, they state that skilled hackers had the ability to rewrite bitcoin withdrawals with the speed of lightning before they reached the bitcoin network, implying that hackers changed valid transactions enroute. This, skilled hacking, was the cause of all their problems, they claimed. But that’s not what happened at all. MtGox were creating invalid transaction records for some small but significant portion of their bitcoin withdrawals.
What this means is that MtGox wasn’t the subject of some skilled hacking related to transaction malleability. Instead, bad code hygiene was causing MtGox to broadcast invalid transactions, which could trivially be corrected and re-broadcast, causing all these problems downstream.
This, in turn, leads to all the described problems with double-spending, internal databases of account records getting out of sync with the blockchain records, et cetera. Once somebody has corrected one of MtGox’ malformed transactions and re-broadcast it, MtGox would still consider it unsuccessful, making things go out of sync.
So, is this hard to do the right way? No. I can say that authoritatively – I spent seven years as a CTO-for-hire putting exactly these kinds of hygiene, accountability, trackability, and predictability processes into place at startups with growth pains, saving more than one startup from the blame-game death spiral. MtGox is dying from the lack of a very basic leadership and management toolkit.
Oh, and that Swedish saying about the Finns in the audio clip above? The one that references how the protocol strictness tightened but MtGox went gung-ho ahead anyway? It means “The road turned, but Pekka didn’t”.
The author is personally affected by the MtGox malfunction, having a five-figure dollar amount in stuck unprocessed bitcoin withdrawals.
As a final note, I can’t help feeling a bit of immature glee at all the doomsday sayers that screamed crash! all over the media, who seemed just waiting to pounce on the opportunity to declare Bitcoin dead. Uhm, yeah. It turns out that over the whole day of February 10, the Bitcoin price fell a total of 19 US cents. As of this writing (01:30 UTC on Feb 11), it’s up a bit (705) from where it opened yesterday (688).
Sources: this post by TheComputerScientist, this post by nullc (Greg Maxwell), and a few other sources whose identity I’ll protect.
