Antonio Garcia-Martinez’s blog post, Why Founding a Three Person Startup with Zero Revenue Is Better Than Working for Goldman Sachs is worth reading. One parenthetical comment, especially, rang true with me:
Regtests ran nightly, and no one could trade a model without thorough testing (that might sound like standard practice, but you have no idea how primitive the development culture is on the Street).
A couple of years ago I worked on a large infrastructure IT project for a big Wall Street bank — not Goldman. As part of the requirements gathering process, we talked to many of system’s customers, internal IT users. These users were very savvy, demanding customers in their own way: infrastructure mattered because it could never fail and never be fast enough. Although, like everyone else, they didn’t really care about their technology infrastructure, they were happy to spend money on faster processors and exotic networking technologies, because any advantage would make them money. Their servers were located physically adjacent to the trading systems they were talking to, to take advantage of the reduced distance that the light had to travel through space. They were what Sun used to call their ‘red shift’ customers, for whom IT really made a difference.
For all of this sophistication, though, they had incredibly primitive development processes.
The programmers at the bank we were working for were set up in small teams with traders, working on narrow segments of the financial markets: one group would trade Latin American fixed income futures, another on some aspect of spreads in forex between certain currency pairs. They wrote and maintained their own code, totally separate from anyone else’s. The coders and traders would work very closely together, competing against similar teams at other banks.
The scale was massive; they’d routinely be dealing with hundreds of millions of dollars and tens of thousands of messages per second. Everything was written in C to be as fast as possible and they would make updates and changes to the underlying logic almost daily, sometimes multiple times a day.
The programmers were typically Eastern European math and science types, nerdy at the coder/poet end of the IT spectrum, not nerdy in the sys admin/help desk end of the IT spectrum.
“So,” I asked in an interview, “what’s your dev and test procedure?” The Romanian mathematician didn’t know what I was talking about.
“I mean,” what do you do before you promote to production?”
“Nothing. We just run it.”
“Okay, but how do you send your updates to the servers running in the production environment?”
“Oh, we control those.”
I laughed, uneasily. “So,” I said, “you make changes directly to live production servers?”
“Yes,” he replied, confused. “Of course.”