- Email: [email protected]
Where have all the traders gone?
WOORBOOK STOCK
Where have all the traders gone?
42 | NewScientist | 2 June 2007
www.newscientist.com
Time is running out for Wall Street’s high rollers. A new breed of traders is muscling in, says Robert Matthews
●
BRAD BAILEY was visiting the trading floor of an investment bank in New York City when he first noticed it. As a former Wall Street trader, he should have felt at home amid all the screens, phones and bustle of billions of dollars in trades. But that was just it: there wasn’t any bustle. In fact, there were hardly any traders. “You could hear a pin drop,” he recalls. Then it sank in: machines had taken over the role of people and computer servers don’t make any noise. There’s a quiet revolution happening all over the financial world. Gone are the days of Gordon Gekko lookalikes screaming obscenities and dumping a loss-making stock onto an unsuspecting market. Investors have realised that the processing speed and sheer volume of trades a computer can make can help them to outwit the sharpest of dealers. As a result, they are investing heavily in what has fast become an arms race between investors. Their goal is to develop the best “algorithmic trading” systems – software that helps decide which trades are the most profitable, and then does the deals. Ten years ago, algo-trading was almost non-existent, but according to a recent report by Bailey, now at the Boston-based consulting firm Aite Group, one-third of all trading decisions in US markets are now made by machines. He predicts that by 2010 more than half will be done this way. At Deutsche Bank in London, over 70 per cent of a category of foreign currency trades, called “spot trades”, are now carried out without human intervention every day. All this will have an
As a result, investment houses are becoming increasingly tech-savvy. Eavesdrop on traders today and you are more likely to hear talk of “low latency access” (which we’ll get to later) than of what they’d like to do to a rival’s neck. “Anyone who’s been on Wall Street since the early 1990s will have had to reinvent themselves,” says Bailey. Back then, success as a trader hinged on an instinct for what the market was “thinking”, plus reactions fast enough to make the most of the opportunities the market presented. In the early years of computerised trading, when machines were simply communication tools, hitting a key a few tenths of a second faster than a competitor could make a real difference to your profit margin. Nowadays human traders struggle to keep up. “Silicon is taking over from carbon on Wall Street,” says Bailey, as beige boxes proliferate across trading floors. Computers have the edge over humans in many ways. Take something as simple as reaction time. When a human trader sees a stock change price, he might react in a few hundred milliseconds. A computerised trader is at least 10 times faster: depending on how much you are willing to fork out to speed things up. A few hundred milliseconds might seem insignificant, but if the price changes by a fraction of a per cent in the split-second before a trade worth many millions, it could mean a swing of tens of thousands of dollars. The key for any trader is “low latency” market access – that is, minimal delay between placing an order and seeing it fulfilled.
“70 per cent of foreign currency
trades are carried out without human intervention” impact on more than just high-rolling investors. Even if you don’t own any shares you can bet that millions of those owned by your pension fund are already being bought and sold using “algo” trading techniques. It’s not hard to see why algorithmic trading is so attractive. Machines can make multiple trades, monitor thousands of stocks and do it all at breakneck speed. Crucially, they can do it without anyone noticing. There are big profits to be made in buying and selling shares that other traders haven’t yet realised are being lucratively traded. The more discreetly you can do this – by spreading the deal over lots of small trades, for example – the less likely other traders are to wake up to the opportunity and dilute your profit potential. Such discretion is near impossible for a human, as it requires constant monitoring of the market to make sure your trades don’t alter stock prices in an unfavourable direction. www.newscientist.com
To achieve this, traders naturally have ultra-fast software, running on top-of-the-line computers with the very best processors and memory capacity. But there is a more direct, and perhaps less obvious, way to speed things up: moving closer to the source. Trading companies pay top dollar to snuggle their servers as close as possible to those of the stock exchange. With access to such “proximity servers”, their electrons can beat those of their rivals to the punch. Last year one of the biggest hitters in the algo-based trading world, Deutsche Bank, paid an undisclosed sum to proximity-server supplier BT Radianz to shave milliseconds off its trading times. Stealth-trading is another area in which machines have the upper hand. For example, many of the leading brokerage firms now have computers running so-called volumeweighted average price (VWAP) algorithms. These algos slice up big transactions, then 2 June 2007 | NewScientist | 43
limit the amount bought and sold in each trade to a specified proportion of the total value of trades in those stocks on that day. The idea is to avoid drawing other traders’ attention to what you are doing, and so prevent rivals getting in on the act. Or at least that’s how it used to be, until the Darwinian imperative kicked in. “At the start of the decade, if you used a simple slice-anddice program, you got a big improvement in trading performance,” says Richard Balarkas, an algorithmic-trading expert at Credit Suisse in London. Now those big improvements have evaporated. People started using monitoring algorithms to spot trades where, for example, someone was selling 5000 shares every 15 minutes, he says. “It’s what we call ‘signalling risk’, and it has become a real issue. Other people can make money from working out what you’re trying to do.”
As a result, the arms race has accelerated. Pattern-spotting software now looks for signs of algos trying to sneak their transactions onto the market, and studies the size of the transactions to make a guess at what’s going on. That in turn has led to the development of algos which create smokescreens by combining their transaction strategies with a small amount of randomness in, say, the timing of the trades. This particular arms race now seems to be heading towards some kind of stalemate, driven by the fact that there’s a limit to how small and apparently random a transaction can be. “It’s getting to the point where there is so much noise that it’s going to be hard to detect market opportunities,” says Balarkas. And that is pushing algo traders to find new ways of reaching their ultimate goal: “adding alpha”. Everyone has heard the stories of kids sticking pins in stock market listings and outperforming Wall Street’s finest. There is a solemn truth behind such anecdotes: investment managers who try to beat the market with their cunning usually fail. According to David McCraw of Aberdeen Asset Management, based in Edinburgh, UK, two out of three investment managers secured worse returns than those achieved by tracker funds that automatically follow the overall market. In the jargon, these managers fail to add “alpha” – roughly speaking, the measure of extra return on investment achieved, compared to a tracker. Being able to predict where the market is heading would be one way to beat the trackers – but that is easier said than done. “Any pension fund trader will have hundreds of trades to make, and there’s an awful lot of information to assimilate,” says Balarkas. “It’s 44 | NewScientist | 2 June 2007
MARK SEGAL/PANORAMIC IMAGES
Arms race
“Traders now venture far
from the beaten path in search of ‘dark pools of liquidity’ ” really impossible to expect one person to analyse it all equipped with just two optical receptors, two aural receptors and 10 digits.” Human traders are increasingly relying on computerised sidekicks to boil all the data down to a “buy” or “sell” punchline. Another strategy for stealing a march on trackers is to look for signs of “liquidity” – the presence of buyers and sellers for particular stocks, which opens up a chance to trade. As before, there’s scope for cunning. Trade too eagerly, and you could ruin the price, wait too long and the window of opportunity will slam shut as others beat you to it. This has led to the emergence of “sniper” algos, which wait for a suitable buyer or seller to emerge and then conduct the trade as fast as possible, before the price can be affected by other traders. The obvious place to find such liquidity is in the big stock markets of London and New York. Looking in the obvious places, however,
is no longer good enough for big-alpha hunters, algo-wielding traders now venture far from the beaten path, in search of “dark pools of liquidity”. These are groups of buyers and sellers trading specific stocks in markets outside the mainstream exchanges. “If the big stock markets are the equivalent of regular stores, these dark pools are like eBay,” says Bailey. “Around 15 per cent of US market liquidity is hidden away in them.” Algos are being designed that go “fishing” in the dark pools, dangling small numbers of shares in the markets like bait. The speed with which they get snapped up indicates the liquidity level and the likelihood of profitable trading. As with much else in algo-trading, implementing this simple idea is getting harder all the time as the competition between financial institutions for custom increases with Darwinian relentlessness. “As soon as one bank has a system significantly www.newscientist.com
J>;H?I;E<J>;C79>?D;I
profit. In the years since, competition has forced such systems to become far more sophisticated – and far more complex. Cliff’s original algo was tuned with eight parameters to capture the essence of market conditions. In its latest manifestation, it must initially be fed with no fewer than 60 parameters, which are then tweaked by Cliff’s software until they reach values that give the highest chances of making a profit. Finding the right combination is a monumental task. And speed is everything.
Secret ingredients
better than the others, its rivals come out with one themselves,” says Dave Cliff, a computer scientist at Southampton University and founder of Syritta, a UK-based consultancy firm that develops algo-trading software. Banks are finally realising it’s the only way to preserve their profit margins. Some banks are trying to protect their algos using patent and copyright laws, but they can do little to hold back the real driving force for change: the continued battle for supremacy in which only the fittest algos survive. So instead of trying to deny the inevitable, some programmers are turning to evolutionary concepts for inspiration in designing new algos. Cliff is one of them. In the mid-1990s, he designed one of the first commercially successful algos. It didn’t make the decision about when to buy or sell, but in experiments where human traders were pitted against his system it turned a bigger www.newscientist.com
Cliff turned to so-called “genetic algorithms” for help. His new system takes an initial set of guesses about the optimal selection of market parameters, tests how well each parameter describes prevailing market conditions, and then “breeds” a new selection from these to arrive at a more effective set. This evolutionary cycle is repeated until optimum values for the parameters are reached which the algo then uses to trade with. Such optimisation methods are not new (New Scientist, 3 December 1994, p 25), but like real evolution they need time to work their magic – and time is one luxury no trader has. So to speed up the search for the best parameters, Cliff has added some secret ingredients to Darwin’s basic recipe. “The problem with throwing away most of the parameter values after trying them out is that you’re also throwing away a lot of information,” he says. Instead, he keeps all the different parameter values and examines them, looking for clues as to why some work better than others. On some days it turns out that only a handful of the 60 values are really needed to get the algo to work well. All this requires truckloads of computing power and investment banks just cannot get enough of it. Much of it is hired from companies offering metered access to giant computer clusters, but in many banks it’s not unusual to see desktop PCs pressed into service, running trading analysis programs whenever they fall idle. For the company’s most treasured algorithm processing, the number-crunching is done on
+& *& )& IEKH9;07?J;=HEKF"(&&,
Scenes like this, where hundreds of people crowd trading room floors, could soon become history
F[hY[djW][e\\_dWdY_Wb[nY^Wd][i f[h\ehc[Zki_d]Wb]eh_j^c_YjhWZ_d]
Ceh[WdZceh[_dl[ijehiedj^[KI\_dWdY_Wb[nY^Wd][iWh[ h[bo_d]edWb]eh_j^c_YjhWZ_d]ie\jmWh[jej[bbj^[cm^[djeXko WdZi[bbj^[_hijeYaiWdZi^Wh[i
(& '& &
dedicated high-security servers, away from prying eyes. Some investment banks have whole floors of their most expensive real estate packed with humming beige boxes. Despite the onward march of algo-trading, human traders still have a role. While computers are increasingly doing the breadand-butter trades, the Darwinian quest for alpha is taking trading firms ever further from the conventional markets, into territory where even the beige boxes can stumble and fall. Among these are “illiquid” markets – the flip side of the dark pools of liquidity. The relatively few buyers and sellers in these markets mean whole hours can pass with nothing much happening, and this causes all sorts of problems for algo trading. “The statistical models of trading patterns fail to give any meaningful results when a stock may only trade 10 times a day,” says Bruce Bland, head of algorithmic research at trading technology supplier Fidessa. Human traders can make up for the lack of data with instinct and experience, and hooking human instinct up to computing power is now at the leading edge of algo trading. The result is software that helps the trader come up with ideas for bagging some alpha, and tests those ideas in simulations to see if they’ll fly. With so many variables, it’s easy to make mistakes, but the computer can spot them before unleashing the algo upon the market. “It’s rather like the fly-by-wire systems used by the latest fighter aircraft,” Bland explains. “The pilot can only instruct the plane to perform manoeuvres that the frame and wings will tolerate.” Balarkas says human traders will still have plenty to keep them occupied, for the time being at least. “People who think computers are going to put them out of business really don’t understand traders,” he says. “We don’t have algos that predict where the market is going – we have people, and they are still much better at it.” The question is, for how much longer? ● Robert Matthews is visiting reader in science at Aston University, Birmingham, UK 2 June 2007 | NewScientist | 45
Where have all the traders gone?
42 | NewScientist | 2 June 2007
www.newscientist.com
Time is running out for Wall Street’s high rollers. A new breed of traders is muscling in, says Robert Matthews
●
BRAD BAILEY was visiting the trading floor of an investment bank in New York City when he first noticed it. As a former Wall Street trader, he should have felt at home amid all the screens, phones and bustle of billions of dollars in trades. But that was just it: there wasn’t any bustle. In fact, there were hardly any traders. “You could hear a pin drop,” he recalls. Then it sank in: machines had taken over the role of people and computer servers don’t make any noise. There’s a quiet revolution happening all over the financial world. Gone are the days of Gordon Gekko lookalikes screaming obscenities and dumping a loss-making stock onto an unsuspecting market. Investors have realised that the processing speed and sheer volume of trades a computer can make can help them to outwit the sharpest of dealers. As a result, they are investing heavily in what has fast become an arms race between investors. Their goal is to develop the best “algorithmic trading” systems – software that helps decide which trades are the most profitable, and then does the deals. Ten years ago, algo-trading was almost non-existent, but according to a recent report by Bailey, now at the Boston-based consulting firm Aite Group, one-third of all trading decisions in US markets are now made by machines. He predicts that by 2010 more than half will be done this way. At Deutsche Bank in London, over 70 per cent of a category of foreign currency trades, called “spot trades”, are now carried out without human intervention every day. All this will have an
As a result, investment houses are becoming increasingly tech-savvy. Eavesdrop on traders today and you are more likely to hear talk of “low latency access” (which we’ll get to later) than of what they’d like to do to a rival’s neck. “Anyone who’s been on Wall Street since the early 1990s will have had to reinvent themselves,” says Bailey. Back then, success as a trader hinged on an instinct for what the market was “thinking”, plus reactions fast enough to make the most of the opportunities the market presented. In the early years of computerised trading, when machines were simply communication tools, hitting a key a few tenths of a second faster than a competitor could make a real difference to your profit margin. Nowadays human traders struggle to keep up. “Silicon is taking over from carbon on Wall Street,” says Bailey, as beige boxes proliferate across trading floors. Computers have the edge over humans in many ways. Take something as simple as reaction time. When a human trader sees a stock change price, he might react in a few hundred milliseconds. A computerised trader is at least 10 times faster: depending on how much you are willing to fork out to speed things up. A few hundred milliseconds might seem insignificant, but if the price changes by a fraction of a per cent in the split-second before a trade worth many millions, it could mean a swing of tens of thousands of dollars. The key for any trader is “low latency” market access – that is, minimal delay between placing an order and seeing it fulfilled.
“70 per cent of foreign currency
trades are carried out without human intervention” impact on more than just high-rolling investors. Even if you don’t own any shares you can bet that millions of those owned by your pension fund are already being bought and sold using “algo” trading techniques. It’s not hard to see why algorithmic trading is so attractive. Machines can make multiple trades, monitor thousands of stocks and do it all at breakneck speed. Crucially, they can do it without anyone noticing. There are big profits to be made in buying and selling shares that other traders haven’t yet realised are being lucratively traded. The more discreetly you can do this – by spreading the deal over lots of small trades, for example – the less likely other traders are to wake up to the opportunity and dilute your profit potential. Such discretion is near impossible for a human, as it requires constant monitoring of the market to make sure your trades don’t alter stock prices in an unfavourable direction. www.newscientist.com
To achieve this, traders naturally have ultra-fast software, running on top-of-the-line computers with the very best processors and memory capacity. But there is a more direct, and perhaps less obvious, way to speed things up: moving closer to the source. Trading companies pay top dollar to snuggle their servers as close as possible to those of the stock exchange. With access to such “proximity servers”, their electrons can beat those of their rivals to the punch. Last year one of the biggest hitters in the algo-based trading world, Deutsche Bank, paid an undisclosed sum to proximity-server supplier BT Radianz to shave milliseconds off its trading times. Stealth-trading is another area in which machines have the upper hand. For example, many of the leading brokerage firms now have computers running so-called volumeweighted average price (VWAP) algorithms. These algos slice up big transactions, then 2 June 2007 | NewScientist | 43
limit the amount bought and sold in each trade to a specified proportion of the total value of trades in those stocks on that day. The idea is to avoid drawing other traders’ attention to what you are doing, and so prevent rivals getting in on the act. Or at least that’s how it used to be, until the Darwinian imperative kicked in. “At the start of the decade, if you used a simple slice-anddice program, you got a big improvement in trading performance,” says Richard Balarkas, an algorithmic-trading expert at Credit Suisse in London. Now those big improvements have evaporated. People started using monitoring algorithms to spot trades where, for example, someone was selling 5000 shares every 15 minutes, he says. “It’s what we call ‘signalling risk’, and it has become a real issue. Other people can make money from working out what you’re trying to do.”
As a result, the arms race has accelerated. Pattern-spotting software now looks for signs of algos trying to sneak their transactions onto the market, and studies the size of the transactions to make a guess at what’s going on. That in turn has led to the development of algos which create smokescreens by combining their transaction strategies with a small amount of randomness in, say, the timing of the trades. This particular arms race now seems to be heading towards some kind of stalemate, driven by the fact that there’s a limit to how small and apparently random a transaction can be. “It’s getting to the point where there is so much noise that it’s going to be hard to detect market opportunities,” says Balarkas. And that is pushing algo traders to find new ways of reaching their ultimate goal: “adding alpha”. Everyone has heard the stories of kids sticking pins in stock market listings and outperforming Wall Street’s finest. There is a solemn truth behind such anecdotes: investment managers who try to beat the market with their cunning usually fail. According to David McCraw of Aberdeen Asset Management, based in Edinburgh, UK, two out of three investment managers secured worse returns than those achieved by tracker funds that automatically follow the overall market. In the jargon, these managers fail to add “alpha” – roughly speaking, the measure of extra return on investment achieved, compared to a tracker. Being able to predict where the market is heading would be one way to beat the trackers – but that is easier said than done. “Any pension fund trader will have hundreds of trades to make, and there’s an awful lot of information to assimilate,” says Balarkas. “It’s 44 | NewScientist | 2 June 2007
MARK SEGAL/PANORAMIC IMAGES
Arms race
“Traders now venture far
from the beaten path in search of ‘dark pools of liquidity’ ” really impossible to expect one person to analyse it all equipped with just two optical receptors, two aural receptors and 10 digits.” Human traders are increasingly relying on computerised sidekicks to boil all the data down to a “buy” or “sell” punchline. Another strategy for stealing a march on trackers is to look for signs of “liquidity” – the presence of buyers and sellers for particular stocks, which opens up a chance to trade. As before, there’s scope for cunning. Trade too eagerly, and you could ruin the price, wait too long and the window of opportunity will slam shut as others beat you to it. This has led to the emergence of “sniper” algos, which wait for a suitable buyer or seller to emerge and then conduct the trade as fast as possible, before the price can be affected by other traders. The obvious place to find such liquidity is in the big stock markets of London and New York. Looking in the obvious places, however,
is no longer good enough for big-alpha hunters, algo-wielding traders now venture far from the beaten path, in search of “dark pools of liquidity”. These are groups of buyers and sellers trading specific stocks in markets outside the mainstream exchanges. “If the big stock markets are the equivalent of regular stores, these dark pools are like eBay,” says Bailey. “Around 15 per cent of US market liquidity is hidden away in them.” Algos are being designed that go “fishing” in the dark pools, dangling small numbers of shares in the markets like bait. The speed with which they get snapped up indicates the liquidity level and the likelihood of profitable trading. As with much else in algo-trading, implementing this simple idea is getting harder all the time as the competition between financial institutions for custom increases with Darwinian relentlessness. “As soon as one bank has a system significantly www.newscientist.com
J>;H?I;E<J>;C79>?D;I
profit. In the years since, competition has forced such systems to become far more sophisticated – and far more complex. Cliff’s original algo was tuned with eight parameters to capture the essence of market conditions. In its latest manifestation, it must initially be fed with no fewer than 60 parameters, which are then tweaked by Cliff’s software until they reach values that give the highest chances of making a profit. Finding the right combination is a monumental task. And speed is everything.
Secret ingredients
better than the others, its rivals come out with one themselves,” says Dave Cliff, a computer scientist at Southampton University and founder of Syritta, a UK-based consultancy firm that develops algo-trading software. Banks are finally realising it’s the only way to preserve their profit margins. Some banks are trying to protect their algos using patent and copyright laws, but they can do little to hold back the real driving force for change: the continued battle for supremacy in which only the fittest algos survive. So instead of trying to deny the inevitable, some programmers are turning to evolutionary concepts for inspiration in designing new algos. Cliff is one of them. In the mid-1990s, he designed one of the first commercially successful algos. It didn’t make the decision about when to buy or sell, but in experiments where human traders were pitted against his system it turned a bigger www.newscientist.com
Cliff turned to so-called “genetic algorithms” for help. His new system takes an initial set of guesses about the optimal selection of market parameters, tests how well each parameter describes prevailing market conditions, and then “breeds” a new selection from these to arrive at a more effective set. This evolutionary cycle is repeated until optimum values for the parameters are reached which the algo then uses to trade with. Such optimisation methods are not new (New Scientist, 3 December 1994, p 25), but like real evolution they need time to work their magic – and time is one luxury no trader has. So to speed up the search for the best parameters, Cliff has added some secret ingredients to Darwin’s basic recipe. “The problem with throwing away most of the parameter values after trying them out is that you’re also throwing away a lot of information,” he says. Instead, he keeps all the different parameter values and examines them, looking for clues as to why some work better than others. On some days it turns out that only a handful of the 60 values are really needed to get the algo to work well. All this requires truckloads of computing power and investment banks just cannot get enough of it. Much of it is hired from companies offering metered access to giant computer clusters, but in many banks it’s not unusual to see desktop PCs pressed into service, running trading analysis programs whenever they fall idle. For the company’s most treasured algorithm processing, the number-crunching is done on
+& *& )& IEKH9;07?J;=HEKF"(&&,
Scenes like this, where hundreds of people crowd trading room floors, could soon become history
F[hY[djW][e\\_dWdY_Wb[nY^Wd][i f[h\ehc[Zki_d]Wb]eh_j^c_YjhWZ_d]
Ceh[WdZceh[_dl[ijehiedj^[KI\_dWdY_Wb[nY^Wd][iWh[ h[bo_d]edWb]eh_j^c_YjhWZ_d]ie\jmWh[jej[bbj^[cm^[djeXko WdZi[bbj^[_hijeYaiWdZi^Wh[i
(& '& &
dedicated high-security servers, away from prying eyes. Some investment banks have whole floors of their most expensive real estate packed with humming beige boxes. Despite the onward march of algo-trading, human traders still have a role. While computers are increasingly doing the breadand-butter trades, the Darwinian quest for alpha is taking trading firms ever further from the conventional markets, into territory where even the beige boxes can stumble and fall. Among these are “illiquid” markets – the flip side of the dark pools of liquidity. The relatively few buyers and sellers in these markets mean whole hours can pass with nothing much happening, and this causes all sorts of problems for algo trading. “The statistical models of trading patterns fail to give any meaningful results when a stock may only trade 10 times a day,” says Bruce Bland, head of algorithmic research at trading technology supplier Fidessa. Human traders can make up for the lack of data with instinct and experience, and hooking human instinct up to computing power is now at the leading edge of algo trading. The result is software that helps the trader come up with ideas for bagging some alpha, and tests those ideas in simulations to see if they’ll fly. With so many variables, it’s easy to make mistakes, but the computer can spot them before unleashing the algo upon the market. “It’s rather like the fly-by-wire systems used by the latest fighter aircraft,” Bland explains. “The pilot can only instruct the plane to perform manoeuvres that the frame and wings will tolerate.” Balarkas says human traders will still have plenty to keep them occupied, for the time being at least. “People who think computers are going to put them out of business really don’t understand traders,” he says. “We don’t have algos that predict where the market is going – we have people, and they are still much better at it.” The question is, for how much longer? ● Robert Matthews is visiting reader in science at Aston University, Birmingham, UK 2 June 2007 | NewScientist | 45