- Email: [email protected]
Let the mind games begin
Special report | Home entertainment
Let the mind games begin Video games are about to go mental, as Duncan Graham-Rowe reports
●
40 | NewScientist | 15 March 2008
TIM MARRS
TWO players sit across a table from one another, staring at a small white ball on a track between them. Both are wearing headbands and concentrating, trying to nudge the ball towards their opponent. All they can use is the power of thought. This is Mindball, an addictive “mind game” in which the winning strategy is to remain as focused and relaxed as possible in the heat of battle. The ball rolls away from the player with the calmest mind, as measured by sensors on their headbands. The sensors are similar to those in an electroencephalogram (EEG), which probes brain activity by detecting “brainwaves” – tiny electrical currents playing across the scalp. Because EEGs are a non-invasive and nearinstantaneous way to read brain activity, they have long been touted as potentially useful in gaming. It now looks as if that promise will be fulfilled, and not just in Mindball. Several companies are developing hardware and software which they claim can detect brainwaves and use them in video games. If all goes to plan, the first of a new generation of games with mind control as a central feature will hit the high street this year. Mind gaming has its roots in the way new games are tested. Since 2004, EmSense, a company based in Monterey, California, has been using biofeedback to help game designers evaluate new products. Testers play a game wearing EmSense’s headset, which uses an EEG to record their brainwaves, and also measures their heart rate and the sweatiness of their skin. EmSense then builds up a blow-by-blow profile of the player’s emotional state and levels of arousal during play so the game can be made more engaging. www.newscientist.com
The basic technologies inside EmSense’s headset are nothing new. Neurologists have been using the EEG as a diagnostic tool for nearly a century, while measuring heart rate with an electrocardiogram (ECG) has an even longer pedigree. Galvanic skin response (GSR), which measures emotional arousal via the conductivity of the skin – a proxy for sweatiness – has been a central element of lie detectors since the first world war. Now, though, developers are finding ways to go beyond merely improving traditional games, and incorporating biofeedback into the games themselves. One of the leaders in the field is Emotiv of San Francisco. It has developed a headset with 16 sensors that it says allows players to control aspects of a game simply by thinking about them: concentrating on an on-screen object, for example, might allow their avatar to pick it up and move it around. Similarly, NeuroSky of San Jose, California, has developed a headset which chief executive Stanley Yang says can tell whether you are focused, challenged, relaxed, afraid, anxious and so on using a single sensor held against the temple. Developers have been trying to incorporate biofeedback into gaming for years. In 1984, Atari experimented with a headband called MindLink, which used electromyographic (EMG) sensors to detect muscle movements, allowing players to move an on-screen cursor with a frown or a raised eyebrow. In 1998, AmTex developed a game called Bio Tetris for the Nintendo 64. A heart-rate sensor clipped to the player’s ear lobe allowed players to slow the speed at which the Tetris blocks fell by remaining calm. Neither took off. So what’s different this time? One factor is the dazzling success of Nintendo’s Wii – 20 million consoles sold and rising. It dispenses with the traditional joystick and instead uses gestures to control the game via a hand-held wireless motion sensor. Its success has made it clear that people are ready for new ways to interact with games. Another factor is that the core technology is different, though whether it works is another matter. On the surface the claims seem plausible. Neurologists have long known how to read emotional states off an EEG, and Mindball apparently picks up alpha waves – a hallmark of mental calmness. But there are also many reasons to be sceptical. Where neurologists use as many as 120 EEG sensors all over the scalp, gaming headsets have just a www.newscientist.com
handful – or, in NeuroSky’s case, just one. The headsets don’t use the sticky conductive gel that medical EEGs need to transmit the signal from the scalp to the electrode. On top of that, EEGs are notoriously “noisy” – prone to interference from nearby electrical devices as well as the electrical activity produced by muscles, especially the heart. Even blinking
“Several firms say
their software can detect brainwaves and use them in gaming” can play havoc with an EEG signal. So how do they do it? Although the firms are cagey about how exactly their technologies work, there are a few details to go on. The number of electrodes seems to come down to a question of resolution. Medical and researchgrade EEGs need to be sensitive enough to detect subtle signals amid a chorus of electrical brain activity. For gaming, the chorus itself is sufficient. “We can’t achieve the same resolution as medical EEGs, but it’s enough to
detect the basic brainwaves,” Yang says. Dealing with interference is another matter and perhaps the fledgling industry’s biggest challenge. EMG signals, produced by muscle activity, are a particular problem because they can be an order of magnitude bigger than those produced by the brain, says Desney Tan, a researcher at Microsoft in Redmond, Washington, who has worked on diagnosing cognitive states using EEGs. Yang agrees that this is a challenge, but says the trick is to develop software that can recognise and filter out unwanted signals. “Our core technology is filtering,” he says. In any case, it may not be necessary to filter out all EMG signals, says Tan. Some could be turned to the developers’ advantage, as there is a strong correspondence between involuntary facial muscle contractions and your cognitive states and emotions, he says. So EMG signals can be used to supplement the EEG. In fact, the companies already use other bio-information from ECGs and GSR. Heart rate and sweating are both good measures of how physically and mentally aroused someone is. The trick is to combine all the measurements to get an overall sense of the player’s mental and physical state. “Each of these sensors gives us a little piece of the
Action packed Electrical activity isn’t the only way to read a gamer’s mind EEG is just one form of braincomputer interface that the games industry is toying with. The other is optical topography, which overcomes some of the problems with EEG. Though EEG provides lightningquick read-outs – perfect for fast-reaction games – speed comes at the cost of resolution. This makes some experts doubt its usefulness in gaming. An EEG does little more than tell you the general area that brain activity originated in, says John-Dylan Haynes of the Bernstein Center for Computational Neuroscience in Berlin, Germany. At the other end of the spectrum is functional magnetic
resonance imaging (fMRI), which provides readings accurate to down to a millimetre. For gaming, however, fMRI has crippling disadvantages: a price tag in the millions, bulky equipment and the fact that it takes at least 10 seconds to collect each data point. Optical topography offers a compromise. It relies on the fact that oxygenated blood absorbs more infrared radiation than deoxygenated blood. When the brain is busy, the body responds by pumping oxygen to active regions. Optical topography uses near infrared spectroscopy (NIRS) to monitor changes in blood oxygenation, measuring brain activity by proxy.
Technology companies are beginning to see the advantages. Hitachi, for example, showcased an optical topography headset in London last month. The headset can distinguish between an active and a resting brain, and is mobile enough to tempt game developers. “Our wireless headset can be worn without interfering with other activities,” says Atsushi Maki of the Hitachi Advanced Research Laboratory in Hatoyama, Japan. Maki thinks that the future might be a combination device. “Light doesn’t interfere with electric fields,” he says. “You could develop a headset that combines NIRS with EEG to give complementary readings.” Colin Barras
15 March 2008 | NewScientist | 41
Cover story | Home entertainment
Web 3.0 What will the next era of web culture bring? Annalee Newitz has some predictions
TIM MARRS
picture,” Tan says. Using slight variations of this approach NeuroSky and its competitors claim to have cracked it. But even if the technology works as advertised, there’s no guarantee of success in a competitive games market. How do these companies intend to succeed where others failed? One answer is by following Nintendo’s example with Wii. MindLink and Bio Tetris failed in part because they didn’t make the most of their novel interface. The games were really no different from what was already out there. “With the Wii, Nintendo did something right in designing a suite of tailored games,” Tan says. Wii games offer features that are not possible with a regular joystick, such as swinging the controller like a tennis racquet or brandishing it like a sword. The mind-game companies intend to emulate this, though without abandoning the traditional controller altogether. Their games will still be largely controlled by hand, with biofeedback offering additional features. For example, Emotiv has adapted a game based on the Harry Potter books so that players can lift boulders and throw thunderbolts just by concentrating on making it happen. Whatever form biofeedback games take, the world is ready for them, says Kiel Gilleade a computer games researcher at Lancaster University in the UK. The current market is less interested in finding new game genres than in looking for new hardware to enhance the gaming experience, he says. Not everyone is convinced. Michael Zyda, director of the University of Southern California’s GamePipe Laboratory in Marina del Rey, says biofeedback seems to work as an evaluation tool but he believes not enough research has been done to confirm its reliability in the real world. Hans Lee, head of technology at EmSense, agrees that more work is needed. One outstanding problem, he says, is that the hardware and software don’t work for everyone. Reading emotional and cognitive states reliably is difficult because of each individual’s variation in brain activity. Even so, at least one company believes the technology is ready. Emotiv says its headsets will be on the shelves later this year, alongside a suite of biofeedback games developed by its partners. Biofeedback has been talked about in video gaming for years, but the real quest for hearts and minds begins here. ● Duncan Graham-Rowe is a writer based in Brighton, UK 42 | NewScientist | 15 March 2008
www.newscientist.com
Let the mind games begin Video games are about to go mental, as Duncan Graham-Rowe reports
●
40 | NewScientist | 15 March 2008
TIM MARRS
TWO players sit across a table from one another, staring at a small white ball on a track between them. Both are wearing headbands and concentrating, trying to nudge the ball towards their opponent. All they can use is the power of thought. This is Mindball, an addictive “mind game” in which the winning strategy is to remain as focused and relaxed as possible in the heat of battle. The ball rolls away from the player with the calmest mind, as measured by sensors on their headbands. The sensors are similar to those in an electroencephalogram (EEG), which probes brain activity by detecting “brainwaves” – tiny electrical currents playing across the scalp. Because EEGs are a non-invasive and nearinstantaneous way to read brain activity, they have long been touted as potentially useful in gaming. It now looks as if that promise will be fulfilled, and not just in Mindball. Several companies are developing hardware and software which they claim can detect brainwaves and use them in video games. If all goes to plan, the first of a new generation of games with mind control as a central feature will hit the high street this year. Mind gaming has its roots in the way new games are tested. Since 2004, EmSense, a company based in Monterey, California, has been using biofeedback to help game designers evaluate new products. Testers play a game wearing EmSense’s headset, which uses an EEG to record their brainwaves, and also measures their heart rate and the sweatiness of their skin. EmSense then builds up a blow-by-blow profile of the player’s emotional state and levels of arousal during play so the game can be made more engaging. www.newscientist.com
The basic technologies inside EmSense’s headset are nothing new. Neurologists have been using the EEG as a diagnostic tool for nearly a century, while measuring heart rate with an electrocardiogram (ECG) has an even longer pedigree. Galvanic skin response (GSR), which measures emotional arousal via the conductivity of the skin – a proxy for sweatiness – has been a central element of lie detectors since the first world war. Now, though, developers are finding ways to go beyond merely improving traditional games, and incorporating biofeedback into the games themselves. One of the leaders in the field is Emotiv of San Francisco. It has developed a headset with 16 sensors that it says allows players to control aspects of a game simply by thinking about them: concentrating on an on-screen object, for example, might allow their avatar to pick it up and move it around. Similarly, NeuroSky of San Jose, California, has developed a headset which chief executive Stanley Yang says can tell whether you are focused, challenged, relaxed, afraid, anxious and so on using a single sensor held against the temple. Developers have been trying to incorporate biofeedback into gaming for years. In 1984, Atari experimented with a headband called MindLink, which used electromyographic (EMG) sensors to detect muscle movements, allowing players to move an on-screen cursor with a frown or a raised eyebrow. In 1998, AmTex developed a game called Bio Tetris for the Nintendo 64. A heart-rate sensor clipped to the player’s ear lobe allowed players to slow the speed at which the Tetris blocks fell by remaining calm. Neither took off. So what’s different this time? One factor is the dazzling success of Nintendo’s Wii – 20 million consoles sold and rising. It dispenses with the traditional joystick and instead uses gestures to control the game via a hand-held wireless motion sensor. Its success has made it clear that people are ready for new ways to interact with games. Another factor is that the core technology is different, though whether it works is another matter. On the surface the claims seem plausible. Neurologists have long known how to read emotional states off an EEG, and Mindball apparently picks up alpha waves – a hallmark of mental calmness. But there are also many reasons to be sceptical. Where neurologists use as many as 120 EEG sensors all over the scalp, gaming headsets have just a www.newscientist.com
handful – or, in NeuroSky’s case, just one. The headsets don’t use the sticky conductive gel that medical EEGs need to transmit the signal from the scalp to the electrode. On top of that, EEGs are notoriously “noisy” – prone to interference from nearby electrical devices as well as the electrical activity produced by muscles, especially the heart. Even blinking
“Several firms say
their software can detect brainwaves and use them in gaming” can play havoc with an EEG signal. So how do they do it? Although the firms are cagey about how exactly their technologies work, there are a few details to go on. The number of electrodes seems to come down to a question of resolution. Medical and researchgrade EEGs need to be sensitive enough to detect subtle signals amid a chorus of electrical brain activity. For gaming, the chorus itself is sufficient. “We can’t achieve the same resolution as medical EEGs, but it’s enough to
detect the basic brainwaves,” Yang says. Dealing with interference is another matter and perhaps the fledgling industry’s biggest challenge. EMG signals, produced by muscle activity, are a particular problem because they can be an order of magnitude bigger than those produced by the brain, says Desney Tan, a researcher at Microsoft in Redmond, Washington, who has worked on diagnosing cognitive states using EEGs. Yang agrees that this is a challenge, but says the trick is to develop software that can recognise and filter out unwanted signals. “Our core technology is filtering,” he says. In any case, it may not be necessary to filter out all EMG signals, says Tan. Some could be turned to the developers’ advantage, as there is a strong correspondence between involuntary facial muscle contractions and your cognitive states and emotions, he says. So EMG signals can be used to supplement the EEG. In fact, the companies already use other bio-information from ECGs and GSR. Heart rate and sweating are both good measures of how physically and mentally aroused someone is. The trick is to combine all the measurements to get an overall sense of the player’s mental and physical state. “Each of these sensors gives us a little piece of the
Action packed Electrical activity isn’t the only way to read a gamer’s mind EEG is just one form of braincomputer interface that the games industry is toying with. The other is optical topography, which overcomes some of the problems with EEG. Though EEG provides lightningquick read-outs – perfect for fast-reaction games – speed comes at the cost of resolution. This makes some experts doubt its usefulness in gaming. An EEG does little more than tell you the general area that brain activity originated in, says John-Dylan Haynes of the Bernstein Center for Computational Neuroscience in Berlin, Germany. At the other end of the spectrum is functional magnetic
resonance imaging (fMRI), which provides readings accurate to down to a millimetre. For gaming, however, fMRI has crippling disadvantages: a price tag in the millions, bulky equipment and the fact that it takes at least 10 seconds to collect each data point. Optical topography offers a compromise. It relies on the fact that oxygenated blood absorbs more infrared radiation than deoxygenated blood. When the brain is busy, the body responds by pumping oxygen to active regions. Optical topography uses near infrared spectroscopy (NIRS) to monitor changes in blood oxygenation, measuring brain activity by proxy.
Technology companies are beginning to see the advantages. Hitachi, for example, showcased an optical topography headset in London last month. The headset can distinguish between an active and a resting brain, and is mobile enough to tempt game developers. “Our wireless headset can be worn without interfering with other activities,” says Atsushi Maki of the Hitachi Advanced Research Laboratory in Hatoyama, Japan. Maki thinks that the future might be a combination device. “Light doesn’t interfere with electric fields,” he says. “You could develop a headset that combines NIRS with EEG to give complementary readings.” Colin Barras
15 March 2008 | NewScientist | 41
Cover story | Home entertainment
Web 3.0 What will the next era of web culture bring? Annalee Newitz has some predictions
TIM MARRS
picture,” Tan says. Using slight variations of this approach NeuroSky and its competitors claim to have cracked it. But even if the technology works as advertised, there’s no guarantee of success in a competitive games market. How do these companies intend to succeed where others failed? One answer is by following Nintendo’s example with Wii. MindLink and Bio Tetris failed in part because they didn’t make the most of their novel interface. The games were really no different from what was already out there. “With the Wii, Nintendo did something right in designing a suite of tailored games,” Tan says. Wii games offer features that are not possible with a regular joystick, such as swinging the controller like a tennis racquet or brandishing it like a sword. The mind-game companies intend to emulate this, though without abandoning the traditional controller altogether. Their games will still be largely controlled by hand, with biofeedback offering additional features. For example, Emotiv has adapted a game based on the Harry Potter books so that players can lift boulders and throw thunderbolts just by concentrating on making it happen. Whatever form biofeedback games take, the world is ready for them, says Kiel Gilleade a computer games researcher at Lancaster University in the UK. The current market is less interested in finding new game genres than in looking for new hardware to enhance the gaming experience, he says. Not everyone is convinced. Michael Zyda, director of the University of Southern California’s GamePipe Laboratory in Marina del Rey, says biofeedback seems to work as an evaluation tool but he believes not enough research has been done to confirm its reliability in the real world. Hans Lee, head of technology at EmSense, agrees that more work is needed. One outstanding problem, he says, is that the hardware and software don’t work for everyone. Reading emotional and cognitive states reliably is difficult because of each individual’s variation in brain activity. Even so, at least one company believes the technology is ready. Emotiv says its headsets will be on the shelves later this year, alongside a suite of biofeedback games developed by its partners. Biofeedback has been talked about in video gaming for years, but the real quest for hearts and minds begins here. ● Duncan Graham-Rowe is a writer based in Brighton, UK 42 | NewScientist | 15 March 2008
www.newscientist.com