- Email: [email protected]
Dubbing with deepfakes
News Technology
Dubbing with deepfakes Artificial intelligence is so good at manipulating videos, companies are using it to dub adverts and messages, reports Donna Lu which has generated Donald Trump’s voice. Canny AI’s system needs about a minute of speaking footage of both the person being deepfaked in the video and the voice actor saying the words that will be edited in. The system learns to transfer the movements of the lower half of the dubber’s face and neck into the video being edited. It works for both footage where the speaker is Italian Oscar-winning film Life is Beautiful was dubbed in English in 1999
facing the camera and footage where they are side-on. The result is a video in which the subject looks and sounds like they are saying the new dialogue. The algorithm is trained on the footage scene by scene, so dubbing five languages into one scene requires less processing power than dubbing for five separate scenes, says Omer BenAmi, co-founder of Canny AI. UK tech firm Synthesia also offers a dubbing AI, based on facial mapping. Its deepfakes were behind a malaria awareness
campaign video in which David Beckham appeared to speak in nine languages. Using such deepfake AIs commercially brings up legal questions, says Lilian Edwards at Newcastle University in the UK. “There’s an underlying issue there about what parts of yourself you own,” she says. “Do you own your face, do you own your image, do you own the voices coming out of your face?” These are questions that companies and celebrities will have to consider when entering into contracts, says Edwards. “Every client we’re working with has to declare they are responsible for copyrights and liability issues,” says Ben-Ami. The AI is an example of how deepfakes can be used positively, he says. “Any technology is a doubleedged tool,” says Edwards. The danger is that as AI video editing becomes more user-friendly and widespread, people could use deepfakes “to put words in the mouths of politicians, public figures, people they hate,” she says. ❚
The results are better than other AI-powered crowd estimation systems, proving 15 per cent more accurate than the nearest competitor at reaching the correct number in a crowd. The system is much faster than hand counting, taking 0.03 milliseconds to compute the number of people in each square (arxiv.org/abs/1909.12743). At present, the researchers have used the AI only in lab conditions,
but Bahmanyar hopes to soon mount the system onto planes and helicopters to do real-time counts. “I think in the right places, this technology could be really useful,” says Keith Still at Manchester Metropolitan University, UK, who invented one of the best current manual methods for estimating crowds. However, Still questions whether protest groups or governments actually want the real figures. “They want marketing numbers. Will they invest in something that punctures their claim?” ❚ Chris Stokel-Walker
ALLSTAR PICTURE LIBRARY/ALAMY STOCK PHOTO
FAKE videos created by artificial intelligence are now so good that film-makers are taking note. Israeli tech firm Canny AI is one of several companies cashing in on so-called deepfakes, using the technology to edit videos into different languages. The firm is currently using its AI to dub advertisements and messages from celebrities for audiences in different countries. It plans to use the technology for television shows and films in the future. Deepfakes make it easy for people with a bit of technical know-how to create fake videos. So far, they have mostly been used to make pornographic films involving celebrities or to create videos where well-known figures appear to say something that they haven’t. Canny AI, for example, created a satirical deepfake of Facebook founder Mark Zuckerberg that went viral in June. The firm’s technology requires a voice actor to provide replacement audio, unlike other algorithms that have learned to synthesise convincing fake speech, such as that of UK-based company Faculty,
Machine learning
AI could count how many people are in large crowds ARTIFICIAL intelligence may be able to settle the debate over how many people attend protests or gatherings. Vast numbers of people took to the streets of London last weekend to call for a second referendum on the UK’s membership of the European Union. But exactly how many people were there is disputed. Protest organisers say there were 1 million people, but when similar 8 | New Scientist | 26 October 2019
claims were made earlier in the year, they were disputed by fact-checking organisations. A method developed by Reza Bahmanyar at the German Aerospace Center and his colleagues that uses artificial intelligence could improve counts in the future. To create the system, the researchers hand-counted nearly a quarter of a million people in 33 images taken from planes, drones and helicopters, then used this to train an algorithm called MRCNet. MRCNet divides each image into small squares and analyses how many people are in each one.
“The team hopes to soon mount the system onto planes and helicopters to do real-time counts”
Dubbing with deepfakes Artificial intelligence is so good at manipulating videos, companies are using it to dub adverts and messages, reports Donna Lu which has generated Donald Trump’s voice. Canny AI’s system needs about a minute of speaking footage of both the person being deepfaked in the video and the voice actor saying the words that will be edited in. The system learns to transfer the movements of the lower half of the dubber’s face and neck into the video being edited. It works for both footage where the speaker is Italian Oscar-winning film Life is Beautiful was dubbed in English in 1999
facing the camera and footage where they are side-on. The result is a video in which the subject looks and sounds like they are saying the new dialogue. The algorithm is trained on the footage scene by scene, so dubbing five languages into one scene requires less processing power than dubbing for five separate scenes, says Omer BenAmi, co-founder of Canny AI. UK tech firm Synthesia also offers a dubbing AI, based on facial mapping. Its deepfakes were behind a malaria awareness
campaign video in which David Beckham appeared to speak in nine languages. Using such deepfake AIs commercially brings up legal questions, says Lilian Edwards at Newcastle University in the UK. “There’s an underlying issue there about what parts of yourself you own,” she says. “Do you own your face, do you own your image, do you own the voices coming out of your face?” These are questions that companies and celebrities will have to consider when entering into contracts, says Edwards. “Every client we’re working with has to declare they are responsible for copyrights and liability issues,” says Ben-Ami. The AI is an example of how deepfakes can be used positively, he says. “Any technology is a doubleedged tool,” says Edwards. The danger is that as AI video editing becomes more user-friendly and widespread, people could use deepfakes “to put words in the mouths of politicians, public figures, people they hate,” she says. ❚
The results are better than other AI-powered crowd estimation systems, proving 15 per cent more accurate than the nearest competitor at reaching the correct number in a crowd. The system is much faster than hand counting, taking 0.03 milliseconds to compute the number of people in each square (arxiv.org/abs/1909.12743). At present, the researchers have used the AI only in lab conditions,
but Bahmanyar hopes to soon mount the system onto planes and helicopters to do real-time counts. “I think in the right places, this technology could be really useful,” says Keith Still at Manchester Metropolitan University, UK, who invented one of the best current manual methods for estimating crowds. However, Still questions whether protest groups or governments actually want the real figures. “They want marketing numbers. Will they invest in something that punctures their claim?” ❚ Chris Stokel-Walker
ALLSTAR PICTURE LIBRARY/ALAMY STOCK PHOTO
FAKE videos created by artificial intelligence are now so good that film-makers are taking note. Israeli tech firm Canny AI is one of several companies cashing in on so-called deepfakes, using the technology to edit videos into different languages. The firm is currently using its AI to dub advertisements and messages from celebrities for audiences in different countries. It plans to use the technology for television shows and films in the future. Deepfakes make it easy for people with a bit of technical know-how to create fake videos. So far, they have mostly been used to make pornographic films involving celebrities or to create videos where well-known figures appear to say something that they haven’t. Canny AI, for example, created a satirical deepfake of Facebook founder Mark Zuckerberg that went viral in June. The firm’s technology requires a voice actor to provide replacement audio, unlike other algorithms that have learned to synthesise convincing fake speech, such as that of UK-based company Faculty,
Machine learning
AI could count how many people are in large crowds ARTIFICIAL intelligence may be able to settle the debate over how many people attend protests or gatherings. Vast numbers of people took to the streets of London last weekend to call for a second referendum on the UK’s membership of the European Union. But exactly how many people were there is disputed. Protest organisers say there were 1 million people, but when similar 8 | New Scientist | 26 October 2019
claims were made earlier in the year, they were disputed by fact-checking organisations. A method developed by Reza Bahmanyar at the German Aerospace Center and his colleagues that uses artificial intelligence could improve counts in the future. To create the system, the researchers hand-counted nearly a quarter of a million people in 33 images taken from planes, drones and helicopters, then used this to train an algorithm called MRCNet. MRCNet divides each image into small squares and analyses how many people are in each one.
“The team hopes to soon mount the system onto planes and helicopters to do real-time counts”