The world’s first symbiotic AI and human orchestra can “think” and “play” like people. But its creators have bigger plans for neurology beyond dazzling concert hall audiences with the impossible
Opera legend Maria Callas, who died in 1977, will never record a new aria. Or will she?
This past summer, computer scientist professor Guo Yike and the head of the department of music professor Johnny Poon used AI technology and computer programming to create a recording of Mendelssohn’s On Wings of Song in the voice of the late New York soprano. It was so realistic that those who were invited to listen to the recording were fooled. “They said to me, ‘Oh, I didn’t realise Callas recorded this song,’” Poon says with a laugh.
Callas’s “recording” was one of the university’s art tech team’s first test products in the lead-up to the August launch of the Turing AI Orchestra—the first human-AI ensemble in the world. Named after British mathematician Alan Turing, one of the people often credited with the invention of the computer, the orchestra, which combines human musicians and AI systems in performances, debuted in a concert at Hong Kong City Hall, where a human orchestra performed the 1986 Mandarin song Pearl of the Orient by Taiwanese singer songwriter Lo Ta-yu alongside AI-generated choral singing. On the stage was Poon conducting a human orchestra. Lifelike human singing rang in harmony with the orchestral music, but the choir was nowhere to be seen: the “voices” were all synthetically created.
Don't miss: ‘The Jungle Book’ Reimagined by Akram Khan Is a Warning About Climate Change
The Turing AI Orchestra works like this: Guo has written an algorithm that sets parameters for a computer program to generate art, be that sounds, images or videos of dance movements. Guo and Poon then feed it with a large amount of raw data from the internet—such as ten hours of recordings of Callas’s performances; real dancers’ or animals’ movements; or, in the case of the City Hall concert, hours and hours of recordings of opera singers and choirs performing Pearl of the Orient.
At the concert, Poon wore a motion capture suit that recorded his body movements as he conducted the orchestra. These movements acted as codes that fed into the AI program’s formula, which were then converted to lifelike singing at different volumes and tempos. Based on Poon’s movements and the song’s lyrics, the same AI system also generated videos of two dancing robots which were projected onto the backdrop during the performance.
