Art and Artificial Intelligence: How ‘Deepfakes’ can help create authentic Museum Experiences!
As ‘Deepfakes’ have become easier and less expensive to create, they have received a lot of attention in recent years. While they started out controversially because of their vulnerability for misuse, they have also shown immense potential for applications that can transform a wide range of experiences. In this article, we take a closer look at a couple of promising examples from the museum space, and explore how cultural institutions can use this technology to create innovative, interactive and authentic experiences.
The word ‘Deepfake’ owes its origin to a combination of “deep learning” and fake. It refers to hyper-realistic visual or/and voice based digital recreations of a human being using Artificial Intelligence and Machine Learning. Deepfakes started as simple face swapping technology and initially proliferated through fake videos of celebrities, often in pornography. Concerns were quickly raised about their potential for misuse if impersonations made of influential public figures like politicians, movie stars or musicians went viral — they could cause escalation of conflict or destroy careers and reputations. Technology majors like Google, Microsoft and Facebook have recognised that risk and initiated projects focused on detection as a result. However, like any technology with far reaching potential, it has the power to do good as it has the risk of being misused, and it is up to us how we want to deploy it. In relatively more meaningful applications, Deepfakes have been explored by Hollywood to complete critical scenes of actors who died midway through the shooting, or to enable synthetic voices for people who had lost their own voice after an illness. This month, a US company called Veritone launched a new platform named Marvel.ai which creates commercial opportunities for synthetic speech by allowing public figures like business leaders and celebrities to clone their voices which can be then used in different promotional contexts with their permission.
Application of Machine Learning to come up with digital avatars of dead artists is no more in the realm of science fiction — there have been a couple of recent creations already. The Dali Museum in St. Petersburg, US rejuvenated Dali as a permanent ‘exhibit’ Dalí Lives in May, 2019 on the eve of Dali’s 115th birth anniversary. Recently, in April 2021, the Museum of Art & Photography (MAP) in Bengaluru, India has followed with a similar digital avatar of MF Husain, one of India’s eminent modern artists. The MAP version is accessible online and can be experienced here. Both Dali and Husain are interactive and have been equipped to participate in a conversation and respond to questions posed by the visitor.
To understand how the process of bringing an artist back to life works, let us understand through the example of Dali’s resurrection. The first step is accessing and studying any kind of archival footage involving the artist — the best source is video footage as it provides access to the voice, the mannerisms and the visual persona. But it can include other sources like published interviews, their quotes, letters and journals: anything that provides a direct line of insight into the life of the artist, his thoughts and their expressed views. The Dali Museum along with their project collaborators Goodby Silverstein & Partners sorted through and studied hundreds of them. They then used more than 6000 frames and over 1000 hours of machine learning to train the AI algorithm to master Dali’s face — his unique features and the movements of his facial muscles as he spoke. The AI then was able to generate an unmistakable version of Dali’s likeness which could be overlayed on an actor’s face and expressions. This had to be complimented with sophisticated sound engineering which would synchronise Dali’s actual words with the new AI generated content. The output was over 45 minutes of created footage over 125 videos. This in turn has translated to a massive 190,512 possible combinations which means that the visitor can experience a wide range of responses, moods and emotions of Dali complete with his mannerisms and idiosyncrasies. According to the museum, 76% of the visitors surveyed post experience have rated the Dalí Lives experience as Very Good or Excellent which is a clear validation that they enjoyed it.
What kind of artists and museums are ideally suited for this? At first glance, artists who have lived in the television era and have been public personalities and were not dour recluses! There is likely to be sufficient material then that can be fed as data sets into the AI system for it to learn from. Popular masters with larger-than-life public personas like Picasso, Dali and Warhol would stand out from the 20th century. I would even argue that artists like Van Gogh who pre-date the motion picture era and did not leave behind too many photographs would be excellent case studies because of the many self-portraits and journals that they have left behind detailing not only their artistic progress and self-critiques but also their emotional journeys. In that context, well known Dutch photographer Ruud Van Empel has already taken the first step of solving the mystery of how Van Gogh looked like by fusing old photographic and collage techniques based on a study of the artist’s self-portraits, which are going to be exhibited at the Van Gogh Huis, Zundert starting this August.
At the same time, the technology offers a way to correct the male skewed imbalance in western art history by reinstating women artists and those of colour. So, while Dali and Husain are low hanging fruits, it must be used to bring back artists like Frida Kahlo and in the Indian context, someone like Amrita Sher-Gill. Monographic museums which focus on single artists are again ideally suited for this. What makes the idea even more powerful is that it can travel with minimal carbon footprint. Artist Foundations or Monographic museums (like the Fundació Gala-Salvador Dali or the Van Gogh Museum in Amsterdam) are likely to be the caretaker of the artists’ legacy and the primary source of archival footage. Hence, they should take the lead in the original creation which they can then license or loan out for a fee to other museums globally when the latter are doing artist retrospectives and can exhibit it on high-definition interactive screens. The other cost-effective way to achieve this is the way MAP has done by making the Husain experience accessible from a desktop or mobile browser online.
What are the likely roadblocks? Museums need to realise that this can be a really powerful innovation and not just another gimmicky idea. A well-executed digital version of the artist can be a more engaging museum guide than the usual audio guides or even an inexperienced gallery assistant. As the underlying system learns from access to more archival footage as well as actual visitor interactions, it will only get better. Museums also need to hire people at the leadership level who can take the mandate forward. MAP has on board as their Chief of Technology Mayank Manish, who also happens to be the founder of a start-up offering 3D Holographic solutions. No wonder that MAP plans to have a 3D holographic ‘in person’ version of Husain when the museum finally opens its door later this year. Every expertise cannot be built in-house, but has to be accomplished through collaborations with partners who can understand the museum’s objectives and deliver that technology solution, like MAP has done through Accenture or the Dali Museum through Goodby Silverstein and Partners. Finally, curators have to shed their bias for perfection and be willing to experiment. The MAP Husain avatar does come across a bit stiff and can improve in terms of the spontaneity and depth of conversation (even more so if you compare it to the Dali version), but the museum has to be appreciated for taking that brave first step which most leading museums globally have yet to take.
In conclusion, academic wall texts, clunky audio guides and monologues by gallery assistants all lead to disappointing experiences for visitors today. For the digitally native next generation who seek something more actively engaging, the gap is more glaring. Bringing back an artist to life through a virtual avatar can create a credible illusion of authenticity to begin with, but at a deeper level also help the visitor find an emotional connection leading to a deeper appreciation for the artist’s work. In the Dali exhibit, the artist at the end even offers to take a selfie and then sends it directly to the visitor’s phone via text. These novel experiences create an instant vow factor and make it more memorable. While relatively less known privately funded museums like the Dali Museum, Van Gogh Huis and Museum of Art & Photography have shown the appetite for risk and innovation, the larger museums by following the lead can enable the idea of AI enabled ‘resurrection’ and making them an authentic voice of storytelling reach its full potential.
If you are still wondering about the relative merits of the idea, it is worth listening to feedback from one of the visitors at the Dali Museum:
“I saw Dali, his actual form, his full figure, speaking personally — to me.”
It cannot get more authentic than that.
The article was published in Italian in Rome Based Art Foundation Fondazione Ducci’s art magazine Art Fond on 1st June, 2021 and can also be accessed here: https://www.artefondazioneducci.com/post/arte-e-intelligenza-artificiale-come-i-deepfakes-aiutano-a-creare-esperienze-museali-autentiche
