AI, Narratives, Sound and Equality

A photographic rendering of a simulated middle-aged white woman against a black background, seen through a refractive glass grid and overlaid with a distorted diagram of a neural network.

An Interview with Spencer Collins by Dr Jenn Chubb

“Sonic narratives matter but not everyone is experiencing the same thing”: What AI sounds like to a Deaf person.

In a previous post for this blog, I suggested that a failure to take into account the significance of sonic framing when considering visual narratives of AI limited the capacity to fully broaden a public understanding of AI. I now explore the potential which sound and music hold in relation to communicating the realities of AI, and what we can learn from someone who does not hear them but accesses them via different means.

Undeniably, music has power. We use it to make us feel good, to change how we think/feel, to distract us or help us concentrate, to help us run faster or slow down and relax. Music manipulates how we perceive people, concepts, or things. When music is used in storytelling, it is chosen for the resonances and emotional overtones that it brings to the topic, capable of framing a story with fear, sadness or joy (Brown & Volgsten, 2005).

In screen media, this power is used liberally to create the pace and the framing of narratives; one need look no further than the two notes and a rising tempo which has forever impacted society’s perception of sharks. When paired with factual content, like documentaries, music has the power to make the audience take sides on a narrative that is notionally objective. This is especially pertinent for new technologies, where our initial introduction and perception is shaped by the narratives presented in media and we are likely to have little direct lived experience. 

The effect of background sound and music for documentaries has been largely overlooked in research (Rogers, 2015). In particular, the effect of sonic framing for storytelling about AI is little explored, and even less so for audiences with disabilities, who may already find themselves disadvantaged by certain emerging technologies (Trewin et al., 2019)

This blog reflects on the sonic framing of Artificial Intelligence (AI) in screen media and documentary. It focuses particularly on the effects of sonic framing for those who may perceive the audio differently to its anticipated reception due to hearing loss, as part of a wider project on the sonic framing of AI in documentary. I will be drawing on a case study interview with a profoundly deaf participant, arguing that sonic narratives do matter in storytelling, but that not everyone is experiencing the same thing.

For those who perceive music either through feeling the vibrations with the TV turned up high, or described in subtitles, lighter motifs are lost and only the driving beats of an ominous AI future are conveyed. This case study also reveals that the assumed audience point of view on AI technology is likely much more negative for disabled audience members- this is not unlike the issues with imagery and AI which can serve to distract or distort. All this feeds into a wider discussion surrounding what we might refer to as responsible storytelling – fair for people with disabilities.

I had the pleasure of chatting to Spencer Collins, an experienced Equality, Diversity and Inclusion Advisor in the private, public, and NGOs sectors. We chatted about AI technologies, imaginaries of how AI might sound, and barriers to engagement, especially with regards to conversational AI. We also discussed the effect his disability has on his experience of storytelling.

“Sonic narratives matter but not everyone is experiencing the same thing.”

Spencer tells me from the start that “AI technologies are not set up for people like me.” He does so as the caption on our Zoom call churns out what most closely resembles nonsense, providing me with an early indication of some of the very real challenges Spencer faces on an everyday basis. Luckily, Spencer can lip-read and I am keen to hear what he has to say.

What becomes clear is that while AI has the potential to massively impact the lives of people living with a disability, they don’t work properly for people with disabilities yet. What is also clear is the role that imagery and sound together have in influencing perceptions of technology in storytelling.  Importantly, Spencer feels there is a need to involve people with disabilities in the development of technology and in the stories that promote it. He asks, “have companies really spoken to people with disabilities? How safe and useful is AI for people like me?”

Can you introduce yourself?

I’m Spencer, with exceptional skills and experience for more than 18 years in the private, public, and NGOs sectors. Many companies are struggling to ensure their Equality and Inclusion policies and procedures really reflect the needs of their businesses and employees, their self-esteem, perceived worth and improving company productivity. So I help with that. 

I am profoundly deaf. I come from a hearing family. I am very lucky that my mother has been so great in helping me get to the best speech therapist in the world. But it is tough. 

How do you perceive the visual representation of AI in the media?

Good question. Having grown up watching ‘Mr Benn’, the character transformed himself into another character or person in an alternate universe similar to the world of Avatar when he transformed himself to become another person. There was a lot of diversity in the play, and Avatar is supposed to be equally interesting. I am aware that there is native and Asian representation, but I wonder how AI portrays disabled characters?

I often ask myself, “Do I trust everything I see, equally so as when my hands touch something I believe what my eyes see?”

Do you have any comments on our alternative images in the BIOAI database?

Better Images of AI embraces honesty and authenticity, as well as rigorous facts for the audience. I’m interested to see how artificial intelligence converts the artist’s emotional state, brushwork, and inner state into tangible artworks by translating the work of Picasso, Frank Stella, etc into photographs, the algorithm can recognise the truth of the image. This is the question I need to observe.

There is a lot of interest in images of AI. I feel as though image and sound are intertwined. Do you have an imagination of what AI sounds like?

The hard thing about technology is that there is a gray area – it doesn’t work for everyone. The dancer in ‘Strictly come dancing’ – relies on vibrations. Deaf people can feel the vibrations, so that’s not coming from a hearing aid. 

We have a sixth sense. We are like animals. We can see sound, we can feel something. I have no hearing but superb senses, I can see in the dark, I can lip-read. We taught ourselves how to use our senses. Technology can’t identify this, it will take a long time for technology to be ready. 

I’m a musician, so I can relate to this when you describe how you ‘feel’ sound – thinking about storytelling, which uses sound so often to influence audiences, if you were to imagine science fiction, for instance, what kind of sound do you sense from these stories?

When I watch Robocop – I have always wanted to be a Pilot/Traffic Patrol, but I get a lot of rejections because of my disability. You need to have 20/20 hearing. Same with the air force – going back to Robocop, I remember some kind of AI robot, he had a power in him, like a spider-man. A strength and power. I remember when I was watching at one point it was all about the audio. You could pick up the translation from the telephone. That would be wonderful for the Deaf community. 

Interestingly the U.S. has introduced a new development where every sound can be identified for Deaf and Hard of Hearing people, so they can hear everything like any other person. It is something that you could check out.

Sometimes when watching a story, you get a physical reaction, you get a shock. Science fiction might seem very loud to me, which might make the concepts seem more frightening. 

How do you interpret sound?

Use of colour and visualizations become really important. I rely on noise, I rely on anything loud, so I can feel the vibrations. Beethoven was profoundly deaf. He cut off the legs of the piano, to hear the difference between high and low note vibrations. Space and distance from technology is important. 

Is there an argument for more immersive storytelling and tools?

VR therapy (using AI) is being used and this is a massive moment for people with a range of challenges such as dementia.

Immersive tools are not quite ready though. Think of the ‘metaverse’ – one of the questions is these ‘Minecraft’ type characters. They don’t have any body features, e.g., lips, etc, so this is not ready for people who are hard of hearing. These characters are developed for people like you, not like me. One answer is to use captions, but companies say that that would take away from what was happening around the room – this is not equal. 

The Gather-town gaming platform is another example of diverse forms of communication. The most confusing part there is people are talking, but I don’t know, I can’t tell who is talking. These platforms are very black and white.  A colour code could be helpful, so we know who is talking. 

How do we look into making sound more accessible?

The first step to investigating this kind of AI is learning how to classify music, and then how to feed this information to the machine so that it can be interpreted by Deaf users visually. In order to determine how the Deaf community currently enjoys music through vibration, and what cues they feel accurately portray the sound, is another question.

Recently I watched a film where a robot was talking and I could not tell what it was saying. I thought – if that was me – I would not have a clue, no visual clue at all. No lips, no movement. It would not do me any good.

Do you use AI in the home?

I have got Alexa – but it doesn’t recognise my speech. If I say something, it completely misunderstands me. I love music, I play all types of music but using AI Apple iTunes, it doesn’t even recognise me. It cannot play music for me. I have to use my hands. 

I know some friends have a [echo] dot that can talk…to turn the light on etc, but I can’t get it to recognise me. That is a worry when we want someone to do a job on our behalf. We don’t want gimmicks: will the product deliver 100% equally to everyone? 

Would it be useful for AI narratives to represent sound visually?

Just imagine you’re watching any music film and the closed captioning is on, but they don’t really give the sound too. Imagine if that’s on the screen too. The company will say ‘no, forget it, we don’t want more captioning over the picture’. How can you incorporate the sound and the text- I don’t have the answer to that. If there was something for sound, that would be great. 

The relationship between image and sound seems very important. For instance, documentarians tend to start with the image when they sound design a documentary about AI. In what ways do you feel they link together?

In the same way as Digital Imagery, cinema and games are made up of CGI models that give a realistic look to delight the audience through sound and image. If you go to the cinema and surround sound, oh my god, it’s amazing. You can feel it when it’s so loud. It’s fantastic. It’s an experience. I was watching Click and they created a code plugged into the guitar. Visualizing sound. Maybe that is the way forward.

Is the future of AI positive?

AI can transform the lives of disabled people. Many advances have been made but they offer too much promise to individuals. It’s crucial to bring diverse voices into the design and development of AI.

In what ways, if any, does being deaf impact your relationship with AI?

Take ‘Story Sign’ for example, which is powered by Huawei AI and developed with the European Union for the Deaf and reads children’s books and translates them into sign language. There have been impacts when the US government believes Huawei is a threat to worldwide cybersecurity. How can we tell if artificial intelligence is safe for the Deaf community? These are the questions I am concerned about.

Documentary is one instance where the story is about documenting fact – what kinds of ways might sonic narratives be used to influence the audience? Do you think they matter?

As a former DJ, I encountered a new way of experiencing music through vibrations when played from the ground to above, which challenged my audience to consider hearing vs. feeling. 

Are there any metaphors you can imagine about AI and whether there is a link between those we think of when we consider imagery?

Imaginary language uses figurative language to describe objects, actions, and ideas in a way that appeals to the physical senses and helps audiences to envision the scene as if it were real. It is not always accurate to call something an AI image. Visual and sounds can be described by action, but tastes, physical sensations, and smells can’t.

So what priorities should we have around future tech development?

We need to think about how we can compromise equally, in a fair and transparent way. I personally feel tech would make life a lot easier for people with disabilities, but from the other side, I am frightened. 

Talking about care homes and robot carers, I wonder, how will that fit someone like me? It just removes the human entirely. We are too afraid about what will happen next. There is always a fear. 

Like anything we are being watched by Big Brother, it’s the same principle for the future of AI, it’s going to be a challenge. It will work for some people, but not all of them. You have fake news saying how wonderful AI is for people with disabilities, but they don’t necessarily ask what those people want. 

Are there ways to make AI narratives more inclusive for deaf people?

It’s probably worth reading this report – technology is not an inherent ‘good for disabled people’.

Recommended: How do blind people imagine AI? 

Buzzword Buzzkill: Excitement & Overstatement in Tech Communications

An illustration of three „pixelated“ cupboards next to each other with open drawers, the right one is black

The use of AI images is not just an issue for editorial purposes. Marketing, advertising and other forms of communication may also want or need to illustrate work with images to attract readers or to present particular points. Martin Bryant is the founder of tech communications agency Big Revolution and has spent time in his career as an editor and tech writer. 

“AI falls into that same category as things like cyber security where there are no really good images because a lot of it happens in code,” he says. “We see it in outcomes but we don’t see the actual process so illustration falls back on lazy stereotypes. It’s a similar case with cyber security, you’ll see the criminal with the swag bag and face mask stooped over a keyboard and with AI there’s the red-eyed Terminator robot or it’s really cheesy robots that look like sixties sci-fi.”

The influence of sci-fi images in AI is strong and one that can make reporters and editors uncomfortable with their visal options. “ “Whenever I have tried to illustrate AI I’ve always felt like I am short changing people because it ends up being stock images or unnecessarily dystopian and that does a disservice to AI. It doesn’t represent AI as it is now. If you’re talking about the future of AI, it might be dystopian, but it might not be and that’s entirely in our hands as a species how we want AI to influence our lives,” Martin says. “If you are writing about killer robots then maybe a Terminator might be OK to use but if you’re talking about the latest innovation from DeepMind then it’s just going to distort the public understand of AI either to inflate their expectations of what is possible today or it makes them fearful for the future.” 

I should be open here about how I know Martin. We worked together for the online tech publication The Next Web where he was my managing editor and I was UK editor some years ago. We are both very familiar with the pressures of getting fast-moving tech news out online, to be competitive with other outlets and of course to break news stories. The speed at which we work in news has an impact on the choices we can make.

“If it’s news you need to get out quickly, then you just need to get it out fast and you are bound to go for something you have used in the past so it’s ready in the CMS (Content management system – the ‘back end’ of a website where text and images are added.),” Martin says. “You might find some robots or in a stock image library there will be cliches and you just have to go with something that makes some sense to readers. It’s not ideal but you hope that people will read the story and not be too influenced by the image – but a piece always needs an image.”

That’s an interesting point that Martin is making. In order to reach a readership, lots of publications rely on social media to distribute news. It was crowded when we worked together and it sometimes feels even more packed today. Think about the news outlets you follow on Twitter or Facebook, then add to this your friends, contacts and interesting people you like to follow and the amount of output they create with links to news they are reading and want to comment upon. It means we are bombarded with all sorts of images whenever we start scrolling and to stand out in this crowd, you’re going to need something really eye-catching to make someone slow down and read. 

“If it’s a more considered feature piece then there’s maybe more scope for a variety of images, like pictures of the people involved, CEOs, researchers and business leaders,” Martin says. “You might be able to get images commissioned or you can think about the content of the piece to get product pictures, this works for topics like driverless cars. But there is still time pressure and even with a feature, unless you are a well-resourced newsroom with a decent budget, you are likely to be cutting corners on images.” 

Marketing exciting AI

It’s not just the news that is hungry for images of AI. Marketing, advertising and other communications are also battling for our attention and finding the right image to pull in readers, clicks or getting people to use a product is important. Important, but is it always accurate? Martin works with and has covered news of countless startup companies, some of which use AI as a core component of their business proposition. 

“They need to think about potential outcomes when they are communicating,” he says “Say there is a breakthrough in deep neural AI or something it’s going to be interesting to academics and engineers, the average person is not going to get that because a lot of it requires an understanding of how this technology works and so you often need to push startups to think about what it could do, what they are happy with saying is a positive outcome.” 

This matches the thinking of many discussions I have had about art and the representation of AI. In order to engage with people, it can be easier to show them different topics of use and influence from agriculture to medical care or dating. These topics are far more familiar to a wider audience than a schematic for an adversarial network. But claiming an outcome can also be a thorny issue for some company leaders.

“A lot of startup founders from an academic background in AI tend to be cautious about being too prescriptive about how their technology could be used because often if they have not fully productised their work in an offering to a specific market,” Martin explains. “They need to really think about optimistic outcomes about how their tech can make the world better but not oversell it. We’re not saying it’s going to bring about world peace, but if they really think of examples of how the AI can help people in their everyday lives this will help people engage with making the leap from a tech breakthrough they don’t understand to really getting why it’s useful.” 

Overstating AI

AI now appears to be everywhere. It’s a term that has broken out from academia, through engineering and into business, advertising and mainstream media. This is great, it can mean more funding, more research, progress and ethical monitoring and attention. But when tech gets buzzy, there’s a risk that it will be overstated and misconstrued. 

“There’s definitely a sense of wanting to play up AI,” Martin says. “There’s a sense that companies have to say ‘look at our AI!’ when actually that might be overselling what is basic technology behind the scenes. Even if it’s more developed than that, they have to be careful. I think focusing on outcomes rather than technologies is always the best approach. So instead of saying ‘our amazing, groundbreaking AI technology does this’ – focusing on what outcomes you can deliver that no one else can because of that technology is far more important. 

As we have both worked in tech for so long, the buzzword buzzkill is a familiar situation and one that can end up with less excitement and more of an eyeroll. Martin shared some past examples we could learn from, “It’s so hilarious now,” he says. “A few years ago everything had to have a location element, it was the hot new thing and now the idea of an app knowing your location and doing something relevant to it is nothing. But for a while it was the hottest thing. 

“Gamification was a buzzword too. Now gamification is a feature in lots and lots of apps, Duolingo is a great example but it’s subtly used in other areas  but for a while startups would pitch themselves saying ‘we are the gamified version of X’.”

But the overuse of language and their accompanying images is far from over and it’s not just AI that suffers. “Blockchain keeps rearing its head,” Martin points out. “It’s Web3 now, slightly further along the line but the problem with Web3 and AI is that there’s a lot of serious and thoughtful work happening but people go ahead with ‘the blockchain version of X or web3 version of Y’ and because it’s not ready yet or it’s far too complicated for the mainstream, it ends up disillusioning people. I think you see this a bit with AI too but Web3 is the prime example at the moment and it’s been there in various forms for a long time now.” 

To avoid bad visuals and buzzword bingo in the reporting of AI, it’s clear through Martin’s experience that outcomes are a key way of connecting with readers. AI can be a tricky one to wrap your head around if you’re not working in tech, but it’s not that hard when it’s clearly explained.”It really helps people understand what AI is doing for them today rather than thinking of it as something mysterious or a black box of tricks,” Martin says. “That box of tricks can make you sound more competitive but you can’t lie to people about it and you need to focus on outcomes that help people understand clearly what you can do. You’ll not only help people’s understanding of your product but also the general public’s knowledge of   what AI can really do for them.”