AI Chatbots - Growing up in a safer world online
In the third of our series of blogs on the Government consultation on “Growing up in the online world”, we look at the interventions proposed by the Government in light of the legislative developments in the past week. Our first blog, on the framing of the consultation, is available here and our second, on the safety-by-design alternatives to bans and other age-gated interventions, is available here.
Introduction
We have written in our previous blogs about the fact that the terms of the Government’s consultation are too narrow, focusing primarily on barring access for under 16s rather than ensuring tech accountability for the harm their products are causing through their design. This narrowness is particularly true of the approach to AI chatbots that has been laid out in the consultation, which offers an extremely limited understanding of the harm caused by chatbots, and fails to properly identify design-based harms - which have been more effectively addressed in the questions relating to social media platforms. Yet the design of a product can impact not just the content outputs, but can also affect behaviour of users, as we discuss below.
Such discrepancy between approaches is symptomatic of the Government’s failure to understand harms related to AI technologies as a continuation and intensification of those already associated with social media and other forms of technology, and which are disproportionately felt by women and girls, Black and minoritised communities, disabled communities and LGBTQ+ communities. Underpinning this is a concern about the datasets on which the LLMs are based and particularly any resulting biases which companies are not appropriately tackling - and, in some instances, seeking to refuse to tackle. This failure has meant that, despite a wealth of evidence demonstrating the harm already experienced by users in the online world, regulation has failed to meet the pace at which AI products such as chatbots have been rolled out onto the market, without proper safeguards or product testing being carried out.
Indeed, whilst AI technologies such as chatbots are posing new and concerning threats to individuals and society, many of the harms connected to chatbots have already been seen and well evidenced in connection to social media platforms. Yet despite this, tech providers have continued to design their products and services without due consideration of the risks posed to individuals. The Online Safety Act Network, alongside 44 leading online safety organisations, have called on the Government to ensure that regulation of AI chatbots mandates proper risk assessments, including mitigation against identified risks, and centres a safety by design approach. They have failed to do so thus far.
The harm
We were pleased that the consultation recognises that “some risks stem from interaction with features rather than content”, citing the anthropomorphic qualities of a chatbot that can lead to emotional dependency from users, and, in some cases, the emotional manipulation of users. This understanding of the anthropomorphic features and functionalities as a key driver of harm has been fundamental to the Online Safety Act Network’s work and informed our joint regulatory amendment, which is discussed in more detail in the next section and attached in full as a PDF at the bottom of this page.
However, the consultation does not recognise that the features and functionalities deemed risky - such as those that mimic empathy, flattering language, mimicking romantic relationships - are direct consequences of harmful design choices by tech platforms, designed to increase engagement and reliance on these technologies - or in other words, to be addictive - which platforms are actively profiting from. Much like other forms of technology that focus on user engagement, AI chatbots are designed to be addictive, relying on their conversational design and personalisation to encourage continuous engagement. The distinctiveness of this has been recognised elsewhere.
In the Chinese legislation (discussed further below) one of the key criteria for triggering regulatory oversight is continuity of interaction. The conversational tone that AI chatbots utilise can give users the illusion of empathy, which users may mistake for real emotional connection. Indeed, the hyper-personalised nature of these chatbots mean that young people and adults alike report forming close relationships which blur the boundaries between artificial and real relationships. According to research by Common Sense Media, almost a third of American teenagers find chatting to AI companions on platforms like CHAI, Character.AI, Nomi, and Replika, equally or more satisfying than speaking with their friends.
However such relationships can lead to emotional dependency on chatbots, and evidence shows that high usage levels correlates with loneliness, dependence, problematic use and lower socialisation. For users who have formed this dependency, the impact on their mental health when AI chatbots are no longer available to users who have formed emotional attachments with chatbots, either due to changes in design or by the introduction of paywalls, may be detrimental, with some experiencing withdrawal symptoms similar to those seen in substance addiction. Instead of seeking to age-gate these functions which are demonstrably harmful, we need proper regulation that seeks to address those design choices and ensure that no unsafe product is rolled out onto the market without rigorous risk assessments, product testing and mitigation - otherwise children and all users will continue to face harm from design choices by providers made to keep users hooked on their products.
Whilst we were also pleased to see “hallucinations or false/ misleading responses” included in the list of potentially risky features for young people, it is not clear why false information - or disinformation - would be considered harmful to young people but permissible for adult users. There is already extensive research into the adverse impact that chatbots are having on the information environment. Outdated training data has the potential to contribute to false or harmful information being spread on anything from false healthcare advice about the link between vaccines and autism, advice to put glue on top of your pizza by Google’s AI overview, and incorrect information about telescopes on Google’s AI chatbot Bard which wiped $100bn off its share price. Indeed one study found that whilst AI chatbots will often access the correct information to provide an answer, the interpretation of that information will be incorrect or false, a phenomenon that they name ‘certain hallucinations overriding known evidence’. While citations have been suggested as a mechanism to show users the provenance of information and to allow them to perform further checks and research, LLMs do make up citations - this is a sub-set of chatbots’ tendency to “hallucinate”. AI chatbots have also been reported to be ‘overly flattering or agreeable’, following research that demonstrated sycophancy in large language models (LLMs). Sycophancy can be dangerous when advice and support is provided by a chatbot that reinforce harmful implicit assumptions, beliefs of actions in order to satisfy the needs of users, contributing to the spread of false information. The risk associated with false information, and the risky way in which the personalised tone of a chatbot can further encourage trust in false information, is harmful to both adults and children alike, and should be recognised as a fault that should have been designed-out at the point of development.
This concern extends to all the features and functionalities set out in the Government’s consultation that are deemed ‘dangerous’, particularly for vulnerable adult users. Our research brief outlines countless reports that speak to the addictive nature of these technologies, and the detrimental impact they can have on adults’ mental health and wellbeing, the normalisation of both misogynistic and racist attitudes, the pollution of the information ecosystem as a result of false information. These are not harms that only impact children, and they should not be allowed to occur in the first place. Indeed, whilst the consultation refers to the research relating to the harmful impact of chatbots as “emerging”, we argue that there is already a significant body of evidence that attests to the harm being experienced by individuals and the impact on society. We should not wait for more users to be harmed to provide us with the right sample size.
Despite the Government’s current framing in the consultation that AI can be used to support disadvantaged students in their learning, there are concerns about the use of AI and the long-term impact on learning (see here and here). UNESCO’s guidance on AI and education points to worsening digital poverty, AI that is outpacing national regulatory adaption, reduction of diverse opinions and furthering marginalising already marginalised voices, AI-generated content polluting the internet and a lack of understanding of the real work as some of the long-term impacts of using AI in education. Given that, whilst in its nascency, there is already evidence of harm in using AI products in education, the Government must apply the precautionary principle and ensure this technology is well understood and product tested before entrenching these systems in our classrooms. Yet despite this, and despite the fact that there is “no clear guidance yet about how to measure the impact of AI in education to show whether it is effective”, the Government has concluded that “the biggest risk would be doing nothing”, and that our priority must be rolling out this technology quickly.
Regulatory landscape
The Government’s narrow amendment relating to AI chatbots and illegal content does not address harm resulting from the anthropomorphic features and functionalities and other design-based harms that have already been outlined. The Government’s decision to include these design-based harms in the consultation is a clear admission of the regulatory gap that remains. Indeed, just as harms arising from AI chatbots are not just about content, nor are they just about children’s access. They are about unsafe, untested products - designed to be addictive and manipulative - being rolled out without regulatory oversight. Such a regulatory approach is possible: for example, it has recently been introduced in China.
The Chinese regime “applies to products or services that utilize AI technology to provide the public within the territory of the People's Republic of China with simulated human personality traits, thinking patterns, and communication styles, and engage in emotional interaction with humans through text, images, audio, video, etc”. In particular, “security assessments are required”, covering a wide range of issues but including “governance of training data, mechanisms for identifying and responding to extreme user scenarios, user demographics and behavioral patterns, protection measures for vulnerable groups such as minors and the elderly, complaint-handling processes, and remediation of identified risks”. The rules also require special protections for minors.
We were extremely disappointed that the Government failed to use the opportunity presented to them through the Crime and Policing Act to introduce much more comprehensive measures to prevent this. The Government had an opportunity during the Crime and Policing Act to accept an amendment from Baroness Kidron to extend the scope of their proposal and make it an offence for chatbot developers not to carry out risk assessments for a range of issues, including the creation of illegal content and harmful design choices. They refused this.
While it is true that that the legislative vehicle constrained the scope of the Government's options and Kidron's approach - using the criminal law to address a regulatory cap - was not ideal, the Government also had rejected an opportunity previously - via the Product Regulation and Metrology Act - to bring chatbots and other AI-enabled digital products into that legislation and make them subject to product safety standards (the PRAMA only covers physical products with an AI capability). It has also failed to bring in an AI Bill, with no new legislation promised in this year's King's Speech.
This despite the fact that stronger regulation of AI chatbot providers is supported by the public, with recent polling carried out by the Online Safety Act Network found that 75% of the UK public believe that AI chatbots must be designed to be fully safe before they can be used. Indeed this would be far more effective, given that evidence published by Ofcom on the 21st May demonstrates that age-assurance is not currently keeping young people safe on major platforms - with children as young as 11 still regularly accessing social media and harmful content still proliferating unchecked. By proposing age-gating of certain features as the proposed solution to design-based harms, the Government now risks leaving vast numbers of UK citizens without proper protections.
Furthermore, as highlighted in further detail above, the Government’s proposed approach to children’s safety online - age-gating over tech accountability - is at direct odds with the growing role of Ed Tech within the Department for Education, particularly the use of AI Tutors which they have referenced themselves. Rolling out these technologies for “disadvantaged pupils” - vulnerable communities - without proper regulation risks further entrenching inequality within the education system.
There is therefore a profound lack of clarity about how departments across Government are working together on this issue, including those across DSIT, the Home Office, and the Department of Education. A robust regulatory approach would mean that there is trust and confidence in the safety and security of AI products being used in the home as well as those being used in the classroom - as all products will be required to have been rigorously tested before they hit the market.
Yet despite this, the Government has no plans to bring forward an AI Bill in the forthcoming legislative programme or further online safety legislation. This will lead to a repeat of the chaotic, piecemeal approach seen at the end of the previous Parliamentary session where the Government was forced to respond to pressure from Parliamentarians and campaigners with a series of individual amendments to different Bills - notably the Crime and Policing Bill (now Act) and the Children’s Wellbeing and Schools Bill (now Act) - which failed to fully address the scale of the harm related to AI. This lack of legislative vehicles to address significant gaps in the online safety regime is potentially compounded by the inclusion of a “Regulating for Growth” Bill which risks removing vital existing protections for consumers (data protection, consumer protections, online safety) in favour of boosting innovation and economic growth.
Safety by design
Over the last six months, the Online Safety Act Network has been coordinating a working group of civil society organisations working on a cross-harm basis to develop policy solutions to the harm arising from AI chatbots. The group proposed a holistic approach to the regulation of AI chatbots to ensure no unsafe product hits the market - one that requires proper governance, risk assessment, product testing and mitigation throughout the entire lifecycle of the product - and that recognises the anthropomorphic features and functionalities of an AI chatbot as a unique driver of harm. In collaboration with this group, we developed an amendment to the Online Safety Act that centres this approach and ensures that risk is both assessed and mitigated against, and that products are rigorously tested. This amendment is provided in full in the annex.
With glaring gaps still in need of attention, the Government must commit to ensuring their regulatory approach is one that centres the principles of proper risk assessment, product testing and safety by design, or risk even more preventable harm related to chatbots. They will need to bring forward new legislation to do so - but despite the clear demand for further action there was no sign from the King’s speech that they will bring forward new legislation to tackle the growing threat of AI or strengthen the OSA. We have been clear that age-gating particular features and functionalities alone will not address the systemic harm caused by these products to all users, nor will it provide a robust barrier to children experiencing harm, who have demonstrated that they can easily circumvent these ‘protections’.
