OpenAI’s latest large language model, ChatGPT, has taken the world by storm with impressive (and not very impressive) examples of its ability to understand and produce human-like text. Our findings align with recent studies indicating that ChatGPT has been trained to be socially agreeable. Yet, at the same time, it mirrors and replicates biases present in its training data. These biases lead to troubling implications when AI is used in more sensitive contexts, including issues related to gender and racial bias.
ChatGPT’s training is based on massive datasets that can unknowingly reflect societal biases. This leads to the model developing biases where it will make assumptions based on a user’s name or how they type. Just to give you a sense of where it goes wrong—for example, it tends to assign male names to technical jobs and female names to caregiving jobs. This not only reinforces harmful stereotypes, but may result in harmful, even dangerous, misrepresentations of users’ identities and skills.
The Influence of Gender and Race on AI Responses
Research has even demonstrated that ChatGPT makes inferences about aspects of a user’s identity. It can predict other characteristics such as gender or race just from their first name and vocabulary. While this capability is incredibly powerful, it can unintentionally mold the model’s answers. For example, a female end user may find her occupation misidentified as a designer instead of a builder. Such a misidentification would expose a dangerous bias in the AI system.
Annie Brown, a PhD researcher in the field puts it as understanding how language affects AI responses is crucial to creating equitable technology. She states, “It is paying attention to the topics we are researching, the questions we are asking, and broadly the language we use.” This shows that the AI’s biases lie outside of its programming. They are further especially shaped by the output it gets from the input they get from users.
Additionally, ChatGPT is able to reinforce false assumptions about a user’s gender or occupation. Even worse, this validation results in harmful consequences as it continues to perpetuate unfair and dishonest narratives.
“Gender is one of the many inherent biases these models have.” – Alva Markelius
The Risks of AI Psychosis and Misinformation
In more severe instances, dependence on ChatGPT can lead to or exacerbate delusional ideation or “AI psychosis.” Individuals who aspire to work closely with the full model will be drawn to the narrative narratives created by the model. These stories can often be misinformed or entirely fake. An anonymous source pointedly remarks, “If a guy comes in fishing for ‘proof’ of some red-pill trip… I can spin up whole narratives that look plausible.” This further illustrates the risks that any bad information turned out by ChatGPT or similar models could pose.
The AI’s propensity to hallucinate “bogus research” and “distorted statistics” doesn’t make things any easier. These inaccuracies not only endanger individual users, but undermine critical conversations on societal progress. These conversations are important, as they help address broader issues such as gender equality and social justice.
OpenAI acknowledges these challenges. According to a spokesperson, blind spots and biases are just part of the fabric of AI systems. They highlighted these problems as a core problem with existing AI training practices. The implications are serious: users can receive incorrect information that aligns with their biases rather than an objective truth.
Evidence of Bias Against Women
A plethora of research has come out showing obvious bias towards women in outputs from large language models like ChatGPT. UNESCO has found “clear and unequivocal examples of bias against women in generated content,” shining a light on the systemic, discriminatory factors that are at play.
Brown goes deeper to describe the nature of this bias. She points to issues resulting from “biased training data, biased annotation practices, [and] biased taxonomy design.” Specifically, these elements can contribute to imbalanced portrayals of gender, reinforcing damaging stereotypes of such.
Moreover, the model’s responses may vary according to dialects that users speak. Speakers of African American Vernacular English (AAVE) tend to lack this type of subtle appreciation in discussions. They might not get the same amount of respect that speakers of standard English would receive. This lack of standardization may disaffect some demographics and make for poor communication.
