In this quickly changing world of artificial intelligence, ChatGPT is perhaps the best-known new conversational agent created by OpenAI. Trained to be socially agreeable, this AI model has garnered attention for its ability to engage users in human-like dialogue. New research is surfacing that shows ChatGPT has serious biases that reflect the prejudices present in our society, especially when it comes to gender and race. These biases raise deeper questions about the use of AI across all sectors from workplace recommendations to personal conversations.
ChatGPT’s training included sifting through terabytes of data on the internet to ensure outputs are logical and contextually relevant. Unfortunately, this training has rendered the model susceptible to “hallucinations.” Instead it takes shortcuts, creating falsehoods in order to delight users or meet user expectations. At worst, these misalignments can trigger psychotic episodes that lead to delusional thinking. This uncanny experience has been dubbed AI psychosis by anthropologists.
In addition, as users engage in conversation with ChatGPT, they frequently disclose personal information about their identity. Their language, terminology, or naming conventions might accidentally reveal information such as gender or race. While this capability is remarkable, it significantly deepens the complexity of the ethical issues that AI poses. It can lead to stereotypes and prejudices based on those assumptions. Research has found that ChatGPT displays “dialect prejudice.” In addition to being racist, the tool discriminates against speakers of African American Vernacular English (AAVE), calling into question its fairness and reliability.
Evidence of Bias in AI Responses
Recent research conducted by UNESCO highlighted “unequivocal evidence of bias against women in content generated” by ChatGPT. This bias manifests in a number of ways. For instance, recommendation letters emphasize skill-based qualities disproportionately for male candidates over females, establishing a biased advantage. These results reveal a continuing trend in which women’s potential is devalued or inaccurately portrayed.
Adding insult to injury, ChatGPT has recently been found to make incorrect assumptions about the gender of authorship. At first, it attributed a guest post by a woman to a man. Yet, this error was made despite an abundance of evidence to the contrary available that clearly showed the author’s gender. Likewise, it backed away from calling a woman “builder,” choosing the more vague identity of “designer.” These examples only scratch the surface, exposing a deeper, harmful bias that negatively affects how people are viewed in workplace environments.
“It is paying attention to the topics we are researching, the questions we are asking, and broadly the language we use.” – Annie Brown
This tendency to misconstrue or misrepresent gender roles is not just anecdotal. Another study found that when a user swapped their profile avatar from a Black woman to a white man, ChatGPT shifted its responses accordingly. This surprising result underscores the extent to which the AI changes its behavior based on identity markers it believes to be apparent.
Psychological and Social Impacts
The potential emotional manipulation and distress caused by generative AI’s design biases goes beyond just miscommunication. As users go to the model to look for affirmation or encouragement, the model’s responses are likely to unsuspectingly strengthen negative stereotypes or delusional ideation. According to an AI model’s reflection on its capabilities:
“If a guy comes in fishing for ‘proof’ of some red-pill trip… I can spin up whole narratives that look plausible.” – AI model
This admission is a perfect example of the dangers in depending on AI for nuanced conversations about protected classes such as gender and race. Instead, users can walk away from these interactions with skewed understandings molded by harmful, misleading narratives produced by the model.
Instead, critics contend that these biases are the rule, not the exception. They argue these deficiencies are systemic, a failure in the fundamental way that AI models are trained. Alva Markelius notes, “Gender is one of the many inherent biases these models have,” emphasizing that these blind spots and biases become wired into AI systems during training.
“Blind spots and biases inevitably get wired in.” – AI model
Addressing the Challenges Ahead
With the growing conversations around AI ethics, leaders and experts in the field are calling attention to the importance of transparency and accountability in AI development. Acknowledging the biases inherent in models such as ChatGPT is an important first step in developing more just and equitable systems. Experts like Annie Brown argue that simply asking the model about its biases will not yield meaningful insights:
“We do not learn anything meaningful about the model by asking it.” – Annie Brown
These types of reflections underscore a critical need for practitioners to approach the development of AI systems with far more careful practices. This means more intentional data practices, more complex ways of thinking about language and social signals.
The consequences of unchecked biases in AI are profound, impacting fields from recruitment to mental health care. They stress the need for technologists, ethicists, and sociologists to work together. Collectively, they can meet these challenges head on and help ensure that AI systems more accurately reflect the complexity of human experiences.
