Presented at
33C3 (2016),
Dec. 27, 2016, 4:45 p.m.
(30 minutes).
Artificial intelligence and machine learning are in a period of astounding growth. However, there are concerns that these technologies may be used, either with or without intention, to perpetuate the prejudice and unfairness that unfortunately characterizes many human institutions. We show for the first time that human-like semantic biases result from the application of standard machine learning to ordinary language—the same sort of language humans are exposed to every day. We replicate a spectrum of standard human biases as exposed by the Implicit Association Test and other well-known psychological studies. We replicate these using a widely used, purely statistical machine-learning model—namely, the GloVe word embedding—trained on a corpus of text from the Web. Our results indicate that language itself contains recoverable and accurate imprints of our historic biases, whether these are morally neutral as towards insects or flowers, problematic as towards race or gender, or even simply veridical, reflecting the status quo for the distribution of gender with respect to careers or first names. These regularities are captured by machine learning along with the rest of semantics. In addition to our empirical findings concerning language, we also contribute new methods for evaluating bias in text, the Word Embedding Association Test (WEAT) and the Word Embedding Factual Association Test (WEFAT). Our results have implications not only for AI and machine learning, but also for the fields of psychology, sociology, and human ethics, since they raise the possibility that mere exposure to everyday language can account for the biases we replicate here.
There is no Alice and Bob in this talk. This talk is intended for an audience that genuinely cares for humanity and believes in equality while supporting fairness and acts against discrimination. This talk might not be interesting for folks who promote exclusion while discouraging diversity. Many of us have felt excluded in certain situations because of our gender, race, nationality, sexual orientation, disabilities, or physical appearance. This talk aims to communicate how big data driven machine learning is pushing the society towards discrimination, unfairness, and prejudices that harm billions of people every single day.
This year, I will not talk about de-anonymizing programmers, re-identifying underground forum members, or anonymous writing. I will be talking about a human right, namely equality, and the issue of unfairness which happens to be embedded in machines that make decisions about our future, what we see and read, or whether we go to prison or not.
Machine learning models are widely used for various applications that end up affecting billions of people and Internet users every day. Random forest classifiers guide the U.S. drone program to predict couriers that can lead to terrorists in Pakistan. Employers use algorithms, which might be racist, to aid in employment decisions. Insurance companies determine health care or car insurance rates based on machine learning outcomes. Internet search results are personalized according to machine learning models, which are known to discriminate against women by showing advertisements with lower salaries, while showing higher paying job advertisements for men. On the other hand, natural language processing models are being used for generating text and speech, machine translation, sentiment analysis, and sentence completion, which collectively influence search engine results, page ranks, and the information presented to all Internet users within filter bubbles. Given the enormous and unavoidable effect of machine learning algorithms on individuals and society, we attempt to uncover implicit bias embedded in machine learning models, focusing particularly on word embeddings. We show empirically that natural language necessarily contains human biases, and the paradigm of training machine learning on language corpora means that AI will inevitably imbibe these biases as well.
We look at “word embeddings”, a state-of-the-art language representation used in machine learning. Each word is mapped to a point in a 300-dimensional vector space so that semantically similar words map to nearby points. We show that a wide variety of results from psychology on human bias can be replicated using nothing but these word embeddings. We primarily look at the Implicit Association Test (IAT), a widely used and accepted test of implicit bias. The IAT asks subjects to pair concepts together (e.g., white/black-sounding names with pleasant or unpleasant words) and measures reaction times as an indicator of bias. In place of reaction times, we use the semantic closeness between pairs of words. In short, we were able to replicate every single implicit bias result that we tested, with high effect sizes and low p-values.
These include innocuous, universal associations (flowers are associated with pleasantness and insects with unpleasantness), racial prejudice (European-American names are associated with pleasantness and African-American names with unpleasantness), and a variety of gender stereotypes (for example, career words are associated with male names and family words with female names). We look at nationalism, mental health stigma, and prejudice towards the elderly. We also look at word embeddings generated from German text to investigate prejudice based on German data. We do not cherry pick any of these IATs, they have been extensively performed by millions of people from various countries and they are also available for German speakers (https://implicit.harvard.edu/implicit/germany/).
We go further. We show that information about the real world is recoverable from word embeddings to a striking degree. We can accurately predict the percentage of U.S. workers in an occupation who are women using nothing but the semantic closeness of the occupation word to feminine words! These results simultaneously show that the biases in question are embedded in human language, and that word embeddings are picking up the biases.
Our finding of pervasive, human-like bias in AI may be surprising, but we consider it inevitable. We mean “bias” in a morally neutral sense. Some biases are prejudices, which society deems unacceptable. Others are facts about the real world (such as gender gaps in occupations), even if they reflect historical injustices that we wish to mitigate. Yet others are perfectly innocuous. Algorithms don’t have a good way of telling these apart. If AI learns language sufficiently well, it will also learn cultural associations that are offensive, objectionable, or harmful. At a high level, bias is meaning. “Debiasing” these machine models, while intriguing and technically interesting, necessarily harms meaning.
Instead, we suggest that mitigating prejudice should be a separate component of an AI system. Rather than altering AI’s representation of language, we should alter how or whether it acts on that knowledge, just as humans are able to learn not to act on our implicit biases. This requires a long-term research program that includes ethicists and domain experts, rather than formulating ethics as just another technical constraint in a learning system.
Finally, our results have implications for human prejudice. Given how deeply bias is embedded in language, to what extent does the influence of language explain prejudiced behavior? And could transmission of language explain transmission of prejudices? These explanations are simplistic, but that is precisely our point: in the future, we should treat these as “null hypotheses’’ to be eliminated before we turn to more complex accounts of bias in humans.
Presenters:
-
Aylin Caliskan
Aylin Caliskan is a Postdoctoral Research Associate and a CITP Fellow at Princeton University. Her work on the two main realms, security and privacy, involves the use of machine learning and natural language processing. She currently works on big-data-driven discrimination and inference through machine learning. She also has ongoing research on privacy preserving information disclosure and contextual integrity.
In her previous work, she demonstrated that de-anonymization is possible through analyzing linguistic style in a variety of textual media, including social media, cyber criminal forums, source code, and executable binaries. She is extending her work to develop countermeasures against de-anonymization. Aylin's other research interests include designing privacy enhancing tools to prevent unnecessary private information disclosure while quantifying and characterizing human privacy behavior. She holds a PhD in Computer Science from Drexel University and a Master of Science in Robotics from the University of Pennsylvania.
Links:
Similar Presentations: