We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether we can get a job or a loan, how much we pay for health insurance—are being made not by humans, but by machines. In theory, this should lead to greater fairness: Everyone is judged according to the same rules.
But as mathematician and data scientist Cathy O’Neil reveals, the mathematical models being used today are unregulated and uncontestable, even when they’re wrong. Most troubling, they reinforce discrimination—propping up the lucky, punishing the downtrodden, and undermining our democracy in the process. Welcome to the dark side of Big Data.
Cathy O’Neil is an American mathematician and the author of the blog mathbabe.org. In 2016, her book Weapons of Math Destruction was published and long-listed for the National Book Award for Nonfiction. O’Neil attended UC Berkeley as an undergraduate, received a Ph.D. in mathematics from Harvard University in 1999, and afterward held positions in the mathematics departments of MIT and Barnard College, doing research in arithmetic algebraic geometry.
Cathy also casts in CODED BIAS, a documentary that explores the fallout of MIT Media Lab researcher Joy Buolamwini’s discovery that facial recognition does not see dark-skinned faces accurately, and her journey to push for the first-ever legislation in the U.S. to govern against bias in the algorithms that impact us all.
Chapter 1 – Bomb Parts: What is a Model?
Readers are introduced to the concept of mathematical models using everyday examples, such as baseball statistics and meal planning, and discuss factors that can influence these models. It then discusses how simple models, like the presence of smoke indicating a fire, can be problematic when applied to humans and issues like race and class. It concludes by defining the three elements of a model that turn it into a weapon of math destruction (WMD): opacity, scale, and damage.
Chapter 2 – Shell Shocked: My Journey of Disillusionment
O’Neil discusses her career, beginning with her time as a quant at the hedge fund D.E. Shaw, (“quant” is short for “quantitative analyst”–you can read more about what they do on Investopedia), the type of work she did while employed there, a description of the industry in general, and her realization of her own participation in the creation and implementation of WMDs. O’Neil changed careers to try in an attempt to fix some of these WMDs.
Chapter 3 – Arms Race: Going to College
The evolution of the U.S. News & World Report college ranking scores, how they affect specific colleges, how some colleges try to game the system, and the rankings’ impact on higher education overall are discussed within this chapter. It details the process of applying to college and how certain privileges, like wealth and legacy, can affect a student’s college application process.
Chapter 4 – Propaganda Machine: Online Advertising
This chapter explores online targeted advertising and machine learning, paying particular attention to how targeted advertising is used on vulnerable populations. One of the major examples of this phenomenon highlighted in this chapter is the recruitment strategies used by for-profit universities.
Chapter 5 – Civilian Casualties: Justice in the Age of Big Data
Crime prediction software and the criminal justice system, in general, are the main themes contained within this chapter. It addresses how crime prediction software affects policing, which populations are disproportionately affected by it (and which aren’t), and other issues with the software’s algorithm. (Content warning: some of these discussion questions and the ensuing conversations may mention violence against POC, including police brutality and the murder of Black people.)
Chapter 6 – Ineligible to Serve: Getting a Job
Within this chapter, O’Neil examines how human biases can affect the job application process and how some companies try to use machines to eliminate bias.
Chapter 7 – Sweating Bullets: On the Job
This chapter discusses how WMDs are used in the workplace, from tracking call center employee productivity to the effectiveness of K-12 teachers, what these models get wrong, and what is missing from their metrics.
Chapter 8 – Collateral Damage: Landing Credit
An exploration of how credit scores and e-scores are used for applying for a bank loan or a job and the flaws with that system are featured in this chapter. It also discusses data brokers and the sale of consumer profiles, and the mistakes that are easily made within that system.
Chapter 9 – No Safe Zone: Getting Insurance
WMDs affect how people get insurance and how much they pay for it. This chapter looks back to the discussion on e-scores from the previous chapter and points out how that model can also affect insurance rates.
Chapter 10 – The Targeted Citizen: Civic Life
This chapter discusses how engagement in our communities is modeled and targeted, whether through something as seemingly innocuous as social media to political microtargeting.
