In a recent podcast series called Instaserfs, a former Uber driver named Mansoor gave a chilling description of a new computer-controlled workplace. At first, the company tried to persuade him to take a predatory loan to buy a new car. Clearly, someone in the payroll felt he was at high risk of default. Secondly, Uber never responded to him personally - he only sent text messages and emails. This style of oversight was a series of take-or-leave ultimatums—a digital boss coded in advance.
Then the company suddenly took a large share of the revenue from him and other drivers. And finally, what seemed to Mansour the most outrageous: his job could be fired without warning if several passengers gave him a one-star rating, as this could reduce his average score below 4.7. Uber doesn't have a real appeal or other due process for a ratings system that can instantly put a driver out of a job, he said — it just counts the numbers.
Mansour's story compresses long-standing trends in lending and employment - and is by no means unique. Online retailers live in fear of "Google's death penalty" - a sudden, mysterious drop in search engine rankings if they do something that Google's spam detection algorithms see as a scam. Applicants for jobs at Walmart in the US and other major companies are subjected to cryptic "personality tests" that manipulate their responses in unknown ways. And white-collar workers are faced with resume sorting software that can downgrade or completely ignore their qualifications. One algorithmic CV analyzer found that all 29,000 people who applied for a "reasonably standard engineering position" were unqualified.
The childhood of the Internet is over. As the online space matures, Facebook, Google, Apple, Amazon, and other powerful corporations are setting rules that govern competition between journalists, writers, programmers, and e-commerce firms. Uber, Postmates and other platforms add a layer of code to jobs like driving and maintenance. Cyberspace is no longer an escape from the "real world". Now it is the power that controls it through algorithms: recipe-like sets of instructions for solving problems. From Google searches to OkCupid matchmaking, the software arranges and weighs hundreds of variables in clear and simple interfaces, taking us from query to solution. Such answers are based on complex mathematics, but they are hidden from prying eyes either due to secrecy established by law, or due to complexity that outsiders cannot figure out.
Algorithms are becoming increasingly important because businesses that are rarely considered high-tech have learned from the success of the internet giants. Following Jeff Jarvis' "What Would Google Do" advice, they collect data from both employees and customers, using algorithmic decision-making tools to separate the wishful from the disposable. Companies can analyze your voice and credit history when you call them to determine if you qualify as an "ideal client" or just a "waste" to be treated with disdain. Epagogix advises film studios on which scripts to buy based on how they compare to past successful scripts. Even winemakers make algorithmic judgments based on statistical analysis of the weather and other characteristics of good and bad vintage years.
For wines or films, the stakes are not very high. But as algorithms begin to affect important employment, career, health, credit, and education opportunities, they deserve a closer look. US hospitals use big data-driven systems to determine which patients are at high risk, and data far beyond traditional medical records is used to make these determinations. IBM now uses algorithmic scoring tools to sort employees around the world according to profitability criteria, but spares top managers the same obsessive tracking and ranking. In government, algorithmic risk assessment could also lead to longer sentences for convicted or no-fly lists for travelers. Credit scoring provides billions of dollars of credit, but scoring methods remain opaque.
This trend towards using more data in more obscure ways for ranking and scoring may seem inevitable. However, the precise development of such computerized sorting methods is far from automatic. Search engines, for example, are a paragon of algorithmic technology, but their current appearance owes much to legal interventions. For example, thanks to
In a 2002 FTC action, US consumer protection laws require ads to be separated from unpaid "organic" content. In a world where media companies are constantly trying to blur the distinction between content and native advertising, this law matters. European Union regulators are now trying to ensure that irrelevant, outdated, or biased material doesn't haunt individuals' "search by name" results - a critical task in an era when so many potential employers search Google for those they are considering for a job. . The EU has also urged search engines to take human dignity into account, for example by endorsing the request "a victim of physical abuse [who] requested that results describing an attack be removed for searches against her name."
Such disagreements spawned the movement for algorithmic accountability. At the 2013 Governance of Algorithms conference at New York University, a community of scientists and activists came together to critique the results of algorithmic processes. Today, these scholars and activists are promoting an active dialogue about algorithmic accountability, or #algacc for short. Like the “access to knowledge” (A2K) mobilization in the 2000s, #algacc draws attention to a key social justice issue of the 2010s. Someone in the business world would rather see the work of this community ended before it even started. Representatives and lobbyists of insurance companies, banks and large businesses usually believe that key algorithms deserve the ironclad protection of trade secrets, so they can never be verified (let alone criticized) by outsiders. But lawyers have already faced such a fence and will do it again.
Regulators can make data-driven firms more accountable. But first, they need to be aware of the many reasons why business computing can go wrong. The data used may be inaccurate or inappropriate. Algorithmic modeling or analysis may be biased or incompetent. And the use of algorithms is still opaque in many critical sectors - for example, we may not even know if our employers evaluate us according to secret formulas. In fact, however, at every stage of algorithmic decision-making, simple legal reforms can move basic protections (such as due process and anti-discrimination law) into the age of computing.
Everyone knows how inaccurate credit reports can be and how difficult it is to correct them. But credit histories are actually one of the most regulated areas of the data economy, and there are plenty of protections available to savvy consumers. Much more worrying is the shady world of thousands of largely unregulated data brokers who create profiles of people, profiles created without people's knowledge, consent, and often without the right to review or correct. One random insult to you can end up in a random database without your knowledge, and then continue to fill out hundreds of other digital dossiers purporting to report your health, finances, competence, or criminal record.
This new digital underworld can destroy reputation. One woman was falsely accused of dealing methamphetamine by a private data broker, and it took her years to fix the situation — years during which she was denied housing and credit by landlords and banks. Government databases can be even worse, such as in the US, where innocent people are marked with "suspicious activity reports" (SARs) or contain inaccurate arrest records. Both problems have haunted unlucky citizens for years. Data voraciousness on the part of both the government and market participants means that surrogate reports can circulate quickly.
No matter how much knowledge about every moment of a worker's life is added to the bottom line, a democratic society must resist it.
When false, damaging information can spread instantly between databases, but takes months or years of running around and propaganda to fix, the data architecture is fundamentally defective. Future reputation systems should be able to eliminate stigma as quickly as they contribute to its spread. This is not an unsolvable problem: in 1970, the US Congress passed the Fair Credit Reporting Act to regulate the data collection practices of credit bureaus. Expanding and modernizing its defenses will embed accountability, justice and redress mechanisms into data systems that are currently connected only to quick profits, not to people or citizens.
Data collection problems go beyond imprecision. Some data processing practices are too invasive to be allowed in a civilized society. Even if job seekers are so desperate for a job that they allow themselves to be filmed in the bathroom as
As a condition of employment, the privacy law should put an end to such transactions. Digital data collection can also cross the line. For example, a former employee of international wire transfer service Intermex claims she was fired after she disabled an app that allowed the firm to constantly track her whereabouts.
Note that the employer may have business reasons for such tracking other than voyeurism; she might discover that employees who always get home by 8 pm tend to perform better the next day, and then gradually introduce incentives or even require that behavior from their entire workforce. No matter how much knowledge about every moment of a worker's life is added to the bottom line, a democratic society must resist it. There should be some separation between work and non-work life.
The limits on data collection will frustrate big data connoisseurs. ZestFinance's CEO proudly declared that "all data is credit data", meaning predictive analytics can take almost any information about a person, analyze if it matches the characteristics of people known to be creditworthy, and extrapolate accordingly. Such data may include sexual orientation or political opinions. But even if we knew that George W. Bush supporters were more likely to be late with their bills than John Kerry voters, do we really trust our banks or loan officers? Is this the knowledge they must have? Marriage counseling can be seen as a signal of impending instability and lead to higher interest rates or lower credit limits - one American company, CompuCredit, has already settled (no wrongdoing) a lawsuit for just that. But such intimate information should not be monetized. Too many big data mavens strive to analyze all available information, but when their feverish dreams of a perfectly known world conflict with core values, they must give in.
While most privacy advocates focus on the issue of data collection, the threat posed by reckless, poor, or discriminatory analysis may be more serious. Consider the “probable employment success score,” which is highly dependent on the applicant's race, zip code, or lack of a current job. Each of these pieces of data can be innocent or even relevant in the right context. (For example, Entelo is trying to match minority bidders to firms that want more diversity.) But they also need to be scrutinized.
Consider racism first. There is a long and disturbing history of discrimination against minorities. Existing employment discrimination laws already prohibit bias and can result in heavy fines. So, many proponents of algorithmic decision making say, why bother with our new technology? Discrimination in any form - personal, technological, whatever - is already prohibited. This is naive at best. Algorithmic decision-making processes collect personal and social data from a society with a problem of discrimination. Society is rife with data that are often mere substitutes for discrimination, such as a zip code or zip code.
Consider a variable that at first glance appears to be less charged: months since last job. Such data can help employers who favor workers who move quickly from job to job or discriminate against those who need time off to recover from an illness. Concerned about the potentially unfair impact of such considerations, some jurisdictions have banned employers from posting ads that they need help inviting the unemployed not to apply. It's a commendable political move, but whatever its merits, what teeth will it have if employers never see resumes excluded by an algorithm that blacklists those whose last entry is more than a few months old? Big data can easily become a sophisticated tool to deepen already prevalent forms of unfair disadvantage.
Law enforcement officers of the future may find it difficult to study all the variables that influence credit and employment decisions. Protected by trade secrets, many algorithms remain impenetrable to outside observers. As they attempt to uncover them, the parties may be confronted by Catch-22. With a legitimate interest in stopping "fishing expeditions", the courts are likely to grant disclosure requests only if the plaintiff has amassed some evidence of discrimination. But if the key decision maker was a faceless black box algorithm, what about
Were there initial suspicions of discrimination?
Indeed, since the Equal Credit Opportunity Act (1974), US regulators have often encouraged businesses to use algorithms to make decisions. Regulators want to avoid the irrational or subconscious biases of decision makers, but of course decision makers have developed algorithms, modified the data and influenced their analysis. No "layer of code" can create a "plug and play" level playing field. Politics, human judgment and law will always be needed. Algorithms will never offer an escape from society.
Governments must ensure that the algorithms they promote serve, and do not run counter to, their stated goals. The subprime crisis offers a good example of past legal failures and an innovative solution to this problem. Rating agencies such as Moody's and S&P have used algorithmic credit ratings to assign questionable mortgage-backed securities (MBS) an AAA status, the highest rating. These ersatz imprimaturs, in turn, attracted a flood of subprime money. Critics argue that the agencies have changed their valuation methods to attract more customers from those who sell MBS. The Triple-A ratings after the method change could mean something completely different from before, but many investors lacked knowledge of the transition.
The government can deny contracts to companies that use secret algorithms to make employment decisions.
To address this issue, the Dodd-Frank Act requires rating agencies to disclose material changes to their practices. This openness helps those involved in the markets understand the “innards” of the AAA rating, rather than mindlessly assuming that it has always been and will always provide a certain benchmark of reliability. As any investor will tell you, information is power, and credit scores aren't necessarily information - they're just a transcript.
While credit scores evaluate the value of securities, algorithmic consumer scoring evaluates people on any number of parameters, including (but not limited to) creditworthiness. As the 2014 World Privacy Forum's Assessment of America report showed, there are thousands of such assessments. When an important decision maker decides to use it, he or she should explain to the people who were ranked and evaluated exactly what data was used, how it was analyzed, and how potential errors, biases, or violations of the law can be identified and corrected. or disputed. In areas ranging from banking and employment to housing and insurance, algorithms could very well play a decisive role in deciding who gets hired or fired, who gets promoted and who gets demoted, who gets a 5% or 15% interest rate. . People should be able to understand how they work or don't work.
The growing "predictive analytics" industry will object to this proposal, arguing that its way of ranking and evaluating people deserves absolute trade secret protection. Such intellectual property is well protected by applicable law. However, the government may condition funding on the use or disclosure of data and methods used by its contractors. The government's power to use its leverage as a buyer is enormous, and it can deny contracts to companies that, say, used secret algorithms to make hiring decisions or based credit decisions on dubious data.
In the US, it's time for the federal budget to reward the creation of accountable algorithmic decision making, and not just pay for whatever tools its contractors come up with. We will not tolerate parks studded with eavesdropping equipment that records every conversation of a wheelchair, or denying toilets to those labeled as "risk of vandalism" through secret software. We should have similar expectations of privacy and fair treatment of the thousands of algorithmic systems that the government directly or indirectly funds every year.
CSome participants in clinical trials have found that people who own minivans, don't have small children, and subscribe to many cable TV channels are more likely to be obese. At least in their databases, and possibly others, van drivers, childless cable lovers suddenly become a new group - "more obese" - and this conclusion is a new piece of data created about them.
Such a conclusion in itself may not be worth much. But once people are identified in this way, they can be easily combined and recombined with other lists—say, plus size shoppers or frequent fast food shoppers—that
rye confirm the conclusion. Facebook's new algorithm instantly categorizes people in photos by body type or posture. The holy grail of algorithmic reputation is the most complete database of every person, combining credit, telecommunications, location, retail, and dozens of other data streams into a digital twin.
No matter how confident they are in our height, weight, or health status, it is convenient for data collectors to keep the classification in the fog. In principle, a person could bring a defamation suit against a data broker who falsely claimed that the person concerned was diabetic. But if a broker instead chooses a more vague classification, such as "member of a household with diabetes," it looks to the courts much more like opinion than fact. Opinions are much harder to prove to be defamatory - how can you demonstrate without a doubt that your family is in no way "concerned about diabetes"? But a softer classification can lead to exactly the same disadvantageous results as a more rigid fact-based classification.
Similar arbitrage strategies could attract other companies as well. For example, if an employer tells you that they are not hiring you because you are diabetic, this is clearly illegal. But what if there is some euphemistic terminology that rates your "reliability" as an employee? Even if the score is partly based on health-related information, it can be nearly impossible to prove because candidates almost never know what is behind an employer's decision not to interview or give them a job. The employer may even claim that they do not know what is included in the bill. Indeed, at some point in the hiring or evaluation process, job seekers are likely to run into managers or HR staff who don't really know what constitutes a "reliability" rating. When so much anti-discrimination legislation requires plaintiffs to prove intent to use prohibited classifications, ignorance can be bliss.
It will be much easier to manage these worrisome opportunities before they become widespread, ubiquitous business practices. The Equal Employment Opportunity Commission (EEOC) addresses disputes arising from employers' personality tests involving questions that appear to be looking for mindsets related to mental illness but not related to bona fide job qualifications or performance. This research should continue and expand into a growing class of algorithmic estimates of past or likely outcomes. In some cases, simply disclosing and analyzing algorithmic estimates is not enough to make them fair. Rather, their use may need to be banned in important contexts ranging from employment to housing, credit and education.
When problems with algorithmic decision making come to light, large firms tend to play the game of musical expertise. Lawyers speak and are told that they do not understand the code. Coders talk and are told they don't understand the law. Economists, sociologists, and ethicists hear variations on both objections.
Algorithmic accountability is a pressing global issue that motivated and mobilized experts need to support.
In truth, it took a combination of computational, legal, and social science skills to unearth each of the examples discussed above—alarming collection, poor or biased analysis, and discriminatory use. Collaboration between experts in different fields is likely to lead to even more important work. For example, legal scholars Ryan Kahlo of the University of Washington and James Grimmelmann of the University of Maryland, along with other ethicists, have proposed frameworks for assessing the algorithmic manipulation of content and people. Based on well-established social science empirical methods, their models can and should inform the regulation of firms and governments through algorithms.
Empiricists may be disappointed by the "black box" of algorithmic decision making; they can work with legal scholars and activists to uncover certain aspects of this (through freedom of information and honest data practices). Journalists have also teamed up with computer scientists and sociologists to expose new technologies for collecting, analyzing and using data that violate privacy, as well as push regulators to crack down on the worst offenders.
Researchers are moving beyond analyzing existing data and joining coalitions of observers, archivists, open data activists, and public interest advocates to provide a more balanced set of "raw materials" for analysis, synthesis, and critique. Sociologists and other professionals should devote
Become a vital long-term project to ensure that algorithms provide reliable and up-to-date documentation; otherwise, governments, banks, insurance companies, and other major powerful players will create and own more and more inaccessible data about society and people. Algorithmic accountability is a big project that requires the skills of theorists and practitioners, lawyers, sociologists, journalists and others. This is an urgent global cause in which interested and mobilized experts are seeking support.
The world is full of algorithmic solutions. One erroneous or discriminatory piece of information can ruin someone's job or credit prospects. It is imperative that citizens have the ability to view and regulate the digital dossiers of business giants and government agencies. Even if one believes that no information should be "deleted"—that every slip and error that anyone makes should be permanently recorded—that still leaves important data processing decisions to be made. Algorithms can be made more accountable by respecting the rights to justice and dignity that generations have fought for. The challenge is not technical, but political, and the first step is a law that allows people to see and challenge what the algorithms say about us.