Artificial Intelligence is Only as Fair as Its Data

As artificial intelligence becomes increasingly involved in decisions about employment, housing, healthcare, and online content, it risks reproducing inequalities already embedded in society.

Artificial intelligence (AI) increasingly shapes decisions about who gets hired, who qualifies for housing, and who has access to medical care. AI can appear more objective than human decision-making because it relies on data. However, the outcome of using this data is not necessarily objective.

In recent years, governments at both local and state levels have begun responding to algorithmic discrimination. New York City’s Local Law 144 requires certain automated employment decision tools to undergo independent bias audits, while California has clarified that existing employment discrimination protections also apply when employers use automated decision systems. At the federal level, lawmakers have proposed the Artificial Intelligence Civil Rights Act of 2025 (AI Civil Rights Act), which would go further than currently implemented local laws by establishing protections against algorithmic discrimination in consequential decisions involving employment, housing, healthcare, education, and credit.

Housing provides a clear example of how AI can reinforce existing discrimination. Companies increasingly use algorithms in mortgage lending, tenant screening, property valuation, and housing advertisements. Although these systems may not directly consider race, they can rely on factors such as ZIP code, credit history, income, and previous housing records that are closely connected to existing racial inequalities.

This issue is especially concerning given the history of redlining in the United States. For decades, discriminatory housing policies restricted where many Black families could purchase homes or receive mortgages. The effects of these policies did not disappear when redlining became illegal. They contributed to lasting racial disparities in homeownership, household wealth, property values, and access to credit. When developers train AI systems on data shaped by these disparities, those systems can learn that certain neighborhoods or financial characteristics are associated with greater “risk.” As a result, an algorithm can reproduce patterns similar to past housing discrimination without ever explicitly considering an applicant’s race.

AI can therefore create a new version of an old problem. Instead of a lender drawing a red line around a neighborhood, an algorithm may assign applicants different levels of risk based on hundreds of seemingly neutral variables. These risk assessments can make discrimination more difficult to identify because the decision appears to come from data rather than human prejudice.

AI regulation in housing should focus not only on whether an algorithm directly uses race, but also on whether the algorithm produces discriminatory outcomes. Policymakers should require testing across racial and demographic groups and greater scrutiny of variables that may act as proxies for protected characteristics. Without these protections, AI risks reinforcing the inequalities created by earlier forms of housing discrimination rather than correcting them.

Beyond race, AI-driven discrimination can also exacerbate inequality for lesbian, gay, bisexual, transgender, and queer (LGBTQ) people. Across social media platforms, automated moderation and recommendation systems shape what information users can access. Automated content moderation systems sometimes misclassify LGBTQ content as sexual, explicit, or inappropriate, leading legitimate content to be removed, restricted, or made harder to find. This misclassification can have an especially large impact on LGBTQ youth, who may rely on online spaces to find information, community, and support that they cannot easily access at home or in their schools. 

Policies addressing algorithmic discrimination should therefore consider more than hiring or lending algorithms. Greater transparency around automated content moderation, testing for disparate effects on marginalized communities, and meaningful human review of automated removals could help prevent AI systems from restricting legitimate information simply because it concerns LGBTQ identities.

AI has the potential to make decisions faster and possibly fairer. However, using an algorithm does not automatically remove human bias. If AI learns from data produced by an unequal society, it can reproduce those inequalities while making them appear objective.

Future AI policy should focus not only on identifying discrimination but on preventing and correcting it. Policymakers should require evaluations of high-stakes AI systems across demographic groups, and regulators and independent auditors should have enough information—such as training-data composition and performance metrics—to identify gaps in the data used to develop and test them. Most importantly, discovering substantial disparities should require companies to address them rather than simply disclose them.

Image Credit

Tara Winstead, Public Domain via Pexels