Artificial intelligence run amok has the potential for lurking biases. Reuters reports Amazon’s machine-learning specialists made a troubling discovery that their new recruiting engine had a bias against women.
Automation factors into processes across Amazon’s business from operations to pricing. The team also uses artificial intelligence to tackle hiring and had been building tools for this aim since 2014. This experimental hiring AI tool rated job candidates on a 5-star scale similar to the product ratings Amazon shoppers give.
“Everyone wanted this holy grail,” said one of the specialists. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
AI learned a preference for male candidates; subsequently, resumes with the word “women’s” were discounted so where those of graduates of two, unspecified all-women’s colleges.
Evidence of gender bias for software developer and other technical job candidates emerged as early as 2015 and was the result of the computer models relying on an inadvertently male-heavy resume pool drawn from a decade-long period. AI learned a preference for male candidates; subsequently, resumes with the word “women’s” were discounted so where those of graduates of two, unspecified all-women’s colleges.
While Amazon made updates to prevent bias for these particular terms, no safeguards were made against other forms of discrimination, according to the team. Amazon eventually scrapped the project in 2017, disappointed with the results. Anonymous sources also revealed Amazon recruiters took ratings into consideration as only one part of their hiring process.
In response, Amazon had no comment on the tool’s issues of bias, but told Reuters the tool “was never used by Amazon recruiters to evaluate candidates” with no further comment. The company did not dispute that recruiters saw these generated recommendations.
This saga illustrates the challenges of machine learning in automating the hiring process. A 2017 survey by CareerBuilder found that some 55 percent of U.S. human resources managers said artificial intelligence, or AI, would be a regular part of their work within the next five years. Machine learning has a ways to go for this application, experts note. “How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable - that’s still quite far off,” explained Nihar Shah, assistant professor at Carnegie Mellon.
Amazon’s experiment coincided with a hiring spree and inexpensive computing power that made machine learning appealing. The company had formed an engineering team in Edinburgh to create an AI sourcing tool. Unfortunately, among other issues, this tool learned to favor masculine language (words such as “executed” and “captured”).
While Amazon’s foray with AI in hiring went south, other companies are undeterred and believe automation can even force employers to look beyond elite credentials. Companies such as Goldman Sachs and Microsoft have used AI in their recruiting processes.
AI is no substitute for humans when it comes to recruiters, cautions John Jersin, vice president of LinkedIn Talent Solutions.
“I certainly would not trust any AI system today to make a hiring decision on its own,” Jersin said. “The technology is just not ready yet.”
The opacity surrounding the technology has also drawn the ire of organizations such as the American Civil Liberties Union who are pushing for algorithmic fairness. Critics argue that even being aware of discrimination due to automated hiring would be a challenge given the nature of the technology.
Amazon has not cut AI entirely from its recruiting. Its AI now takes the form of a “much-watered down version” of the recruiting engine that handles basic tasks such as de-duplication, according to an anonymous source. The next AI recruiting experiment rolling out from the Edinburgh team is taking on a new challenge: diversity.