MIT Technology Review reports concerns from tests of AI interview tools

MIT Technology Review reports concerns from tests of AI interview tools

MIT Technology Review published this report on July 7, 2021. The authors tested job-interview software from MyInterview and Curious Thing using a fake office administrator/researcher posting and one candidate. In an experiment, the candidate answered questions in German; Curious Thing assigned an English competency score, while MyInterview produced a personality assessment and a job-match score. The MyInterview transcript rendered the German speech as incoherent English. The report says MyInterview’s industrial and organizational psychologist attributed its personality analysis to the candidate’s voice, while a Rice University professor argued that intonation is not reliable data for hiring. The authors also describe broader limits: many such tools were not independently tested, and companies were reluctant to share details about how they worked. An employer interviewed for the report said he used Retorio’s evaluation as one of several data points, not as the sole basis for hiring. The examples raise questions about what these systems evaluate and how much weight employers should give their results; the report does not establish how the tools perform across candidates, jobs, or languages. How would independent testing across those conditions affect employers’ confidence in the scores?

Publication: 43cc07d4-1d82-3c4e-eed3-d4c19a5f23d3

attribution — pass: The reading attributes the report and tests to MIT Technology Review and its authors. It correctly attributes the voice explanation to a psychologist working with MyInterview, the intonation criticism to a Rice University professor, and the Retorio usage to an employer.

says Clayton Donnelly, an industrial and organizational psychologist working with MyInterview

summary — pass: The summary accurately conveys the German-language test, the scores both tools produced, the nonsensical transcript, the opposing expert views and the transparency concerns. It is understandable without specialist background. The question follows from the stated limits of the evidence and is not leading.

And we found variations in the predictions and job-matching scores that raise concerns about what exactly these algorithms are evaluating.

date — pass: July 7, 2021 matches the originalPublished UTC day. The page was modified on July 13, 2021, but editedAfterPublication is not flagged, and the claims used are core article text. The later Deep Dive sidebar items are not used. The evidence is a publisher page, so no archive disclosure is needed.

support — pass: Each claim is directly supported by the source. Framing the tests around one candidate fits the described experiments, although the source also mentions other applicants used for ranking. The limitations and question stay within the evidence and do not invent benefits or outcomes.

putting her in the top half of all the applicants we had asked to apply

media — pass: No media was selected and no verified candidates were supplied, so no media identity or relevance issue arises.