I Ran the Same Resume Through an AI Filter Three Times. The Scores: 90, 74, 88

Right now, the hardest part of job hunting isn’t impressing a hiring manager. It’s getting your resume in front of a human at all. Before any person reads your application, an AI filter scores it first. And a recent experiment found that feeding the same resume into the same system three times produced scores of 90, 74, and 88 — a 16-point swing. If the line between a callback and the reject pile is essentially a coin flip, can we still call that a fair evaluation?

The First Interviewer Is Code, Not a Person

Quick definitions first. An ATS — Applicant Tracking System — is software that automatically sorts and scores the thousands of resumes a company receives. The bigger the company, the more certain it is using one. Workday, Greenhouse, and Taleo are household names in HR departments, even if job seekers never see them.

The problem is that the ATS now functions as your first-round interviewer. The era of a recruiter reading every application is over. AI screens first, and only resumes above a certain score get passed up the chain. Score low, and your resume disappears before a single human lays eyes on it.

Which raises the obvious question: can you trust that score?

Same Input, Different Output: There’s No Reproducibility

Developers who pulled apart open-source resume-screening tools keep flagging the same flaw. The systems aren’t reproducible.

Reproducibility just means the same input should yield the same output — punch 2 plus 2 into a calculator and you always get 4. LLM-based resume scorers don’t behave that way.

There are two reasons. First, many of these tools call a large language model under the hood, and unless a setting called temperature is pinned to zero, the answer shifts a little every time. Temperature controls how much randomness the model injects into its responses. Great for creative writing. Poison for scoring. Second, these tools tend to dump the entire resume text into a prompt, so trivial differences — document length, where the line breaks fall — make the result wobble.

Put bluntly: whether you advance might depend less on the quality of your resume than on the dice the model happened to roll that day.

Evaluation Reduced to a Keyword Game

Crack open the code and something else surfaces. A large share of the screening logic ultimately leans on keyword matching.

The approach treats the overlap between the words in the job posting and the words in your resume as a major axis of the score. It sounds reasonable. It’s riddled with holes. Phrase the same skill differently and you get nothing. If the listing says “Node.js” but you wrote “Node,” the match can simply fail.

The nastiest side effect is that this structure is trivial to game. Some applicants stuff keywords into their resumes in white text on a white background. Invisible to a human reader, perfectly legible to an AI scraping the text. The skill that decides the outcome isn’t engineering — it’s tricking the parser. When the honest applicant is the one who loses, that’s not a filter. It’s a trap.

So Why Does Open Source Matter Here?

Here’s the part worth sitting with. Every flaw above almost certainly exists in commercial ATS products too — possibly worse. We just can’t verify it.

Commercial systems keep their internal logic as a trade secret. How they score, what biases are baked in — none of it is visible from the outside. Rejected candidates don’t even learn why.

That’s exactly where open source earns its keep. The code is public, so anyone can inspect it and say, “this score makes no sense.” The scoring instability got exposed precisely because the code was open. The irony writes itself: the system caught with its flaws showing is more honest than the one hiding them. The real danger lives inside the black boxes nobody can pry open.

What We Should Actually Be Checking

This isn’t a bug report. It’s a case study in what verification looks like when you point AI automation at decisions that shape someone’s life.

Three things, at minimum. First, reproducibility. The same resume should produce the same score, every time. Forcing temperature to zero for scoring is table stakes. Second, explainability. You should be able to tell a candidate why they got the score they got. Third, a human making the final call. AI is an assistant. It should never decide who’s in and who’s out on its own.

If you run a hiring pipeline, try this: submit the same resume two or three times and watch the number. If it drifts, your system isn’t evaluating people. It’s evaluating luck. And for the rest of us — the applicants — it’s time to start asking a harder question. The score that decided your rejection: would it have read the same on the second try?