What happened
AWS says Amazon Connect Talent is designed for organizations hiring hundreds or thousands of people. It applies structured behavioral interviewing: interviews use consistent, job-related questions and explicit scoring criteria. The service builds interviews from role requirements and established occupational frameworks, generating behaviorally anchored scorecards that define strength and concern signals. Each question is intended to elicit a particular competency.
The interviewer adapts to each response by checking whether it contains enough evidence for the competency under assessment. When details are missing, it asks targeted follow-up questions. AWS’s example is a candidate who says they improved a slow system; the interviewer may ask what changed, what actions the candidate took and how they measured the improvement. Before use, hiring teams can review, select and edit competencies, scorecards and questions.
The service separates interviewing from scoring. The AI interviewer gathers evidence but does not assign a score during the conversation. A later evaluation process identifies demonstrated behaviors and cites supporting transcript passages. Competency scores run from 1 to 5 and are calculated with a fixed formula. When a transcript does not contain sufficient evidence, the service flags the competency rather than producing a score; AWS says missing evidence does not automatically count against the candidate.
AWS describes safeguards intended to keep scoring focused on job-related behavior. Scores use transcribed responses, not video, voice tone or physical appearance. Candidate identity information is excluded, and each competency is evaluated separately. An independent industrial-organizational psychology firm reviewed baseline generated content before human editing. It found that critical job information was retained in 100% of reviewed job descriptions; 91% of generated competencies were clearly traceable to the job description, 98% of questions targeted intended competencies and 94% of rubric signals clearly measured the intended competency.
For scoring, the firm reviewed 36 interviews and 252 rubric signals. Connect Talent’s decisions aligned with expert scoring in 89.3% of cases, rising to 97.6% when candidates provided clear evidence. AWS reports a Cohen’s kappa of 0.79, which it characterizes as substantial agreement and just below the 0.81 standard for almost-perfect agreement. In a preliminary follow-up evaluation, scores based only on candidates’ first responses were up to 1.14 points lower on a five-point scale than scores based on the full interview. AWS also says behaviorally anchored scoring used all five points of its scale, while holistic LLM-as-judge scoring concentrated ratings in the middle three.
What the archive adds
New Runtime’s earlier coverage on Proton’s canceled AI-use re-interviews (November 1, 2026) examined workplace pressure around AI adoption. This is a distinct story: AWS is describing an AI-supported hiring process, not employee monitoring or internal adoption expectations.
Trend signal
This case offers a limited example of agent economics moving to completed work: AWS presents the service around a consequential hiring task and the evidence it gathers, rather than model calls or tokens. The reported evaluations do not establish hiring outcomes or cost per successful hire, so the connection is about framing and workflow design, not proof of economic performance.
Why now
AWS is promoting the service’s interview design and evaluation ahead of its AWS re:Invent 2026 sessions on November 30 and December 3. The announcement positions those sessions as a forum for discussing structured interviews, adaptive follow-ups and evidence-based scoring.
Why it matters
The most transferable design choice is the boundary between eliciting evidence and judging it. If an AI system both conducts an interview and produces an opaque overall impression, teams have little basis for checking whether a rating reflects job-relevant behavior. Connect Talent’s described approach makes the intermediate evidence visible, links it to transcript passages and uses a predefined formula to aggregate competency signals.
The insufficient-evidence flag is equally consequential. It distinguishes “the candidate did not demonstrate this behavior” from “the interview did not establish whether they demonstrated it.” That distinction can prevent missing information from silently becoming a negative judgment. Teams building AI-assisted evaluation workflows can borrow the pattern: define the evidence needed for each criterion, ask targeted follow-ups when evidence is missing, and preserve an explicit unknown state rather than forcing a score. These controls improve inspectability, but do not by themselves show that decisions are fair or predict job performance.
Limits and uncertainty
The reported figures come from AWS’s evaluations and an independent firm’s review commissioned by Amazon; they are not a published, independently replicated assessment of real-world hiring outcomes. The sample described for scoring comprises 36 interviews and 252 rubric signals, and the follow-up comparison is called preliminary. AWS does not report whether use of Connect Talent improves hires’ subsequent job performance, reduces time-to-hire or changes demographic disparities. Nor does the announcement specify deployment pricing, customer adoption, or how hiring teams resolve disagreements with model-generated evidence. Agreement with expert ratings is a measure of consistency, not proof that the ratings themselves are valid or unbiased.
Skill to build now
For any AI-assisted evaluation, write a rubric as observable behaviors rather than broad traits. For each behavior, define what counts as evidence and what should happen when evidence is absent; then require the system to cite the source passage behind each finding. Test the workflow on sample transcripts, including incomplete answers, before using scores in decisions.
Reader takeaway
AI-led evaluation is more auditable when the system gathers evidence first, scores explicit behaviors separately and leaves a criterion unscored when evidence is insufficient. AWS reports promising agreement results for Connect Talent, but those figures do not establish better hiring outcomes or fairness in production.