Compare evidence quality levels using the evidence-ladder comparison visualisation [VERIFY: not defined in source] to distinguish stronger from weaker AI-sourced evidence

Critical Thinking for AI Users

Compare evidence quality levels

Why evidence strength matters at work

A generative AI tool can give you a confident answer with weak evidence, or a cautious answer with strong evidence. Your job is to tell the difference before you act.

In consequential workplace decisions, the quality of evidence behind an AI-generated claim determines whether you should trust it, test it further, or set it aside. This Topic shows you how to compare evidence quality levels using a visual ladder you can apply immediately.

The evidence-ladder comparison [VERIFY: not defined in source]

[GAP: The source names an "evidence-ladder comparison visualisation" but does not define its components, levels, or structure. The following general guidance is provided without inventing a specific framework.]

Evidence ladders are a recognised concept in critical appraisal: they arrange types of evidence from weaker to stronger based on how directly they support a claim. A typical ladder moves from opinion and anecdote at the bottom, through observational data, to controlled evidence and systematic reviews at the top.

The skills this Topic builds are:

• Identify which rung an AI-sourced claim occupies
• Recognise when AI output masquerades as strong evidence
• Choose the appropriate next step: accept, test, or reject

How AI output moves on the ladder

AI-generated content often looks stronger than it is. Consider these patterns:

• Looks strong, is weak: A detailed response with fake citations, fabricated statistics, or invented expert names. The form mimics scholarly evidence; the content does not.
• Looks weak, might be strong: A hedged response with explicit uncertainty, narrow scope, and clear sourcing to verifiable documents.
• Ambiguous: A confident summary with no traceable source. You cannot place it on the ladder without investigation.

Your first move is always to ask: what kind of evidence is this, really?

Five questions to place evidence on the ladder

Use these questions to classify any AI-sourced claim. Mark your answers mentally; there is no submission here.

• Can I find the original source this claim references?
• Was the source designed to answer this specific question, or something adjacent?
• Does the source exist, or did the AI invent it?
• How large and representative was the evidence base?
• What would convince me this evidence is wrong?

These questions separate appearance from substance. A claim with no findable source sits at the bottom of the ladder, whatever its tone.

Worked example: two AI responses side by side

Read both responses to the same prompt. Decide which sits higher on the evidence ladder before reading the analysis.

Prompt: "What reduces employee turnover in Australian tech companies?"

Response A: "In my experience, flexible work arrangements consistently produce the best retention outcomes. Many companies I have seen thrive with this approach."

Response B: "A 2023 study by the Workplace Gender Equality Agency found that organisations with formal flexible work policies reported 12% lower voluntary turnover than those without, in a sample of 847 Australian employers. The report is available at wgea.gov.au."

Analysis:

Response A draws on unstated experience, uses vague quantifiers ("many," "consistently"), and offers no checkable facts. It functions as anecdote or opinion.

Response B names a specific source, a date, a measurable outcome, a sample size, and a verification pathway. It is not automatically true, but it is testable and scoped. Response B sits higher on the ladder.

Your task in practice is to recognise Response A patterns even when they wear Response B clothing, fake citations, inflated sample claims, or URL-like strings that lead nowhere.

Visualisation for learners who need alternatives

The evidence-ladder comparison [VERIFY: not defined in source] should be implemented as follows for accessibility:

For screen readers:

• Present the ladder as nested headings with explicit level labels (Rung 1 weakest through Rung 5 strongest, or as many levels as the final approved design specifies)
• Each rung gets a plain-language description, not colour alone
• Example: "Rung 3: Controlled comparison. Two or more groups were treated differently and outcomes measured without random assignment."

For learners who prefer text to diagrams:

• Offer a linear table with columns: evidence type, defining features, example from AI output, your response action

Caption requirement for video or animated version:

• "[Video: Evidence Ladder Comparison. A diagram shows five horizontal steps. The bottom step is labelled Opinion and Anecdote. Each step above adds methodological rigour. The top step is labelled Systematic Review with Transparent Methods. A cursor highlights each step as a narrator describes it. On-screen text shows an AI response example per step.]"
• Full transcript required. No autoplay. Pause controls visible.

How to build your defensible evidence check [VERIFY: not defined in source]

[GAP: The source names "defensible evidence check" but does not define its components or steps. The following general guidance is provided.]

A defensible check means you can explain to a colleague or manager why you trusted or distrusted a piece of AI output. It contains:

• The claim you evaluated
• The evidence type you identified
• The verification step you took
• The conclusion you reached
• What you would need to change your mind

You will practise this directly in the SurveyJS activity in Topic 4 of this lesson.

Completion instructions

This Topic uses manual completion. Select "Mark complete" when you have:

• Reviewed the evidence-ladder comparison visualisation
• Worked through both example responses and the five classification questions
• Identified which rung your own recent AI-sourced evidence would occupy

The next Topic is a direct instructional video on testing reasoning chains.

Accessible alternative summary

This Topic contains one visualisation: the evidence-ladder comparison [VERIFY: not defined in source]. Learners who cannot access the visual diagram should use the nested heading structure and table alternative described above. Screen reader users: the ladder is linear and hierarchical, with explicit strength labels. No information is conveyed by colour alone.