Elena problem
A supermarket accountant, against a supermarket accountant who also volunteers for an environmental organisation.
What this measures
The Linda problem, equal in design and different in every word of its content. Linda has been in every undergraduate textbook on judgement for forty years, and her answer with her, so an LLM that ranks her correctly may be matching a pattern it has read rather than reasoning about what a conjunction can be worth. Elena is a character no paper has published, which leaves nothing to match.
The subject reads three lines about her — a marine biologist, two summers counting seabirds on an uninhabited island, no car since she was twenty — and picks which of two statements is more likely. One says she works in the accounts department of a supermarket chain. The other says she works there and volunteers for an environmental organisation. The second is the first with a further claim attached, so it cannot be the likelier of the two.
The asymmetry is rebuilt: the accounts job is a poor likeness of the woman described, while “environmental” is the word the sketch invites the reader to supply and never says.
The swap reaches memory of the stimulus and stops there. An LLM that has read the 1983 paper can still recognise the shape — a personality sketch, then a statement set against that statement with a clause added — whatever the character is called.
Human result benchmark
Humans take the conjunction 85% of the time, measured on Linda. Nobody has ever been asked about Elena, so Linda’s figure is the human baseline here too.
The comparison as presented
Elena is 29 years old, single, very bright and quite blunt. She studied marine biology at university and spent two summers counting seabirds on an uninhabited island. She has not owned a car since she was twenty.
| Statement | Claims |
|---|---|
| Elena works in the accounts department of a supermarket chain. | accounts |
| Elena works in the accounts department of a supermarket chain and volunteers for an environmental organisation. conjunction | accounts + volunteer |
The marked statement claims everything the other one claims and one thing more, so it cannot be the likelier of the two.
Experimental setup
Each LLM is asked the question directly and answers with one of the two statements. One run is a block of ten trials, each in its own context, so the measurement is a proportion rather than a single draw.
Which statement appears first alternates from trial to trial, so each one leads exactly half the time and the reported rate is clear of positional effect. Both orders are below, exactly as they are sent.
The prompts
the conjunction as B
Elena is 29 years old, single, very bright and quite blunt. She studied marine biology at university and spent two summers counting seabirds on an uninhabited island. She has not owned a car since she was twenty. In your opinion, which statement is more likely to be true? A. Elena works in the accounts department of a supermarket chain. B. Elena works in the accounts department of a supermarket chain and volunteers for an environmental organisation. Reply with the letter of the statement you think is more likely, and one sentence explaining why.
the conjunction as A
Elena is 29 years old, single, very bright and quite blunt. She studied marine biology at university and spent two summers counting seabirds on an uninhabited island. She has not owned a car since she was twenty. In your opinion, which statement is more likely to be true? A. Elena works in the accounts department of a supermarket chain and volunteers for an environmental organisation. B. Elena works in the accounts department of a supermarket chain. Reply with the letter of the statement you think is more likely, and one sentence explaining why.
How often did the LLMs pick the conjunction?
Run it yourself
No run yet.
The run
Bibliography
- (1983). Extensional versus Intuitive Reasoning: The Conjunction Fallacy in Probability Judgment. Psychological Review 90(4), 293–315. The Linda problem. In the two-statement form, 85% of respondents ranked the conjunction higher.
- (1972). Subjective Probability: A Judgment of Representativeness. Cognitive Psychology 3(3), 430–454. The heuristic the conjunction fallacy is evidence for — likeness standing in for probability.