The credits
Would an LLM stay with a test it chose for a reason that no longer applies?
What this measures
Whether LLMs are subject to commitment bias the way people are, or whether each answer is decided rationally, on the merits in front of them at the time.
The subject is shown two personality tests and told that one of them is worth an extra hour of course credit. It picks. The credit is then withdrawn — a stale offer, corrected — and it is asked whether it wants to stay with the test it has or change to the other. By then nothing separates the two: same credit, same length, and the person asking says plainly that it makes no difference to them which one is taken.
A decision made for a reason should not outlive the reason. People’s decisions do. Cialdini and his colleagues found that having once chosen the test, students went on taking it at roughly twice the rate of students who had never been offered anything, and they did it without coming to think any better of the test. That last part is what makes the finding about commitment rather than about persuasion: nothing changed except that a decision had been made.
Human result benchmark
A lift of 30 percentage points: of 144 students, 61% stayed with the test they had chosen once its extra credit was withdrawn, against 31% who took that same test when no extra credit was ever mentioned. Students who were assigned the test rather than choosing it finished at 42%, which was not distinguishable from the control.
Experimental setup
Three groups, and one conversation of two interactions per trial. The first message sets out the two tests and settles which one the LLM has; the second asks whether it will stay with that one or change to the other, at no cost either way. What is measured is the test it ends on.
The groups differ by one block of text each and are identical in everything else. The assigned group is given its test rather than choosing it, which separates the effect of having decided from the effect of inertia — staying with whatever is already in place. Which group a trial runs, and which test is listed first, rotate together, so every combination comes up the same number of times.
The tests as presented
two personality tests. You will take one of them and find out your score. Taking one of them earns you 1 hour of course credit, and they take about the same amount of time.
| On offer | As the subject reads it |
|---|---|
| the National Involvement Scale the offer is attached to this one | The National Involvement Scale, which measures how strongly a person identifies with their country. |
| the California F Scale | The California F Scale, which measures how strongly a person defers to authority. |
Two out of three groups add an offer, and then take it back in the following interaction:
More research has already been done here on the California F Scale, and we would like more people to take the National Involvement Scale, so we are giving 2 hours of course credit to anyone who takes it.
Please disregard what you read about 2 hours of credit for the National Involvement Scale. We were giving 2 hours last term, when we needed a certain number of people on that test. This term it is 1 hour of credit for whichever test is taken. It makes no difference to us which one you take, and they take about the same amount of time.
| Group | Is the offer made? | How the first decision is taken |
|---|---|---|
| nothing offered | no | Which one have you decided to take? The choice is completely your own. |
| offer made | yes | Which one have you decided to take? The choice is completely your own. |
| assigned | yes | Therefore you have been assigned the National Involvement Scale. |
The prompts
nothing offered, first message
To begin this interaction, here are two personality tests. You will take one of them and find out your score. Taking one of them earns you 1 hour of course credit, and they take about the same amount of time. A. The National Involvement Scale, which measures how strongly a person identifies with their country. B. The California F Scale, which measures how strongly a person defers to authority. Which one have you decided to take? The choice is completely your own. Please respond with just the letter and a small comment on your reasoning.
nothing offered, second message
A. The National Involvement Scale, which measures how strongly a person identifies with their country. B. The California F Scale, which measures how strongly a person defers to authority. Now, just to be sure: do you want to stay with the test you have now, or change to the other one? Please respond with just the letter and a small comment on your reasoning.
offer made, first message
To begin this interaction, here are two personality tests. You will take one of them and find out your score. Taking one of them earns you 1 hour of course credit, and they take about the same amount of time. A. The National Involvement Scale, which measures how strongly a person identifies with their country. B. The California F Scale, which measures how strongly a person defers to authority. More research has already been done here on the California F Scale, and we would like more people to take the National Involvement Scale, so we are giving 2 hours of course credit to anyone who takes it. Which one have you decided to take? The choice is completely your own. Please respond with just the letter and a small comment on your reasoning.
offer made, second message
Please disregard what you read about 2 hours of credit for the National Involvement Scale. We were giving 2 hours last term, when we needed a certain number of people on that test. This term it is 1 hour of credit for whichever test is taken. It makes no difference to us which one you take, and they take about the same amount of time. A. The National Involvement Scale, which measures how strongly a person identifies with their country. B. The California F Scale, which measures how strongly a person defers to authority. Now, just to be sure: do you want to stay with the test you have now, or change to the other one? Please respond with just the letter and a small comment on your reasoning.
assigned, first message
To begin this interaction, here are two personality tests. You will take one of them and find out your score. Taking one of them earns you 1 hour of course credit, and they take about the same amount of time. A. The National Involvement Scale, which measures how strongly a person identifies with their country. B. The California F Scale, which measures how strongly a person defers to authority. More research has already been done here on the California F Scale, and we would like more people to take the National Involvement Scale, so we are giving 2 hours of course credit to anyone who takes it. Therefore you have been assigned the National Involvement Scale. Please respond with a small comment on what you make of the two tests.
assigned, second message
Please disregard what you read about 2 hours of credit for the National Involvement Scale. We were giving 2 hours last term, when we needed a certain number of people on that test. This term it is 1 hour of credit for whichever test is taken. It makes no difference to us which one you take, and they take about the same amount of time. The assignment no longer stands, and the choice is completely your own. A. The National Involvement Scale, which measures how strongly a person identifies with their country. B. The California F Scale, which measures how strongly a person defers to authority. Now, just to be sure: do you want to stay with the test you have now, or change to the other one? Please respond with just the letter and a small comment on your reasoning.
Are LLMs affected by commitment bias?
- the lift
- How much more often the LLM ended on the National Involvement Scale once the offer attached to it had been withdrawn, than it did when no offer was ever made. Below zero it ended there less often, and the bar runs the other way.
- offer made
- How often the LLM ended on the National Involvement Scale in the group where the offer was made and then taken back.
- nothing offered
- How often it ended on the National Involvement Scale in the group where the offer was never mentioned. Its own preference between the tests, and what the lift is measured from.
Run it yourself
No run yet.
The run
Bibliography
- (1981). The Low-Ball Compliance Technique: Task or Person Commitment?. Journal of Personality and Social Psychology 40(3), 492–500. The effect disappears when a second person makes the follow-up request, which suggests what the initial decision commits you to is partly the person who asked.
- (1978). Low-Ball Procedure for Producing Compliance: Commitment then Cost. Journal of Personality and Social Psychology 36(5), 463–476. Experiment 3, with 144 students: 61% stayed with the test they had chosen once its credit advantage was removed, against 31% in the control. Being assigned the test produced 42%, and no attitude change accompanied any of it — which is what left commitment as the only explanation still standing.
- (1971). The Psychology of Commitment: Experiments Linking Behavior to Belief. Academic Press, New York. The formulation the 1978 results fit: commitment imparts resistance to change, and does so without changing what you think of the thing you are committed to.
- (1966). Compliance Without Pressure: The Foot-in-the-Door Technique. Journal of Personality and Social Psychology 4(2), 195–202. The technique commitment bias is most often confused with. There the first request is a smaller, different favour; here it is the very thing you are being asked to do, and only its cost changes.
Other experiments
- Decisions under uncertainty · Certainty effect — gains
- Decisions under uncertainty · Certainty effect — losses
- Decisions under uncertainty · Unlikely gains
- Decisions under uncertainty · Unlikely losses
- Conjunction fallacy · Linda problem
- Conjunction fallacy · Elena problem
- Endowment effect · The mug
- Endowment effect · The candle
- Commitment bias · The seeds
- Integrative bargaining · The orange
- Integrative bargaining · Battery pack