Imagine you want to learn whether customers found a new feature useful. You could ask:
I agree that the feature was useful.
Then offer a scale from “Strongly disagree” to “Strongly agree.”
It looks balanced. It feels familiar. But the question has already placed an opinion in front of the respondent and asked them to react to it. That small design choice can quietly produce more agreement than the underlying experience warrants.
A better question is much simpler:
How useful was the feature?
- Extremely useful
- Very useful
- Moderately useful
- Slightly useful
- Not at all useful
- I did not use this feature
The second version measures usefulness directly. The first measures usefulness only after filtering it through a person’s willingness and ability to agree or disagree with a statement.
That difference matters.
Agreement is not a neutral task
Some respondents have a general tendency to agree with assertions, regardless of their content. Survey researchers call this acquiescence bias.
This does not mean respondents are careless or dishonest. Agreement can feel like the polite, cooperative or cognitively easier response, especially when someone is moving quickly through a survey, has no strong opinion or assumes the researcher knows more than they do.
The result is a subtle upward pull on positive statements.
Consider these two items:
The onboarding process was easy.
The onboarding process was difficult.
Someone susceptible to acquiescence may agree with both, even though the statements point in opposite directions. Reversing the wording does not necessarily solve the problem. It can introduce confusion while leaving the underlying agreement tendency intact.
Agree/disagree questions are not deliberately written to manipulate people. But structurally, they invite endorsement. The survey presents a claim, and the respondent decides whether to accept or reject it.
Agree/disagree questions make respondents do extra work
Suppose a customer sees this statement:
I agree that the feature was very useful.
To answer accurately, the customer must:
- Decide how useful the feature actually was.
- Interpret what “very useful” means.
- Compare their experience with that claim.
- Translate the result onto an agreement scale.
Asking “How useful was the feature?” removes the final translation. Respondents can report their judgment directly using response options that match the question.
This is the principle behind item-specific response options: the answers describe the exact dimension being measured.
- Usefulness: Extremely useful → Not at all useful
- Ease: Very easy → Very difficult
- Frequency: Always → Never
- Satisfaction: Very satisfied → Very dissatisfied
- Importance: Extremely important → Not at all important
- Confidence: Completely confident → Not at all confident
The response scale changes with the question because the subject changes. That requires slightly more work from the survey designer, and less work from every respondent.
Disagreement is often ambiguous
Agreement scales can also make responses difficult to interpret.
Imagine someone disagrees with this statement:
The setup process was easy.
What does that response mean?
Perhaps setup was moderately difficult. Perhaps it was extremely difficult. Perhaps the respondent never completed setup. Perhaps “easy” simply felt too positive, even though the experience was not especially hard.
A direct question reveals more:
How easy or difficult was the setup process?
- Very easy
- Somewhat easy
- Neither easy nor difficult
- Somewhat difficult
- Very difficult
- I did not complete the setup process
Now each answer has a clearer relationship to the experience being measured.
The evidence favors direct questions
This is more than a theoretical concern.
Saris, Revilla, Krosnick and Shaeffer compared agree/disagree questions with item-specific alternatives using randomized survey experiments and a method designed to estimate reliability and validity. Their studies used representative samples across multiple European countries and covered different topics, question orders and survey modes.
The researchers found that item-specific questions generally produced substantially higher-quality data. Their advantage remained even when the agree/disagree scales offered more response categories. Validity was consistently better for item-specific questions, while reliability was also better in most comparisons.
The authors concluded that the overall pattern was robust across countries, topics, modes and question order. In other words, replacing five agreement choices with seven did not fix the underlying problem. Asking a more direct question did.
Read the full study in Survey Research Methods.
Why agree/disagree scales remain popular
Agreement scales are convenient.
A survey designer can write ten statements and place the same five options beneath every one:
- Strongly agree
- Agree
- Neither agree nor disagree
- Disagree
- Strongly disagree
This makes questionnaires faster to build and visually consistent. It also makes large matrix questions possible.
But that consistency primarily benefits the survey creator. Respondents must repeatedly infer what each statement is really trying to measure, usefulness, frequency, satisfaction, confidence or something else, and then convert that judgment into agreement.
A reusable scale is not necessarily a valid scale.
How to rewrite agreement questions
Start by identifying the construct hidden inside the statement. Then ask about it directly.
Instead of:
I agree that the dashboard is easy to use.
Ask:
How easy or difficult is the dashboard to use?
Instead of:
I agree that this feature saves me time.
Ask:
How much time, if any, does this feature save you?
Instead of:
I agree that I trust the information shown.
Ask:
How much do you trust the information shown?
Instead of:
I agree that I would use this product again.
Ask:
How likely are you to use this product again?
The goal is not merely to remove the words “agree” and “disagree.” It is to make the question and its answers describe the same underlying dimension.
A practical rule for better surveys
Before publishing a question, ask:
What exactly am I trying to measure?
If the answer is usefulness, ask about usefulness. If it is difficulty, ask about difficulty. If it is frequency, ask about frequency.
Then make every response option a meaningful position on that dimension. Add “Not applicable” or “I did not use this feature” when appropriate so respondents are not forced to manufacture an opinion.
Agreement scales may save the researcher a few minutes while designing a survey. Item-specific questions save every respondent effort, and produce answers that are easier to interpret and act on.
A good survey should measure what people think, not how readily they agree with something you said.