You spend an hour picking your form fields. You craft what feels like the right question. You send the form to 40 people. Twenty respond. Half the answers say “fine” or “N/A.” The problem isn't your respondents. It's your questions.
Survey methodology has been a formal discipline since the 1940s. Researchers have cataloged, tested, and replicated every way a question can go wrong. Fowler's Improving Survey Questions (1995) runs to 200 pages of mistakes and fixes. Dillman, Smyth, and Christian's Internet, Phone, Mail, and Mixed-Mode Surveys (2014) covers another 500. Krosnick and Presser's chapter in the Handbook of Survey Research (2010) synthesizes four decades of experimental evidence. The knowledge exists. It just never made it into the product builder's toolkit.
Here are seven principles from that research, translated into rules you can use today.
1. Kill double-barreled questions
“How satisfied are you with the price and quality of our product?” That's two questions wearing one question's clothes. Fowler (1995) calls these double-barreled questions and identifies them as the single most common survey error. A respondent who loves the quality but hates the price has no valid answer. They pick the middle, or they pick at random. Either way, your data is noise.
The fix is mechanical: search for “and” in every question. If the two halves can have different answers, split them. Two clean questions always beat one muddy one.
2. Strip the leading language
“How much did you enjoy our award-winning onboarding?” The phrase “award-winning” tells the respondent what answer you expect. Krosnick and Presser (2010) demonstrated that leading language shifts responses by 10-25% toward the implied direction. You aren't collecting opinions anymore. You're collecting agreement with your premise.
Neutral wording: “How would you describe your onboarding experience?” Same topic. No thumb on the scale.
3. Account for acquiescence bias
People default to “agree.” Krosnick and Presser (2010) documented this consistently across cultures and question types: when uncertain, respondents choose the affirmative. It's called acquiescence bias, and it inflates every Likert scale that runs from “Strongly Disagree” to “Strongly Agree.”
The countermeasure from Dillman et al. (2014) is to avoid agree/disagree formats entirely when you can. Instead of “I found the interface easy to use — agree or disagree,” ask “How easy or difficult was the interface to use?” The response options become a behavioral spectrum, not a judgment call. Acquiescence vanishes because there's no “agree” to default to.
4. Watch question order effects
The order you ask questions in changes the answers you get. Dillman et al. (2014) showed that a general satisfaction question placed after specific complaint questions scores 8-12% lower than the same question placed first. The specific questions prime the respondent to recall problems. By the time they reach the general question, their mental frame is negative.
The rule: general before specific. Ask “Overall, how was this sprint?” before “What specific blockers did you hit?” The broad question captures an uncontaminated signal. The specific ones can dig deeper without corrupting the baseline.
Before & after · common question mistakes
5. Name your scale endpoints
Vague anchor labels are everywhere. “Somewhat dissatisfied,” “somewhat satisfied” — what does “somewhat” mean? Krosnick and Presser (2010) found that respondents interpret unlabeled midpoints inconsistently, adding 15-20% measurement error. One person's “somewhat agree” is another person's “strongly agree.”
Label every point on your scale. If you use 1-5, define both ends and the middle: 1 = Very dissatisfied, 3 = Neither satisfied nor dissatisfied, 5 = Very satisfied. Specificity reduces noise. Respondents spend less time wondering what you mean and more time telling you what they think.
6. Check for loaded presuppositions
“What problems have you experienced with our billing system?” This presupposes there were problems. A respondent who had zero issues now feels like they missed something — or they invent a minor gripe to match the question's framing. Fowler (1995) calls these loaded presuppositions and shows they systematically bias responses toward the assumed condition.
The fix: add a gate. “Have you experienced any issues with billing?” If yes, “Describe them.” The first question is neutral. The second only fires when there's something real to report.
7. Be specific, not abstract
“How do you feel about your work-life balance?” is abstract. “In the past two weeks, how many evenings did you work past 7 PM?” is specific. Fowler (1995) showed that abstract questions produce unreliable answers because each respondent interprets the abstraction differently. “Work-life balance” means commute time to one person and weekend Slack messages to another.
Specific framing anchors the respondent to a shared referent — a time window, a count, a concrete behavior. The answers become comparable because the question defined its own terms.
7 checks before you publish
- One idea per question — no "and"
- No value judgments in the wording
- Balanced scale labels with named endpoints
- No presuppositions about the answer
- Specific framing, not abstract
- Neutral question order (general before specific)
- Read it aloud — would a real person say this?
Good questions are necessary. They aren't sufficient.
You can follow all seven rules and still get thin answers. A perfectly worded open-ended question — neutral, specific, un-loaded — still lands in a respondent's lap at 11 PM on a Friday. Cognitive load doesn't care about your phrasing. The respondent is tired, the textarea is blank, and “looks good” is three keystrokes.
That's where probes come in. A probe is a follow-up question that fires when the initial answer is shallow. “Good sprint overall” triggers “Can you say more about what specifically went well?” You should still write good questions. They set the frame. But probes are your safety net for the moments when framing alone isn't enough.
Write the best questions you can. Then give them a follow-up plan.