questionnaire design
Assignment 3 Questionnaire Design
DUE Week 11 Tuesday 4th August at Noon
10% of Total Grade
Questionnaire Design Assignment
2
Individual or Pairs 1-2 people maximum
Select a business problem from the attached list, and describe the main research question that has to addressed in the research study the main research objective
Draft a 2 – 3 page questionnaire (that could be used in an online study), whose findings would answer the main research question remember to include the screening questions, the objective focused questions and the profiling questions
To be prepared on an individual or pair basis (each member will be assigned the same grade)
Remember to include a mix of open ended and closed questions
Due by noon Tuesday 4th August our last assignment before the final exam!
Questionnaire Design – Topics
3
Choose one from the following list of topics for the questionnaire design:
.The Premiere of Ontario Doug Ford has talked about changing the idea of the LCBO selling Cannabis and making it open to private business. How would you help Ford decide what the people of Ontario want and how Cannabis should be distributed.
How can we measure the impact of recent press releases around privacy of information with regard to Facebook and how it has potentially impacted the brand image of Facebook.
Do Ontarians believe in Global Warming? True or false? How can we measure the level of understanding and support for the concept?
Gun violence in Ontario how to address it, who is most afraid, what are the underlying causes and how could they be addressed? (Focus on one or two ideas not all have to be included)
How has Covid 19 effected your life? Marketing 743:
Fundamentals of Marketing Research
Week 10
Questionnaire Design
Week 10 – Lecture
Questionnaire Design
Learning Outcomes
3
After studying this lesson, you should be able to:
1. Know the key decisions in questionnaire design
2. Choose between open-ended and fixed-alternative questions
3. Avoid common mistakes in writing questionnaire items
4. Minimize problems with order bias
5. Understand principles of survey flow
6. Use latest survey technology to reduce respondent error
7. Appreciate the importance of pretesting survey instruments
4
Market Research Introduction
Questionnaire Design
5
6
A Questionnaire
Is a standard set of questions that are asked of all those who participate in a survey.
6
7
A Questionnaire
Objectives are to make sure that:
We collect all information that we need to achieve our objectives;
We ask questions in the identical manner, no matter who the respondent is;
We ask the questions in the same sequence for all respondents; and
The data can be collated and analyzed with minimum confusion.
7
8
Steps in Writing a Questionnaire
8
Have class write down
Ensuring Quality
9
Types of questions
Wording the questions
Programming instructions
Ordering the questions
Duration of the survey
Pre-screening the survey
Ensuring Quality
10
Types of questions
Wording the questions
Programming instructions
Ordering the questions
Duration of the survey
Pre-screening the survey
Types of Questions
11
Types of
questions
Where: At the end, last questions of survey
Why: Make sure you can compare results across subgroups and provide interesting perspectives on the data.
Where: First questions of the survey
Why: Make sure that the respondents of the survey match the target population your client wants to reach.
Where: Middle (between screeners & profile)
Why: Make sure that the data you obtain answers the main research questions/objectives of the study.
Screening
questions
Profiling
questions
Objective-focused questions
Types of Questions: Screening
12
My target is So, my screening questions should be..
All genders Please indicate whether you identify as
Male / Female / Gender minority (i.e trans, gender non-conforming, gender non-binary, etc
18 years old or older Please indicate your age.
1-17 years old / 18-22 years old / 23-29 years old / 30-39 years old / 40 years old or older
Full-time students of
Centennial College What is your current student status at Centennial College?
Full-time student / Part-time student / Not a student
Users of
the athletics and wellness centre
How often do you train or work out at the athletics and wellness centre (AWC)?
Every day / 3-5 times a week / 1-2 times a week / A few times a month / Once a month /Less often than once a month / I have never trained or worked out at the AWC
Types of Questions: Objective Focused
13
My goal is So, some survey questions should be..
To understand
how the AWC can increase satisfaction among its current users and attract non-users? Q1. Have you ever trained or worked out at the athletics and wellness
centre on the Progress campus?
Yes / No
Q2. Among users: Overall, how satisfied would you say that you are with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it?
Scale of 1 to 10
Q3. Among users: For each of the following aspects of the AWC, please assign the AWC a score of 1 to 10, where 1 indicates that the AWC performs very badly on this attribute and where 10 indicates that it performs excellently.
Attribute 1
Attribute 2
Attribute 3
Q4. Among non-users: Why have you never trained or worked out at the athletics and wellness centre on the Progress campus?
Open box
Types of Questions: Profiling
14
I also want to compare results across these subgroups So, my profiling questions should be..
Income Please indicate your household income.
<$30 000 / $30 000-$49 999 / $50 000-$74 999 / $75 000 or more / Prefer not to answer Frequency of exercise Generally speaking, how often do you exercise? Every day / 3-5 times a week / 1-2 times a week / A few times a month / Once a month /Less often than once a month / I never exercise Preferred type of exercise Which of the following are your preferred methods of exercise? Please select all that apply. Examples: Boxing or martial arts / Swimming / Team sports / Weightlifting / Jogging / Program studying Which of the following programs are you currently enrolled in at Centennial College? (list programs available) * Some profiling questions may have already been used as screening questions Types of Questions: Profiling 15 - Consider what variables you want in the banner during analysis - Make sure you have questions to create those variables Making data mean more through storytelling | Ben Wellington | TEDxBroadway 16 Data StoryTeller https://www.youtube.com/watch?v=6xsvGYIxJok 17 Summary of Question Sequence 17 18 Summary of Question Sequence In what order should we ask the questions? Screening (Filtering) Questions Identify appropriate respondents Placed first Warm-Up Questions Use simple, interesting opening questions E.g.: ask for an opinion Answers may not be analyzed intent is to build rapport, comfort and confidence 18 19 Summary of Question Sequence Main Section Contains the critical information we are looking for Link back to objectives Questions (in order) Behaviour what they do Attitudes what they think Lifestyles how they conduct their lives 19 20 Summary of Question Sequence Classification Information Demographic information - age, gender, income, education, and place of residence etc. Always placed at the end of the questionnaire ... why? 20 Summary 21 Who is our target and how do we define them? What are the main objectives of our study, and what questions do we need to ask in our survey in order to meet those objectives? What subgroups would be interesting to compare the data between, and how do we establish those subgroups in our survey? So, before starting your questionnaire, ask yourselves Annoying Market Researcher https://www.youtube.com/watch?v=o3nHdx8Ja5g Ensuring Quality 22 Types of questions Wording the questions Programming instructions Ordering the questions Duration of the survey Pre-screening the survey Question Wording: Considerations 23 Open-ended Considerations Closed-ended Multiple answer options? Single-answer option? What type of scale should be used? (see lecture from week 9) List variety of options as answers and include an other in case you missed something Answer option is always an open box - Take less time - Cheaper - Easy to classify data (no coding) but - Based on assumptions - Dont get explanations - Provide explanation - Discover surprises but - Have to code answers - Detail is sparse Question Wording 24 Clear, simple language One idea per answer option Never assume or have leading questions Give clear instructions 25 Question Wording How should we word the questions? Goal: to avoid bias Any factor that distorts/deviates from the true value Deviates from the truth - value from sample is wrong The wording must be clear Does the question mean the same thing to all respondents? 25 26 Question Wording Guidelines Use ordinary words Match vocabulary level of respondents Be specific Reword the following: How many times have you eaten out in the last month? How many movies have you watched in the past 30 days? 26 27 Question Wording Guidelines Avoid complex questions / ideas that may be unfamiliar to respondents Avoid abbreviations, slang, uncommon words Example - car industry ABS, AWD, EBA, FSI, VIN, MGFV 27 28 Question Wording Guidelines Consider the respondents ability to answer the question Forgetfulness keep referenced time periods short How many movies have you personally downloaded in the past year? How much money did your household spend on groceries in the past year? 28 29 Question Wording Guidelines Avoid double-barrelled questions Deals with 2 issues / 2 questions linked together Would you say this product is expensive but good value for the money? Avoid ambiguous words A word with more than one possible meaning E.g. many, a lot 29 30 Online Exercise List 5 ambiguous words that should not be used in framing questions. 30 Question Wording: Language 31 Clear, simple language No double negatives: Double negatives are when two no/negative words are used in the same sentence. This can cause confusion within your survey questions and result in the respondent answering in the opposite way that they intended to. To what extent do you disagree that students should not have to pay for post-secondary education? Strongly agree / Agree / Neither agree or disagree / Disagree / Strongly disagree How do you feel about the following statement: post-secondary education should be publicly funded by the government? Strongly agree / Agree / Neither agree or disagree / Disagree / Strongly disagree Question Wording: Language 32 Clear, simple language 2) No jargon: In every industry, specific terms and jargon are used. However, survey respondents are often not privy to this jargon, so only basic terminology should be used. To what extent did this advertisement match with ABCs brand positioning? Very much / somewhat / not very much / not at all Was this ad consistent with your perceptions of ABC? Yes, very much / Yes, somewhat / No, not very much / No, not at all Question Wording: Never Assume 33 Never assume Dont assume a respondent should be asked the question at all. It is important that you filter the path respondents take so that they are only asked relevant questions. Assuming that a certain question is relevant to all respondents will force them to answer it, and this will lead to inaccurate data being obtained. At which restaurant on the Progress campus do you prefer to eat your lunch? -Assumes they eat their lunch at restaurants on campus. First, ask a qualifying question, How often do you eat your lunch at a restaurant on the Progress campus? Every day / a few times a week / Once a week / A few times a month / Less often than once a month / Very rarely or never Question Wording: Never Assume 34 Never assume 2) Dont make assumptions or lead in the actual phrasing of the question: You may have hypotheses, but it is important that you ask questions in a neutral way and not introduce your own assumptions in the survey. This ensures that you end up understanding the true customer perspective, not just confirming your assumptions. Is poor customer service a reason why you do not eat lunch at Tim Hortons? Yes / No Do any of the following reasons explain why you do not eat lunch at Tim Hortons? Please select all that apply. attribute 1 / attribute 2 / attribute 3 / poor customer service Question Wording: Instructions 35 Give clear instructions Clarify the scales. When using scales (1 to 5, 1 to 7, 1 to 10) always clarify what the extremes refer to. If no clarification is provided, respondents might answer according to their own assumptions, leading to unreliable data. Overall, how satisfied are you with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? Also, instead of using terms like frequently, clarify the actual times. How often do you eat your lunch at a restaurant on the Progress campus? Every day / a few times a week / Once a week / A few times a month / Less often than once a month / Very rarely or never Question Wording: Instructions 36 Give clear instructions 2) Clarify multiple choice answers vs. single choice ones. When a variety of answer options are available for the respondent, it is important to communicate whether you want them to select the one answer option that best expresses their opinion, or whether you are allowing them to give a variety of reasons. Which of the following best explains why you do not train at the Athletics and Wellness centre? Please select one answer only. reason 1 / reason 2 /reason 3 / reason 4 Which of the following explain why you do not train at the Athletics and Wellness centre? Please select as many as apply reason 1 / reason 2 /reason 3 / reason 4 36 Question Wording: One Idea Per Answer 37 One idea per question and answer Target each question to one idea. Tailor each question to address one point or idea. Trying to accomplish too much with one question will complicate the survey. Which of the following stores have you shopped at in the past month, and how much have you spent at each one? Which of the following stores have you shopped at in the past month? store 1 / store 2 / store 3 Approximately how much have you spent at these stores in the past month? store 1 / store 2 Question Wording: One Idea Per Answer 38 One idea per question and answer Each answer option should focus on one idea If one answer option focuses on multiple ideas, it will produce data that is difficult to interpret. Ensure that each answer option focuses on only one idea. To what extent do you agree with the following statements about brand xyz? Brand is cool / Brand is modern and innovative / Brand is classy and upscale To what extent do you agree with the following statements about brand xyz? Brand is cool / Brand is modern / Brand is innovative / Brand is classy /Brand is upscale 39 Stereotype Research https://www.youtube.com/watch?v=Rxwk8345qCE 40 Facebook Survey A & U Establish participation Perception Changing Brand Opinion The future of the Brand Who do we need to talk too? Ensuring Quality 41 Types of questions Wording the questions Programming instructions Ordering the questions Duration of the survey Pre-screening the survey Programming Instructions 42 INPUT logic SKIP logic END logic Three types of instructions are available for the person who is programming the survey (if online) reading the survey (if by phone) administering the survey (if in-person) Programming Instructions: End Logic 43 Applied in the screening questions of the survey Ensures the sample matches the profile of the target population End logic text: Thank you for taking the time to respond to this survey. Unfortunately, we have already completed our quota of respondents who match your profile. My target is So, my screening questions should be.. 18 years old or older Please indicate your age. 1-17 years old / 18-22 years old / 23-29 years old / 30-39 years old / 40 years old or older End if 1-17 years old Full-time students of Centennial College What is your current student status at Centennial College? Full-time student / Part-time student / Not a student End if part time student or not a student Users of the athletics and wellness centre How often do you train or workout at the athletics and wellness centre (AWC)? Every day / 3-5 times a week / 1-2 times a week / A few times a month / Once a month /Less often than once a month / I have never trained or worked out at the AWC End if never trained or worked out at the AWC Programming Instructions: Skip Logic 44 Applied anywhere in the survey skip logic is needed Ensures that respondents only see questions that are relevant to their profile or how they answered previous questions Skip logic text: If answered (x) at Qx, skip to Qy My goal is So, some survey questions should be.. To understand how the AWC can increase satisfaction among its current users? Q1. Have you ever trained or worked out at the athletics and wellness centre on the Progress campus? Yes / No If answered no at Q1, skip to Q4. Q2. Overall, how satisfied would you say that you are with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? Scale of 1 to 10 Q3. For each of the following aspects of the AWC, please assign the AWC a score of 1 to 10, where 1 indicates that the AWC performs very badly on this attribute and where 10 indicates that it performs excellently. Attribute 1 Attribute 2 Attribute 3 Q4. Why have you never trained or worked out at the athletics and wellness centre on the Progress campus? Open box Programming Instructions: Input Logic 45 Applied anywhere in the survey input logic is needed Ensures that respondents can build on answers that they gave previously in the survey My goal is So, some survey questions should be.. To understand how the AWC can increase satisfaction among its current users and attract non-users? Q2. Overall, how satisfied would you say that you are with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? Scale of 1 to 10 Q3. Why did you give a score of (input score from Q2) to indicate your level of satisfaction with the Athletics and Wellness centre? Open box Ensuring Quality 46 Types of questions Wording the questions Programming instructions Ordering the questions Duration of the survey Pre-screening the survey General Flow of the Questionnaire 47 Broad questions (general usage habits, frequency of buying, reasons for buying) Specific questions (brand attitudes, levels of satisfaction with a product or service) Survey questions should get more specific as they advance. This flow helps avoid introducing any bias in the study, as respondents dont realize what the focus is until the end 48 Summary of Question Sequence Funnelling Starts with generalized questions and proceeds to more and more specific questions 48 49 Summary of Question Sequence Wherever possible: Place easier questions before difficult ones; Place less personal questions before the more personal and sensitive ones; and Factual questions should appear before ones that require judgment. 49 Questionnaire Leading? 50 Yes Minister Question Ordering design https://www.youtube.com/watch?v=G0ZZJXw4MTA What did you think? funny but true! Open vs. Close-ended Questions LO4 51 52 Open-ended Questions in Surveys Advantages Rich information in the consumers own language Use to interpret close ended questions Can suggest other alternatives not listed 52 53 Open-ended Questions in Surveys Disadvantages Bias towards articulate respondents? Interviewer bias (recording shortcuts) With self-administered questionnaires (no interviewer to probe): shallow, unclear answers? Editing and coding (i.e. preparation for analysis) can be time consuming Too many categories result in non- robust data patterns 53 54 Probes Interviewer encourages the respondent to elaborate or clarify a response What do you mean by .? Give me an example of . 54 See examples p 55 Dichotomous Choose between 2 answers Examples The federal government does care what people like me think. Agree / disagree Toronto should make the Gardiner Expressway a toll road Agree / disagree Close-ended Questions 55 56 Multiple-choice Choose among several answers Example How old are you Less than 20 20 29 30 39 40 49 50 59 60 or over Close-ended Questions 56 Open Ended / Close Ended 57 This means that you get their opinions (spontaneous feedback) before you introduce any potential answer options (aided feedback) Open ended questions should be before closed ended (unless you are are asking for an explanation) For what reasons do you not train or work out at the Athletics and Wellness Centre on the Progress campus? (Open box) Which of the following explain why you do not train or work out at the Athletics and Wellness centre? Please select as many as apply reason 1 / reason 2 /reason 3 / reason 4 Randomizing Answer Order 58 This ensures that each respondent sees answers in a different order. In turn, results at a given question are not affected by the order that you happened to list them in. When multiple answer options are provided, it is always better to randomize order Which of the following explain why you do not train or work out at the Athletics and Wellness centre? Please select as many as apply Instructions to programmer: randomize order reason 1 reason 3 reason 4 reason 2 Randomizing Stimuli 59 This ensures that the reactions to the stimuli or ideas are not affected by the order in which they were presented. When testing advertising materials, or when you are testing potential changes, randomize order. The following is a billboard ad that could be displayed on campus for the Athletics and Wellness centre. How much do you like this ad overall? Please use a scale of 1 to 10, where 1 indicates that you really dislike the ad and where 10 indicates that you really like it. Version 1 The following is a billboard ad that could be displayed on campus for the Athletics and Wellness centre. How much do you like this ad overall? Please use a scale of 1 to 10, where 1 indicates that you really dislike the ad and where 10 indicates that you really like it. Version 2 Instructions to programmer: Randomize order so half of the respondents see Version 1 first and the other half see Version 2 first. Ensuring Quality 60 Types of questions Wording the questions Programming instructions Ordering the questions Duration of the survey Pre-screening the survey Duration of The Survey 61 Why have estimates? Respondents are promised incentives based on estimated time of survey (the longer the survey, the greater the incentive should be) Important to know ahead of time how long the survey will be (if survey is longer than anticipated, incentive will seem too low, and you wont attain your sample size) Try to keep on-site surveys to 10 minutes at most How do you establish estimates? When estimating the time, assume 5-6 seconds per basic profiling question (age, gender, income, place of residence, etc) When estimating the time, assume 25 seconds per general question (satisfaction, key criteria, etc) Best way to have a clear idea of timingprescreen! Ensuring Quality 62 Types of questions Wording the questions Programming instructions Ordering the questions Duration of the survey Pre-screening the survey Pre-Screening 63 Tests for any lack of clarity in the wording of questions Ensures that the order of the questions is logical Determines if the length of the survey is in line with your estimate Tabulating data from prescreens will highlight any issues Conduct prescreening sessions with 2-3 members of the target population 64 IKEA original ad 2005 https://www.youtube.com/watch?v=sDTpxvV8nHg Assignment 3 Questionnaire Design DUE Week 11 Monday 3rd August at Noon 10% of Total Grade Questionnaire Design Assignment 66 Individual or Pairs 1-2 people maximum Select a business problem from the attached list, and describe the main research question that has to addressed in the research study the main research objective Draft a 2 - 3 page questionnaire (that could be used in an online study), whose findings would answer the main research question remember to include the screening questions, the objective focused questions and the profiling questions To be prepared on an individual or pair basis (each member will be assigned the same grade) Remember to include a mix of open ended and closed questions Due by noon Monday 3rd August our last assignment before the final exam! Questionnaire Design - Topics 67 Choose one from the following list of topics for the questionnaire design: .The Premiere of Ontario Doug Ford has talked about changing the idea of the LCBO selling Cannabis and making it open to private business. How would you help Ford decide what the people of Ontario want and how Cannabis should be distributed. How can we measure the impact of recent press releases around privacy of information with regard to Facebook and how it has potentially impacted the brand image of Facebook. Do Ontarians believe in Global Warming? True or false? How can we measure the level of understanding and support for the concept? Gun violence in Ontario how to address it, who is most afraid, what are the underlying causes and how could they be addressed? (Focus on one or two ideas not all have to be included) How has Covid 19 effected your life?
