Definition: MaxDiff analysis helps uncover what attributes are most important to your audience. A MaxDiff survey question shows respondents a set of items, asking them to choose what is most and least important. When the results are displayed, each feature is scored giving you actionable data. MaxDiff, short for maximum difference scaling, is sometimes referred to as best-worst scaling.
Basic Concept: MaxDiff goes beyond a standard rating question. It forces respondents to pick the least and most important attribute from a list, helping to identify what your audience truly values.
Set 1 of 3
|Least Important||Most Important|
|Least Important||Most Important|
|Least Important||Most Important|
MaxDiff is best used when you want to identify a preference. The above sample question could be used by a real estate developer determine what resort attributes/features would be preferred for an upcoming project. To maximize the budget, the company should focus on areas that are most important to potential guests. When respondents evaluate this question, features are compared against one another, and a preference can be identified.
If the real estate developer used a matrix or separate rating questions instead of MaxDiff, then respondents would likely rate all attributes as being important. In that scenario, the developer wouldn't have the data needed to maximize the budget; resources would be spread to areas that guests don’t truly value.
Looking at the sample results below, it seems that "mattress comfort" is the most important and a hotel gym is the least important. This conclusion would not have been drawn by traditional question types. For example, if you simply asked "How important is a hotel gym when choosing a resort?", many people would likely rate a gym as being important. But when a gym is compared to other attributes, it becomes less important overall.
Another way to collect preference data is with conjoint analysis, which is a cousin of MaxDiff. Conjoint analysis is best used when collecting multi-level preference data. MaxDiff is best used to collect single-level preference data, like in this real estate example.
To create a MaxDiff survey, simply create a survey as normal, and then add a MaxDiff question where you see fit. You can add an unlimited amount of attributes / features for respondents to evaluate. You can display up to fifty sets (50) or you can display all attributes inside one single set. The more sets you show, the more times individual features will be compared against one another.
To avoid survey fatigue in your respondents it is best to show roughly five attributes per set. To ensure attributes are evaluated evenly, you would want to show each attribute roughly three to five times per question.
The MaxDiff calculator below will help you determine how many sets to show:
The above calculator uses the following equation to come up with the number of sets required. Variable PR is how many times an attribute will be shown to a respondent. (per respondent)
You would want to collect a minimum of 100 responses for a MaxDiff question. 200 or more responses will produce even better data, as there would be more variation in the sets and attribute combinations. If you wanted to filter your MaxDiff results by a subgroup, for example by gender, you would want to collect a minimum of 100 responses for both males and females. The response requirements would be the same for each additional sub group you wish to filter by.
By default, SurveyKing randomizes attributes when showing multiple sets, and has a system in place to ensure attributes display as evenly as possible.
Some research projects require you to define the attributes in each set. For example, maybe you want to compare “Mattress Comfort”, “Room Cleanliness”, and “Hotel Gym” in the first set.
A feature unique to SurveyKing, is the ability to define the attributes that are displayed for each set. To access this feature, click “Define set attributes” within the question editor. The attributes you want to display in the set will be shown in the top section, and the attributes to choose from will be shown in the bottom section. Simply drag from the bottom section to the top section to define the sets to be displayed in the set. You can use the “Next” and “Previous” buttons to cycle though the sets.
To ensure respondents are evaluating attributes against those in the given set, we do not include a button to go back. If you would like to give respondents the ability to start over, enable the "Reset Button" option inside of the question editor. This button will remove all answers from the MaxDiff question and reset the display back to the first set.
MaxDiff analysis is in effect, viewing the results for a MaxDiff survey question. When you go the results page, you will see a data table with each attribute along with the percentage and count of the times it was ranked as most appealing, least appealing, or not chosen.
The attributes in the data table will be ranked based on the score which is computed using the below formula:
If you download the results to a spreadsheet, a column header for most important and least important per set and will be displayed and then the respondent's selection below that.
From the sample data, we can see "Mattress Comfort" has the highest score and is by far the most important attribute when choosing a hotel resort. "Mattress Comfort" is more than double the next positive attribute, "Room cleanliness". The "All-inclusive package" could be quantified as neutral; some respondents think it's important, while an almost equal number of respondents don't. "Customer service" and "Hotel gym" ended up with negative scores, indicating that respondents do not think these attributes are import overall.
While this sample data is an extremely simple, it shows you how MaxDiff goes beyond other standard question types to identify what your audience truly values. This type of data is what should drive decisions by your organization.
Another feature unique to SurveyKing, is the ability to create segment report for a MaxDiff survey question. This is useful to drill down into the MaxDiff analysis and find hidden relationships. For example, you might include a question in your survey that asks for the respondents’ gender. You could then create a segment report (or a cross tabulation report) by gender. The results would include the MaxDiff scoring for "Male" and "Female" in two different tables. You may notice "Males" prefer a certain attribute that females do not prefer or vice versa.
Some MaxDiff projects require a deeper dive into the survey data. Advanced analysis for MaxDiff is best reserved for projects that have a lot attributes, roughly twenty or more. In these types of projects, combination of attributes will vary more between respondents.
If our hotel example had twenty attributes, more possible combinations would be disaplyed; some pairs might never be shown. For example, if we added various attributes related to the "Room" and "Hotel Amenities", some respondents might have sets weighted heavily towards either type.
In this scenario advanced analysis will "borrow" data from other respondents to draw overall conclusions. Logistic multivariable regression analysis is the underlying mathematical theory. Survey answers are "Yes", "No", or in this case "Most Important" and "Least Important". Because of this, regular regression analysis cannot be used; logistic regression (sometimes referred to as logit) is used to determine the probability of answers/selections relating to each other. Here is an introduction to logistic regression, as well as a video that explains the general concept.
SurveyKing doesn’t offer this type of analysis currently, but we can help with custom Excel exports used in statistical programs.
When doing any type of multilinear regression with survey data, the coefficients of each independent variable are the driving factors. In MaxDiff the independent variable would be each attribute, and the dependent variable would be if an attribute is chosen as "Most Important". Many statistical programs use different models to calculate these coefficients, but the outcome would be similar.
These coefficients are usually centered to sum to zero, making them easy to understand. The centered coefficients are called utilities, a term commonly used in survey research. Utilities aren’t an actual unit of measure; they could be best thought of as "happiness".
If we put our simple MaxDiff example above into a program such a R, we would get an output simialr to this table:
We could interpret this as "Matress Comfort" gives .37 units of happiness to our respondents, while "Hotel Gym" takes away .61 levels of happiness. It's not that a hotel gym is bad, all things being equal that gym isn’t adding to the happiness in a way other attribute are. We could also say "Room cleanliness" at .24, gives us double the happiness that "All Inclusive package" does at .12.
Remember, these are coefficients from a regression equation, and the values are slightly different from the simple counting analysis above. With these coefficients determined, we can use them as base for other types of calculations, such as in latent class analysis.
Lean more about utility scores.
Latent classes analysis group similar MaxDiff responses together, in what are called "classes". This is similar to cluster analysis. For example, the software might give us Class #1, which on average are respondents who ranked attributes in roughly this order: "Mattress comfort" > "Room cleanliness" > "All-inclusive package". The ">" symbol here simply means greater than.
Statistical software will ask you how many classes you want. Once the software computes the classes, each attribute will have a regression coefficient, or utility calculated for it. Once the coefficients are calculated you could compute a probability of an attribute being selected as the best within a group. The below table would be an example output.
Learn more about latent class analysis.