An Interactive MaxDiff Survey Tutorial with Sample Data

MaxDiff Analysis Explained

Definition: MaxDiff analysis helps uncover what attributes are most important to your audience. A MaxDiff survey question shows respondents a set of items, asking them to choose what is most and least important. When the results are displayed, each feature is scored giving you actionable data. MaxDiff, short for maximum difference scaling, is sometimes referred to as best-worst scaling.

Basic Concept: MaxDiff goes beyond a standard rating question. It forces respondents to pick the least and most important attribute from a list, helping to identify what your audience truly values.

Interactive MaxDiff Example Question

When choosing a resort, what are the least and most important factors in your decision?

Set 1 of 3

Least Important Most Important
Hotel gym
All-inclusive package
Room cleanliness
Least Important Most Important
Customer service
Mattress comfort
Room cleanliness
Least Important Most Important
Mattress comfort
Customer service
All-inclusive package

When to Use MaxDiff

MaxDiff is best used when you want to identify a preference. The above sample question could be used by a real estate developer determine what resort attributes/features would be preferred for an upcoming project. To maximize the budget, the company should focus on areas that are most important to potential guests. When respondents evaluate this question, features are compared against one another, and a preference can be identified.

If the real estate developer used a matrix or separate rating questions instead of MaxDiff, then respondents would likely rate all attributes as being important. In that scenario, the developer wouldn't have the data needed to maximize the budget; resources would be spread to areas that guests don’t truly value.

Looking at the sample results below, it seems that "mattress comfort" is the most important and a hotel gym is the least important. This conclusion would not have been drawn by traditional question types. For example, if you simply asked "How important is a hotel gym when choosing a resort?", many people would likely rate a gym as being important. But when a gym is compared to other attributes, it becomes less important overall.

Another way to collect preference data is with conjoint analysis, which is a cousin of MaxDiff. Conjoint analysis is best used when collecting multi-level preference data. MaxDiff is best used to collect single-level preference data, like in this real estate example.

How to Create a MaxDiff Survey

To create a MaxDiff survey, simply create a survey as normal, and then add a MaxDiff question where you see fit. You can add an unlimited amount of attributes / features for respondents to evaluate. You can display up to fifty sets (50) or you can display all attributes inside one single set. The more sets you show, the more times individual features will be compared against one another.

Additional Options:

  • Require an answer (can not proceed until answered)
  • You can edit the data labels. For example instead of "Most Important" you could use "Most Likely", or "Highest Priority". The custom labels will show throughout the survey and in your survey results.

How Many Sets, Attributes, and Responses Are Needed?

To avoid survey fatigue in your respondents it is best to show roughly five attributes per set. To ensure attributes are evaluated evenly, you would want to show each attribute roughly three to five times per question.

The MaxDiff calculator below will help you determine how many sets to show:

The above calculator uses the following equation to come up with the number of sets required. Variable PR is how many times an attribute will be shown to a respondent. (per respondent)

PR
*
total attributes

number of attributes per set

You would want to collect a minimum of 100 responses for a MaxDiff question. 200 or more responses will produce even better data, as there would be more variation in the sets and attribute combinations. If you wanted to filter your MaxDiff results by a subgroup, for example by gender, you would want to collect a minimum of 100 responses for both males and females. The response requirements would be the same for each additional sub group you wish to filter by.

By default, SurveyKing randomizes attributes when showing multiple sets, and has a system in place to ensure attributes display as evenly as possible.

Defining the Attributes for Each Set

Some research projects require you to define the attributes in each set. For example, maybe you want to compare “Mattress Comfort”, “Room Cleanliness”, and “Hotel Gym” in the first set.

A feature unique to SurveyKing, is the ability to define the attributes that are displayed for each set. To access this feature, click “Define set attributes” within the question editor. The attributes you want to display in the set will be shown in the top section, and the attributes to choose from will be shown in the bottom section. Simply drag from the bottom section to the top section to define the sets to be displayed in the set. You can use the “Next” and “Previous” buttons to cycle though the sets.

Resetting Answer Choices, Sets

To ensure respondents are evaluating attributes against those in the given set, we do not include a button to go back. If you would like to give respondents the ability to start over, enable the "Reset Button" option inside of the question editor. This button will remove all answers from the MaxDiff question and reset the display back to the first set.

MaxDiff Scoring & Analysis

MaxDiff analysis is in effect, viewing the results for a MaxDiff survey question. When you go the results page, you will see a data table with each attribute along with the percentage and count of the times it was ranked as most appealing, least appealing, or not chosen.

The attributes in the data table will be ranked based on the score which is computed using the below formula:

# times item was selected as best - # times item was selected as worst

# times the item appeared

  • The higher the score, the more appealing an attribute is to your audience
  • A positive score means that the attribute was selected as MOST appealing more often than LEAST appealing
  • A negative score means that the attribute was selected as LEAST appealing more often than MOST appealing
  • A score of zero means that the attribute was chosen as MOST and LEAST appealing an equal number of times OR it has never been chosen

If you download the results to a spreadsheet, a column header for most important and least important per set and will be displayed and then the respondent's selection below that.

Sample Survey Data

From the sample data, we can see "Mattress Comfort" has the highest score and is by far the most important attribute when choosing a hotel resort. "Mattress Comfort" is more than double the next positive attribute, "Room cleanliness". The "All-inclusive package" could be quantified as neutral; some respondents think it's important, while an almost equal number of respondents don't. "Customer service" and "Hotel gym" ended up with negative scores, indicating that respondents do not think these attributes are import overall.

While this sample data is an extremely simple, it shows you how MaxDiff goes beyond other standard question types to identify what your audience truly values. This type of data is what should drive decisions by your organization.

Attribute
Rank
Distribution
Most Important
Least Important
Times Displayed
Score
Mattress comfort 1
13 3 23 0.43
Room cleanliness 2
8 2 20 0.3
All-inclusive package 3
5 4 14 0.07
Customer service 4
3 9 16 -0.38
Hotel gym 5
1 12 17 -0.65
Least Important
Most Important
Not choosen

MaxDiff Analysis By Question Segments

Another feature unique to SurveyKing, is the ability to create segment report for a MaxDiff survey question. This is useful to drill down into the MaxDiff analysis and find hidden relationships. For example, you might include a question in your survey that asks for the respondents’ gender. You could then create a segment report (or a cross tabulation report) by gender. The results would include the MaxDiff scoring for "Male" and "Female" in two different tables. You may notice "Males" prefer a certain attribute that females do not prefer or vice versa.

Advaced Anaylsis for MaxDiff

Some MaxDiff projects require a deeper dive into the survey data. Advanced analysis for MaxDiff is best reserved for projects that have a lot attributes, roughly twenty or more. In these types of projects, combination of attributes will vary more between respondents.

If our hotel example had twenty attributes, more possible combinations would be disaplyed; some pairs might never be shown. For example, if we added various attributes related to the "Room" and "Hotel Amenities", some respondents might have sets weighted heavily towards either type.

In this scenario advanced analysis will "borrow" data from other respondents to draw overall conclusions. Logistic multivariable regression analysis is the underlying mathematical theory. Survey answers are "Yes", "No", or in this case "Most Important" and "Least Important". Because of this, regular regression analysis cannot be used; logistic regression (sometimes referred to as logit) is used to determine the probability of answers/selections relating to each other. Here is an introduction to logistic regression, as well as a video that explains the general concept.

SurveyKing doesn’t offer this type of analysis currently, but we can help with custom Excel exports used in statistical programs.

Attribute Utility

When doing any type of multilinear regression with survey data, the coefficients of each independent variable are the driving factors. In MaxDiff the independent variable would be each attribute, and the dependent variable would be if an attribute is chosen as "Most Important". Many statistical programs use different models to calculate these coefficients, but the outcome would be similar.

These coefficients are usually centered to sum to zero, making them easy to understand. The centered coefficients are called utilities, a term commonly used in survey research. Utilities aren’t an actual unit of measure; they could be best thought of as "happiness".

If we put our simple MaxDiff example above into a program such a R, we would get an output simialr to this table:

Attribute Utility Score
Mattress comfort .37
Room cleanliness .24
All-inclusive package .12
Customer service -.25
Hotel gym -.52

We could interpret this as "Matress Comfort" gives .37 units of happiness to our respondents, while "Hotel Gym" takes away .61 levels of happiness. It's not that a hotel gym is bad, all things being equal that gym isn’t adding to the happiness in a way other attribute are. We could also say "Room cleanliness" at .24, gives us double the happiness that "All Inclusive package" does at .12.

Remember, these are coefficients from a regression equation, and the values are slightly different from the simple counting analysis above. With these coefficients determined, we can use them as base for other types of calculations, such as in latent class analysis.

Lean more about utility scores.

Latent Class Analysis

Latent classes analysis group similar MaxDiff responses together, in what are called "classes". This is similar to cluster analysis. For example, the software might give us Class #1, which on average are respondents who ranked attributes in roughly this order: "Mattress comfort" > "Room cleanliness" > "All-inclusive package". The ">" symbol here simply means greater than.

Statistical software will ask you how many classes you want. Once the software computes the classes, each attribute will have a regression coefficient, or utility calculated for it. Once the coefficients are calculated you could compute a probability of an attribute being selected as the best within a group. The below table would be an example output.

Attribute
Class #1
43%
Class #2
36%
Class #3
21%
Total
Weighted Avg.
Mattress comfort 54.2 61.5 36.1 53.0
Room cleanliness 27.6 22.9 43.2 29.1
All-inclusive package 12.3 4.4 8.6 8.7
Customer service 3.4 9.8 5.9 6.2
Hotel gym 2.5 1.4 6.2 2.9

Learn more about latent class analysis.