An Interactive Conjoint Survey Tutorial with Sample Data

Conjoint Analysis Explained

Definition: Conjoint analysis is a survey-based research technique used to quantify how people value the individual features of a product or service. A conjoint survey question shows respondents a set of concepts, asking them to choose or rank the most appealing ones. When the results are displayed, each feature is scored, giving you actionable data. This data can help determine optimal product features, price sensitivity, and even market share.

Why Is It Important? Conjoint analysis goes beyond a standard rating question. It forces respondents to pick what product concepts they like best, helping identify what your audience truly values.

Interactive Conjoint Analysis Example Question

Conjoint analysis could be used by any company wanting to do product reach, in this example a restaurant chain is used. If the chain wanted to release a new ice cream slot on their dessert menu, conjoint analysis could help determine optimal flavor, size and price like in the example conjoint survey question below.

If we offered a new menu item for ice cream, which of the following options would be most appealing to you? Please make one choice per set. If no options look appealing, choose "None".

Set 1 of 3
Reset choices
3 options
Flavor
Size
Price

Option #1

Choose

Chocolate
Small
$2 USD

Option #2

Choose

Vanilla
Small
$5 USD

Option #3

Choose

Cookie Dough
Large
$5 USD

None

Choose

-
-
-

Option #1

Choose

Strawberry
Medium
$2 USD

Option #2

Choose

Cookie Dough
Large
$2 USD

Option #3

Choose

Vanilla
Large
$2 USD

None

Choose

-
-
-

Option #1

Choose

Cookie Dough
Small
$5 USD

Option #2

Choose

Chocolate
Large
$5 USD

Option #3

Choose

Vanilla
Medium
$2 USD

None

Choose

-
-
-

This is an interactive example of choice based conjoint

When to Use Conjoint Analysis

Conjoint analysis is used when you want to identify what people value the most in a product or service. Conjoint analysis is commonly used to test new product concepts. The above sample question could be used by a restaurant chain to determine additions to a dessert menu. The chain should focus on concepts that provide the most value for their customers. When respondents evaluate this question, concept features are compared against one another, and a preference for each feature can be identified.

With standard rating questions, it would impossible to ask about prices; a separate rating question for each flavor and size combination is needed. This would lead to fatigue and poor-quality data. Conjoint analysis solves these problems with one simple survey question.

How to Conduct a Conjoint Analysis Study

Oftentimes, preliminary data needs to be collected prior running your conjoint study. This could include a survey with a MaxDiff or a Van Westendorp question to determine important product features or an acceptable price range. The preliminary survey acts as a baseline to help limit the number of total conjoint concepts, reducing survey fatigue and increasing quality data.

You also want to organize and layout any meta data that might be used. For example, if you wanted to segment your research by country (USA vs European customers), you need to make sure that data is valid, complete, and accessible by your team prior to running the conjoint study. If meta data is unavailable, you can add additional questions to the survey prior to the conjoint question.

With the preliminary survey data in hand and meta data organized, you can now create your conjoint analysis study. All product attributes and levels can be quickly uploaded, meta data can be included, and additional questions be added to ensure a successful project.

Conjoint Analysis Terminology

Conjoint analysis is an advanced research technique which uses a variety of unique terminology. To help you get a full understanding, here is a list of commonly used conjoint terminology:

Attribute

These are the high-level product features that respondents will evaluate. Attributes are the very first column in the above sample question. Our example has the following features: flavor, size, and price. If you did a study about a new car offering you might have features such as color, make, model, MPG, and tire type. There is a limit of 20 attributes on the SurveyKing platform.

Levels

These are the items that make up a single attribute. In our example the "Flavor" attribute has levels of "Chocolate", "Vanilla", "Cookie Dough", and "Strawberry". When you create the conjoint survey, you define an attribute and levels that go with each attribute. There is a limit of 15 levels on the SurveyKing platform.

Concept

This a combination of all your attributes and levels, which in turn creates a hypothetical product. In the above example, concepts are the columns that respondents choose. Concepts are sometimes referred to as "cards" in statistical software. There is a limit of 7 concepts on the SurveyKing platform.

Set

Also referred to as a task, a set contains multiple concepts or product offerings. Respondents will choose one concept per set, and then be shown a new set of concepts. There is a limit of 20 sets on the SurveyKing platform.

Part-Worths/Utilities

This is the most important term in conjoint analysis. It defines how a respondent values each attribute level. When all the utilities for all respondents are analyzed, an overall product value can be determined. Utilities are the output of a regression equation.

Utilities have no scale compared to other conjoint projects you run. They only matter in the context of the current question you are looking at.

Sometimes utilities are called "part-worths", or "part-woth utilities". We simply use the term "utility".

Types of Conjoint Analysis

Choice-Based Conjoint

This is the most common form of conjoint. The example question above is a choice-based conjoint question. Respondents pick the most appealing concept for each set. Each set contains a random set of concepts, that are evenly distributed. This type of conjoint best simulates buyer behavior, since each set contains hypothetical products (concepts). When respondents choose a full profile, preferences can be calculated from the tradeoffs made. (e.g even though "Strawberry" isn't a preferred flavor, if the price was low enough, it would still provide consumer utility")

As with most conjoint studies, preliminary research is important to reduce the number of attributes and levels to choose from. The less attributes, and levels, means less possible full concepts, reduced survey fatigue, and better quality data. For example a MaxDiff or ranking survey could be completed to find the top four ice cream flavors.

Currently this is the only type of conjoint offered by SurveyKing.

Best/Worst Conjoint

Sometimes referred to as MaxDiff conjoint. Similar to choice-based conjoint, this method shows respondents a set of concepts. In each set, respondents are asked to pick the most/least (or best/worst) concepts. This approach is used when a product or service has features that cause both positive and negative reactions. An example could be studying how parents select daycare. The number of full-time faculty would draw a positive reaction. The percentage of fellow students that are economically disadvantaged could produce a negative reaction.

This is a future addition to the SurveyKing platform.

Adaptive Conjoint

This method is also similar to choice-based conjoint. Respondents pick the most appealing concept for each set, except with this method, the next set of concepts are not random, but are tailored based on the previous answers. This method is more engaging to respondents, and can help fine-tune the data.

This is a future addition to the SurveyKing platform.

Full-profile conjoint analysis

This method displays a large number of concepts and asks respondents to rate each one based on the likelihood of purchase. This method is outdated and was primarily used prior the introduction of survey tools that offer choice-based conjoint. Asking to rate lots of concepts at once is error prone, easily causes fatigue, and yields poor quality data.

Rating or Ranking Conjoint

Ranking and rating conjoint was the method used for full-profile conjoint analysis. As software has progressed, it is now possible to conduct rating or ranking conjoint similar to a choice-based conjoint. Respondents are shown a set of concepts and asked to rank or rate each concept. They could rank by entering a value for each concept, which sums to 100 for each set, or they could simply enter a number-based a on scale. This method is also sometimes referred to as "Continuous Sum Conjoint".

This is a future addition to the SurveyKing platform.

Menu-Based Conjoint

Menu-based conjoint is a new conjoint method. This method gives respondents the ability to pick multiple levels from a menu. For example, a car manufacturer could ask respondents to choose a base model and price, just like choice-based conjoint. But then they could also ask to check a box for each additional feature desired such as "Alloy Wheels for $1,500", "Sunroof for $1,000", or "Parking Assist for $1,500".

This method is much more advanced in terms of front-end programming and back end statistics than choice-based conjoint. Often custom solutions need to be built for a company wishing to create this type of project.

Creating a Conjoint Survey

Any survey that contains a conjoint question can be referred to as a conjoint survey. SurveyKing currently only offers choice-based conjoint. Here are the steps needed to add a conjoint question:

  1. Navigate to the "Builder" page of your survey
  2. Click on the "conjoint" element box and drag into your questionnaire, or click the "Insert question" dropdown, to add a conjoint question at the end of a specific page
  3. To add a new attribute, click "Add attribute" within the conjoint builder. Levels for the attribute will be shown to the right of each attribute.
  4. Choose how many sets and concepts you want to display
  5. Select any options to further customize the question
Note: Once you collect responses, changes to the question will be limited to ensure data integrity. The number of sets and conjoint subtype will be locked, and you will be unable to add or remove attributes / levels. We recommend fully designing your conjoint survey before collecting live data. Test responses can be deleted enabling you to again add or edit attributes / levels prior to pushing your project live.

Conjoint Survey Options

  • "None" choice - This option will add one additional card, or column, per set that says "None" This option is marked by default. This setting reflects the real world where consumers can choose to not buy a product. This setting should be excluded from projects where customers are forced to pick an option, such as a government service.
  • Reset choices - with this option, respondents can start back at the beginning. All answers for the question will be cleared and the first set will be displayed when the "reset" button is clicked. We recommend reserving this option for specific circumstances, as it could lead to second guessing and low quality data.

How Many Attrbutes, Levels, Concepts, & Sets are Needed?

An ideal conjoint question will have roughly 5 attributes (rows), 4 concepts per set (columns), and roughly 5 - 10 sets. This will help ensure respondents are not fatigued. A detailed breakdown is below:

  • Attributes - Roughly 5 attributes with no more than 10 total levels per attribute. Having less levels per attribute, ensures various concepts can be shown more often.
  • Concepts - Roughly 4 concepts to show each set. Too many concepts per set, and you risk respondents not making effective choices. The total amount concepts available is calculated by multiple the number of levels in each attribute. In the example above we had four flavors, three sizes, and two prices. Total concepts available would be equal to 4 * 3 * 2 = 24. Ideally this number should be no larger than 50. The more total concepts, the harder it becomes to draw meaningful conclusions.
  • Total Sets - showing no more than 10 total sets to respondents to avoid survey fatigue. Generally 3-5 are best

How Many Responses are Needed?

We recommend collecting at least 100 responses for each segment being researched. For example, if you wanted to research both males and females, you would want to collect 100 responses for both.

Conjoint Analysis Scoring & Results

Conjoint analysis uses regression to calculate how different attributes and levels are valued.

Because conjoint uses categorial data (a name like ice cream flavor) instead of continuous data (a number like temperature), a special type of regression is used called logistic regression. Just like any regression equation, the result of this regression calculates coefficients. These coefficients are referred to as "utilities".

Utility is not a standard unit of measure. It can be thought of as "happiness". Let's say a bunch of respondents choose concepts that contain "Cookie Dough" and only a few choose concepts with "Vanilla". Even without doing the math, you can imagine that the coefficient for "Cookie Dough" would be higher than the coefficient for "Vanilla".

To illustrate this concept, we ran the above ice cream example with 20 respondents. Below is the analysis of those responses. This analysis includes the utilities for each level in addition to the relative importance of each attribute.

Sample Survey Data

Attribute Importance Level Utility
Flavor 61%
Chocolcate
Vanilla
Cookie Dough
Stawberry
.44
7.13
14.11
0
Size 17%
Small
Medium
Large
0
4.03
1.61
Price 22%
$2 USD
$5 USD
0
5.06

Walking Through the Analysis

The utilities in the last column are the output of regression analysis. Next to each number is small bar chart for visual representation.

Remember, utilities are not an actual unit of measurement, and could be thought of happiness. If we look the above table, the "Cookie Dough" flavor has a utility of 14 and the "Vanilla" flavor has utility value of 7. We could interpret this as "Cookie Dough has double the happiness of Vanilla". In terms of product size, we could say "a medium size gives a little more than 2 units of happiness than a large".

The importance column is the weighted difference in utilities ranges for the product levels. You can see that flavor has the level with the largest difference of roughly 7. The larger the utility differences for an attribute, the more important they are to consumers. To get a large difference like we see with cookie dough, a lot of respondents had to choose concepts with that flavor. We know the other levels are evenly distributed, meaning that regardless of size or price, cookie dough was a big driving factor in decision making. Here's how you would calculate the importance:

Take the largest number for each level, and sum: 14.11+4.03+5.06 = 23.02

Divide each of the highest levels by this number. The calculation for flavor importance is 14.11 / 23.02 = 61%

Overall, it looks like "Cookie Dough" is the preferred flavor, "Medium" serving size, with a price of "$5 USD". "Vanilla" would also be a good addition to the menu as it bring a high amount of utility. You'll notice that the higher price has a higher utility value than the lower price. Respondents might have a perception that a higher cost has a better taste or better quality. The preference for "Medium" might also be tied to consumers being health conscious. This summarizes why conjoint is so important. Not only did we find the two optimal flavors, we also found the right size, and correct price point. This data would be almost impossible to capture without conjoint analysis.

Statistical Details

SurveyKing uses ChoiceModelR, a package in the R statistical program to compute conjoint utilities. ChoiceModelR calculates a coefficient using logistic regression with maximum likelihood for each attribute level by respondent. When the analysis is complete, utilities for each level are averaged. The output of our example can be found in this Excel file.

We color coded the Excel file for each attribute level. You can see row 22 has an average subtotal. This is the average utility for a specific level. The regression equations use effects coding, to ensure each attribute in total sums to 0. Because of this you will notice the excel file contains negative utilities. We shift each number by a constant, to eliminate negatives, and put the baseline to 0. The dark blue flavor columns were adjusted by 5.43 prior to the results being loaded into our dashboard. Having a 0 baseline makes the data easier to interpret.

Data used to populate ChoiceModelR:

  • Data Matrix - See this Excel file, which is the input for the ice cream example. The first row of each card set contains the card number chosen (column G). The first card selected was 4. This is because the none option was selected. In the case of the none option, the highest index + 1 is the card selection. This is the input required for ChoiceModelR. Other programs use an output similar to this file. You'll see it's the same setup, except column G simply has a 1 if the card is selected, or 0 if not selected. An additional row is added for the none column.
  • R - The total number of iterations of the Markov chain Monte Carlo (MCMC chain) to be performed. Default value: 4000.
  • Use - The number of iterations to be used in parameter estimation. Default value: 2000.
  • Keep - The thinning parameter defining the number of random draws to save. Default value: 5.
  • wgt - the choice-set weight parameter; possible values are 1 to 10. This parameter only needs to be specified if estimating a model using a share dependent variable. Default value: 1.
  • xcoding - A number that specifies the way in which each attribute will be coded. We code each attribute as categorical, which is the value 0. Prices could technically be labeled as continuous, but for ease of calculations and consistency, we code all variables are categorical.

Conjoint Analysis by Question Segments

Sometimes it's important to analyze different segments, such as gender. To create a conjoint segment, simply add a multiple-choice question to your survey for each segment you wish to study. In the reporting section, you can choose "Conjoint Segment Report". From here, select the appropriate question, and the report will output a data table for each answer. Using the ice cream example, you may notice "Males" prefer "Cookie Dough", while "Females" prefer "Vanilla". These are additional data points to fine tune your marketing efforts.

Here is an interactive example of a conjoint comparison report, unique to the SurveyKing platform. The first question in the asks for gender and the second question asks for a preferred ice cream concept. You can see males preference "Cookie dough" with a utility of 23.06, while females prefer "Vanilla" with a utility of 25.63. Each gender segment lists flavor as the most important attribute. The report also includes a segmented ranking of concepts.

Analyzing Concept Profiles

A powerful benefit of conjoint analysis is quantifying how each concept would fare in the market. We can easily see the product with most utility would be Cookie Dough, Medium, for $5 USD. But what about the top 3 products? Or the bottom 3 products? In the ice cream example, there were 24 hypothetical products. Unique to the SurveyKing platform is the ability to scroll through each concept in ranked order, to see what profiles faired the best or worst (or offer the most utility). The reporting section will automatically include the table shown below:

Rank Flavor Size Price Total Utility
1Cookie DoughMedium$5 USD 488.02
2Cookie DoughLarge$5 USD 436.91
3Cookie DoughSmall$5 USD 403.10
4Cookie DoughMedium$2 USD 381.51
5VanillaMedium$5 USD 341.31
6Cookie DoughLarge$2 USD 330.40
7Cookie DoughSmall$2 USD 296.58
8VanillaLarge$5 USD 290.20
9VanillaSmall$5 USD 256.39
10VanillaMedium$2 USD 234.80
11ChocolateMedium$5 USD 200.82
12StawberryMedium$5 USD 191.44
13VanillaLarge$2 USD 183.68
14VanillaSmall$2 USD 149.87
15ChocolateLarge$5 USD 149.71
16StawberryLarge$5 USD 140.33
17ChocolateSmall$5 USD 115.90
18StawberrySmall$5 USD 106.51
19ChocolateMedium$2 USD 94.31
20StawberryMedium$2 USD 84.93
21ChocolateLarge$2 USD 43.20
22StawberryLarge$2 USD 33.81
23ChocolateSmall$2 USD 9.38
24StawberrySmall$2 USD -

To get these figures from the Excel output file, you could create a table with all possible combinations, and use sum product to calculate to total utility. Here is an example.

Conjoint Analysis Tips

  • Keep descriptions simple - For both attributes and levels, keep the descriptions as short as possible. This will make picking choices easier and reduce survey fatigue.
  • Images - Because of limited space we recommend using images inside of each level sparingly. When images are used we recommend each image to be custom made for this project with a size no larger than 150px X 150px.
  • Additional descriptions - Let's say you are researching a new phone. If you have a weight level with 7oz and 11oz, people won't be able to gauge that difference. You would want to say (ideally in the question text), "Use the iPhone 7 as a baseline weight, that weight would be considered average" Then the size product labels would be "Light", "Average", "Heavier".
  • Be aware of incorrect conjoint content - There is a popular online video that explains conjoint analysis in Excel. The video uses "Dummy Variables" to compute the regression. This would be incorrect for two reasons. Excel cannot do logistic regression without any addons. Also, removing dummy variables is unnecessary if logistic regression is done correctly. The video codes a three-level attribute with 1's and 0's, which results in collinearity. Logistic regression assigns categorical data to a unique number. Just like in our example, a four level attribute would have the numbers 1, 2, 3, or 4, depending on what concepts were displayed.