Survey Ranking Question Explained

Definition: A survey ranking question asks respondents to rank attributes based on preference. Depending on the question settings, respondents need to rank all attributes in the set, or only rank a certain number, such as the top three. For each attribute, results include a ranking score, a first-place count, along with the distribution of overall ranking. A ranking question is used in any type of research project to help identify preferences.

A ranking question can also be referred to as an ordinal-polytomous survey question. Ordinal meaning relative to an order and polytomous meaning more than two possible variables.

There two types of ranking questions you can include in your survey. Depending on your project one type might be more beneficial than others. Both types of ranking questions will display the results in the same manner.

A click ranking question asks respondents to rank only a certain number of attributes, such as the top three (3). This ranking type if best used when you have a long list of attributes and want to identify the most valued items in the set. Below is an example of a click ranking question.

This type could be used by food manufacturer wanting to find the most preferred ice cream flavors. In this example the food manufacturer only would want to focus on the most valued items, asking respondents to ranking anything past the top three contributes to survey fatigue, and results in low quality data (respondents would spend time thinking about ranking bottom flavors that aren’t important).

A standard ranking question asks respondents to rank all attributes in a set. This ranking type is ideal when you want to know the preference data for every single attribute. Generally, this type is only used when you have a shorter list of attributes. Asking respondents to rank a long list of attributes would result in survey fatigue and low-quality data.

This type could be used in an employee survey that is trying to best identify how improve the workplace. All of the attributes are areas the employer would want to focus on, but most of initial efforts should be geared towards the top priority.

To create a ranking survey, simply create a survey as normal, and then add a ranking question where you see fit. You can toggle between the standard ranking and click ranking as needed. An unlimited number of attributes can be added to each ranking question. To limit survey fatigue, we recommend including no more than ten (10) attributes.

Generally, ranking questions that require more than ten (10) attributes should be using a MaxDiff question.

• Require an answer (can not proceed until answered)
• For the click ranking, you can define how many attributes respondents need to rank - such as the top two (2) or three (3)
• The click ranking also includes an option to clear the selected choices so respondents can start over

While a ranking question can be extremely beneficial, there are limitations. The biggest limitation is quantifying the ranking differences. The food manufacturing survey might have "Banana" and "Chocolate" as the top two. But the distance between those attributes are unknown. People might like “Banana” 100x compared to “Chocolate”. This data point is crucial, because if true, the food manufacturer should be focusing their efforts on the "Banana” to maximize revenue.

Another downside to a ranking question is the survey fatigue. A long list of attributes takes a lot of effort for respondents to evaluate and can be prone to errors. Even with using the click ranking, respondents would need to evaluate all attributes at once before selecting their top three (3).

A solution to both problems is MaxDiff. MaxDiff can be used to help identify what is most and least important (or most/least desired) from a list of attributes. The basic concept is that respondents are shown a small subset of the total attributes (like a random set of five out ten attributes) and pick what is most and least important. Respondents are shown multiple sets, meaning attributes are compared against one another in. In the manufacturing exmaple, "Banana" would have a much higher score than "Chocolate", since both were compared against eachother.

The ranking score is a weighted calculation. Items ranked first are given a higher value or "weight." The score, computed for each answer option/row header, is the sum of all the weighted values. For example, if there are 5 options, the weighted sum for an option that was placed in the first position (1) would be worth 5. The points are added up and the item with the highest points is ranked first.

The results include how many times an item was ranked first and also shows the distribution of ranking with a small bar chart. The color coding of the ranking distribution makes it easy to see attributes that have net top/bottom rank.

The Excel export will display each attribute as column with the respondent's ranking. If an attribute was not ranked, the column would be blank.

The below sample data are the results from the click ranking survey used by the food manufacturer. Each attribute has its own row with the ranked distribution, first place counts, and total score .

A feature unique to SurveyKing, is the ability to create segment report for a ranking question. This is useful to drill down into the data and spot hidden relationships. For example, you might include a question in your survey that asks for the respondents' gender. You could then create a segment report (or a cross tabulation report) by gender. The results would include the table shown above for both "Male" and "Female". You may notice "Males" prefer a certain attribute that females do not prefer or vice versa.