Survey Ranking Question Explained

Definition: A survey ranking question asks respondents to rank attributes based on preference. Respondents can rank all attributes in the set or only rank a certain number, such as the top three. For each attribute, results include a ranking score, a first-place count, and a distribution of the overall rank. A survey ranking question is used in research projects to help identify preference.

A ranking question is also referred to as an ordinal-polytomous survey question. Ordinal meaning relative to an order and polytomous meaning more than two possible variables.

Below are examples of each ranking type. Depending on your project one type might be more beneficial than others. Both types of ranking questions will display the results in the same manner.

A click ranking question asks respondents to rank only a certain number of attributes, such as the top three (3). This ranking type is used when you have a long list of attributes and want to identify the most valued items in the set. Below is an example of a click ranking question.

A food manufacturer who wants to find the most preferred ice cream flavors could use a click ranking question. In this example, the manufacturer should focus on the most valued items. Asking respondents to rank anything past the top three contributes to survey fatigue and results in low-quality data (respondents would spend time thinking about ranking bottom flavors that aren’t important).

A standard ranking question asks respondents to rank all attributes in a set. This ranking type is ideal when you want to know the preference data for every single attribute. Generally, this type is only used when you have a shorter list of attributes. Asking respondents to rank a long list of attributes would result in survey fatigue and low-quality data.

An employer could use this type of ranking question in an employee survey to identify how to improve the workplace. The employer wants to focus on all of the attributes, but initial efforts should be geared towards the top priority.

To create a ranking survey, create a survey as normal, and then add a ranking question where you see fit. You can toggle between the standard ranking and click ranking as needed. You can add an unlimited number of attributes to each ranking question. To limit survey fatigue, we recommend including no more than ten (10) attributes.

Generally, ranking questions that require more than ten (10) attributes should be using a MaxDiff question.

• Require an answer (can not proceed until answered)
• For the click ranking, you can define how many attributes respondents need to rank - such as the top two (2) or three (3)
• The click ranking also includes an option to clear the selected choices so respondents can start over

While a ranking question can be highly beneficial, there are limitations. The most significant limitation is quantifying the ranking differences. The food manufacturing example might result in "Banana" and "Chocolate" as the top two flavors. But the distance between the two flavors is an unknown variable. People might like "Banana" 100x compared to "Chocolate." This data point is crucial because, if true, the food manufacturer should be focusing their efforts solely on "Banana" to maximize revenue.

Another downside to a ranking question is the survey fatigue. A long list of attributes takes a lot of effort for respondents to evaluate and can be prone to errors. Even with using the click ranking, respondents would need to evaluate all attributes at once before selecting their top three (3).

A solution to both problems is MaxDiff. MaxDiff can be used to help identify what is most and least important (or most/least desired) from a list of attributes. The basic concept is that respondents are shown a small subset of the total attributes (like a random set of five out of ten attributes) and pick what is most and least important. Respondents are shown multiple sets, meaning attributes are compared against one another. In the manufacturing example, "Banana" would have a much higher score than "Chocolate" since respondents compared both against each other.

The ranking score is a weighted calculation. Items ranked first are given a higher value or "weight." The score, computed for each answer option/row header, is the sum of all the weighted values. For example, if there are five options, the weighted sum for an option that a respondent placed in the first position (1) would be worth 5. The points are summarized, and the item with the highest points is ranked first.

The results include how many times an item was ranked first and also displays the ranking distribution with a small bar chart. The color-coding of the ranking distribution makes it easy to see net top/bottom rank attributes.

The Excel export will display each attribute as a column with the respondent's ranking. If a respondent did not rank an attribute, the column would be blank.

The below sample data are the results from the click ranking survey used by the food manufacturer. Each attribute has a row with the ranked distribution, first place counts, and total score.

A feature unique to SurveyKing, is the ability to create a segment report for a ranking question. This report type is helpful to drill down into the data and spot hidden relationships. For example, you might include a question in your survey that asks for the respondents' gender. You could then create a segment report (or a cross-tabulation report) by gender. The results would include the table shown above for both "Male" and "Female". You may notice "Males" prefer a particular attribute that females do not prefer or vice versa.