Introspection
Understand the data powering search, discovery, and ads rankings with full transparency.
Request Retrieval Score
The marketplace’s original quantitative ranking of the given listing.
Platform Retrieval Rank
The ranking position from the platform’s own retrieval system.
Predict Click
A prediction of the likelihood a user will click on that particular listing.
Predict Post Click Purchase
A prediction of the likelihood that the user will purchase after clicking on the listing.
Blender Utility Score
An overall score calculated for each item, usually as a function of the model predictions, text matching, and/or other metrics.
Blender Allocation
The Blender sorting rule, found in the blender_config sheet, that is being applied to that specific position.
Easy Accessibility
After each search query, an Introspection report is automatically generated on Google Sheets, allowing for collaborative analysis and debugging. The Introspection report houses all data pertinent to the ranking, including Promoted’s model predictions and information about the user, content, environment, and query.
Understanding search ranking
A visitor on FindTheBestPizza.com searches for ”Pizza Place Chicago” and sees ranked results. Employees at FindTheBestPizza.com can use Introspection to analyze why these listings are displayed and how the ordering was chosen for this particular visitor and search query.
✱FindTheBestPizza.com
Unexpected search results? Meet Introspection
Addressing unexpected results
Step 1: Explanation
Searching for York Street Pizza should have ranked the pizza shop first, instead of fourth. What happened?
The team examines Promoted’s ranking and the platform’s retrieval results.
The team looks at the Blender Allocation, which shows that Blender rule 1 is being applied to the first three positions, Blender rule 2 is being applied to the fourth, and so on. These rules can be found in the blender_config sheet in the Introspection report.
The Introspection report also shows selected features of the listings. Mama’s Pizzeria, Joe’s Pizza, and Chi City Pizzeria all have a higher average review score than York Street Pizza, potentially explaining the ranking order.
Text matching tells a different story. Three words in the search query matched York Street Pizza — a perfect match. But why is York Street Pizza still fourth?
Lastly, the distance feature shows that York Street Pizza is the closest to the user of all of the ranked shops.
The team surmises that word matching and distance are being undervalued in the sorting rule, and the average review score may be overvalued. The team now investigates the Blender rules.
Step 2: Solution
The next step is to examine the Blender sorting rules and see whether a technical issue or business decision is the root cause of the incorrect ranking.
Using the rules found in blender_config, FindTheBestPizza.com and Promoted look at the ‘sort allocation’ rules that correspond to positions in the search results.
Mama’s Pizza, Joe’s Pizza, and Chi City Pizzeria were ranked at the top because they had a better average review score then York Street Pizza, and the title matching had an issue preventing York Street Pizza from overtaking them. This is discovered from the rules and scores located in the Introspection report.
Each Blender rule shows the specific metrics applied to each position. Here’s an example Blender rule:
SORT_ALLOC(text_match, distance, review_score, utility_score)
This rules shows that we first sort by text matching, then nearest distance, then highest reviews, and finally by a utility score based on click and purchase model predictions.
Some positions may factor in a unique ’challenger’ that prioritizes a specific business goal, like new seller activation.
Looking at the Blender rules, the team sees the problem. The text matching parameter — which should have returned a high score for York Street Pizza — required a minimum match of four words, likely to prevent unintended matches. However, the team now decides to modify this rule to a minimum of two words, for a better balance of accuracy and relevancy.
York Street Pizza will now be ranked first in the results for this search query.
* Each marketplace will have different Blender rules, reflecting its own unique needs
Corrected results
With Introspection, the team was able to fix the search results! Now with the title match parameter set at a minimum of two words, the expected result is returned.
The power of Promoted and Introspection
Promoted’s Introspection empowers FindTheBestPizza.com to monitor search results, delve into the metrics shaping these outcomes, and swiftly solve discrepancies. Moreover, this tool facilitates internal monitoring and enables Promoted’s clients to make decisions independently within their marketplace.
Through full transparency and customizable control, customers can navigate the platform on their own terms, fostering trust and satisfaction.