Here's What to Do Instead.
In the rapidly evolving landscape of retail media networks, trusting publishers to impartially measure the effectiveness of your advertising spend is risky. Organizations like Amazon, Google, Meta, and Walmart have a vested interest in showcasing their platforms as the most effective avenues for your advertising dollars. When the very entities that stand to gain from increased ad revenue are also the ones measuring its effectiveness, they are grading their own homework, and impartiality becomes a significant concern.
However, it's important to acknowledge that even though some algorithms might be biased, they can still offer useful insights. The key lies in understanding these biases and leveraging the strengths of various models to gain a comprehensive view of your media effectiveness.
The Rise of Retail Media Networks and Publisher Bias
The digital advertising ecosystem has seen a surge in retail media networks, with giants like Amazon and Walmart leveraging their vast customer data to offer targeted advertising solutions. While these platforms promise unparalleled reach and personalization, they also control the narrative around the effectiveness of their own advertising products
Meta's Robyn: Favorable Bias in Open-Source Marketing Mix Modeling
Take Meta's open-source Marketing Mix Modeling (MMM) solution, Robyn, as an example. Meta advertising is seen as a middle-of-the-funnel advertising medium. However, third-party MMM providers didn't show Meta as effective. In response, Meta sought to create a tool that would better reflect the effectiveness of advertising on Facebook and Instagram.
This approach raises questions about the objectivity of the measurement. If the algorithm is designed—or tweaked—to produce more favorable outcomes for the publisher, can advertisers truly rely on these insights to make informed decisions? This is the very essence of the conflict of interest that plagues publisher-provided metrics.
Amazon MMM: Insights from Ex-Amazon Employees
Interviews with individuals who have worked on Amazon's advertising products reveal similar challenges. As a lower-funnel advertising medium, Amazon naturally benefits when users convert after searching for branded products. However, in early MMM iterations developed internally, it was discovered that a previously key ad product appeared completely ineffective when combined with a branded search indicator. This finding made sense to data scientists; after all, a branded ad appearing in the path of branded search results shouldn’t take credit for the sales.
But these findings didn't sit well with management. The prevailing sentiment was, "the ad product can't possibly have zero effect." So to reconcile this, the team adjusted control variables, using total searches instead of branded searches. This tweak conveniently salvaged the perceived effectiveness, sidestepping what could have been an embarrassing revelation for the company.
The Walmart Parallel: Client Stories of Favorable Bias
Clients have shared similar experiences with Walmart's MMM solution, which frequently portrays their platform as the superior advertising medium. This pattern of self-favoring metrics isn't isolated to one or two publishers—it's indicative of a broader industry issue where publishers are both the players and the referees in the game of advertising effectiveness. This undermines the trustworthiness of the insights they provide.
The Need for Impartial Media Effectiveness Measurement
These experiences underscore a critical point: advertisers should be cautious when relying on publishers to measure the effectiveness of their media spend. The inherent conflict of interest can lead to biased results, ultimately affecting the allocation of marketing budgets and the overall success of advertising campaigns.
Yet, it's also true that publisher-provided algorithms, despite their biases, can still contribute valuable insights when used appropriately. By recognizing the strengths and limitations of each model, advertisers can piece together a more accurate picture of media effectiveness.
MMM Labs: Your Solution for Transparent Marketing Mix Modeling
At MMM Labs, we are committed to impartiality and transparency. Our focus is on achieving the most accurate results possible by rigorously testing all models—including publishers' algorithms when available. We test algorithms through a conflict/cooperation framework where some models may disagree while others align, providing consensus. We don't shy away from divergence; instead, we embrace it by building a multiple-model framework that surfaces the best options for our clients. This approach serves our clients' interests in obtaining the most optimal budget allocation possible.
By breaking down the walled gardens of media effectiveness and providing an unbiased analysis, we empower advertisers to make data-driven decisions that genuinely benefit their business.
Don't leave the assessment of your advertising effectiveness in the hands of those who might skew the results. Choose impartiality. Choose transparency. Schedule a demo with MMM Labs.
