Sold! For More Than What You Should have Paid
The Move to First-Price Cannot Cure What Ails Programmatic Auctions
By Ben Hovaness
In today’s digital landscape, programmatic auction structures are changing at a rapid pace, often leaving buyers losing auctions when they shouldn’t. While SSPs and DSPs have shifted towards first-price auctions, claiming this will help solve the problem, they have disputed that this change will increase costs for buyers. We recently put these claims to the test in the real-world and discovered that moving to first-price auctions increases costs, even with mitigation measures in place. Hearts and Science, Executive Director of Digital Activation, Ben Hovaness has a three-tiered plan to make bid strategy straightforward and help clear the air around programmatic auctions.
Increasing amounts of programmatic inventory are being sold in first-price auctions (“1PA”, winner pays their bid) in many of the major supply-side platforms (SSPs), a departure from the second-price auction model (“2PA”, winner pays second-highest bid) that has historically prevailed in programmatic ad sales. SSPs moving to 1PA have claimed that this change should be beneficial for buyers, as it resolves the current 2PA design flaw that sometimes results in the highest bidder losing an auction. Certain SSPs and demand-side platforms (DSPs) have also claimed the move to 1PA should be cost-neutral for buyers once they adapt their bidding strategies.
Auction theory predicts that this cost-neutrality claim is inaccurate, and our real-world testing bears this prediction out. Based on our data, for a given set of inventory, advertisers are better off buying through an SSP conducting a 2PA. This is true even when a DSP auto-bidder is used to buy the 1PA inventory; in this case the 2PA-priced inventory still has a lower price. Advertisers can and should adjust their programmatic supply path accordingly, both by shifting investment towards 2PA-priced inventory when possible and by negotiating fixed-price private marketplace (PMP) deals with their biggest publishers. However, these near-term mitigation strategies are not the end of the story.
The programmatic auction system should be restructured as a classic single-stage 2PA. This would retain the benefits of the present 1PA model (highest bidder always wins) while adding the benefits of a 2PA system (bidding truthfully is optimal). This could be achieved through the following reforms:
- SSPs must pass all bids for an impression, not just the highest, through to the publisher
- Publishers must conduct a clean second-price auction. Price floors can be set for inventory, but the practices of conducting a first-price auction and price discrimination (variable price floors per advertiser not tied to a PMP deal) must end.
- Both SSPs and publishers must submit to audits by a neutral third party to ensure compliance with points 1 and 2
This combination of changes would make the marketplace much more advertiser-friendly by making bid strategy straightforward and clearing the air around programmatic auctions, while maintaining or improving yields for publishers. Hearts will be working with partners across the programmatic supply chain to bring this vision to life.
Auction game basics
Auctions have been used for millennia to allocate difficult-to-price items to buyers. While a tremendous variety of auction forms exist, this paper focuses on two: sealed-bid first price auctions (1PA) and sealed-bid second price auctions (2PA), the two forms that dominate the digital media landscape. Buyers must pursue different bid strategies as the auction form dictates (e.g. 1PA vs 2PA) but will typically1 seek to maximize the difference between the value they or their clients place on an item (their “private value”) and what they ultimately pay for it (the “clearing price”). This difference is called a “buyer surplus.”
In a 1PA, the highest bidder wins and pays what they bid, regardless of the behavior of other bidders. By contrast, in a classic 2PA the highest bidder wins the auction but only pays the second-highest bid amount. This is true even if the highest bidder values an item dramatically more than the second-highest bidder; in every case the winner will pay the second-highest bid amount. By extension, the same bid amount will yield a different clearing price depending on whether the auction is 1PA or 2PA.
This has a direct bearing on the optimal bid strategy for each of these auction forms. The dominant strategy in 1PAs is to guess at the lowest bid amount between the bidder’s private value2 and the second-highest bidder’s bid that will result in an auction win. Contrary to what some commentators have written, it is sub-optimal to bid your private value in a 1PA: doing so eliminates the possibility of a buyer surplus.On the other hand, 2PAs are “dominant-strategy incentive compatible” (DSIC) meaning that the best, or “dominant” strategy for bidders is to bid their private value.2
The simple dominant bid strategy afforded by the 2PA form benefits both buyers and sellers. Bidders do not have to concern themselves with determining an optimal bid distinct from their private value. Sellers, assuming a constant set of bidders, should receive an average selling price identical to what they would have received in a first-price auction (more on this later).
Thinking about the marketplace more dynamically, the elegance of the 2PA should give additional bidders the confidence to buy ads at auction - driving up publisher revenues!
Because of these benefits, second-price auctions have been widely adopted in digital media, with Google and Facebook most prominent among the firms using second-price auction variants to sell billions of dollars of digital ad inventory each year. Since all demand for Google’s search ads and Facebook’s social ads flows to those firms directly through their proprietary systems, all bids are aggregated into each relevant ad auction. As such these auctions follow the classic 2PA auction design closely. The programmatic 2PA, on the other hand, diverges from the classic design dramatically.
Demand for programmatic ads is diffused across many DSPs, so digital publishers rely on aggregation systems called SSPs to coalesce demand for their inventory. Not every DSP is integrated with each SSP, so publishers employ multiple SSPs to capture as much demand as possible. SSPs were originally given the opportunity to bid in sequence, with each publisher working its way down its list of partner SSPs until one transmitted a bid above the price floor. This sequential, “waterfall” approach was simple enough, but resulted in forgone revenue for publishers since the SSPs never had to compete directly in a unified auction. “Header bidding” was developed to solve this problem, forcing all SSPs to bid simultaneously into an
instantaneous, yield-maximizing publisher auction.3
The implementation of header bidding, however, broke the logic of classic 2PA. Now, the SSPs would independently receive bids from DSPs, conduct a 2PA, then pass the clearing price to the publisher. When presented with multiple clearing prices from SSP 2PAs for a given impression, the publisher conducts a 1PA, selecting the highest clearing price from the set of SSP auctions. The two-stage nature of this auction - 2PA at the SSP, then 1PA at the publisher - destroys the DSIC quality of a classic 2PA, because the highest initial bidder has no guarantee of victory!
The diagram below illustrates the problem:
The shift to first-price auctions
The use of 2PA at the SSP in header bidding created bid strategy problems for advertisers, who could not be assured that a high bid would result in an auction win. And for SSPs, it created a business problem: SSPs have a business model based on earning a commission, a “take rate,” on winning bids, so their inability to submit the highest received bid unaltered to the final auction hurt their win rates, and therefore their revenues4. Even when an SSP won an auction, their take rate was a function of the second-highest bid only, further depressing their earnings. As a countermeasure, SSPs often non-transparently adjusted the bid transmitted to the publisher such that it was no longer a true second price5. This boosted SSP revenues but damaged advertiser confidence in the integrity of the auction system.
Finally, while the move to header bidding did force SSPs to compete with each other head-to-head, it did not fully aggregate advertiser demand. There is still tremendous bid attenuation between advertisers and the publishers. As the simple model in Figure 3 above illustrates, four advertisers may bid into two SSPs, but the publisher will only see two bidders. This means that the publisher can’t conduct its own 2PA without suppressing revenues.
1PAs in the SSP were envisioned as a way to resolve all of these issues in one stroke. After all, if the two-stage auction is a 1PA in both stages, a bidder can be confident that she will win if she has the highest bid among all bidders, across all SSPs. Concordantly, SSP win rates should be identical to the percentage of the time a given SSP has the highest bid among its competitors. The business implications are positive: for a fixed set of bids, SSP revenues should increase. Advertiser confidence in the integrity of the auction system should be improved, too, as winning bidders’ clearing prices should be identical to the bids submitted by their DSP6.
Unfortunately, the move to 1PA is not a win for advertisers for two key reasons: First, costs will increase. Second, it is impossible to construct an optimal bid strategy. Both predictions are grounded in existing auction theory literature and supported by our testing. The remainder of this paper examines the theoretical basis for our claims, the results of our field experiments, and recommendations for advertisers.
The Limits of the Revenue Equivalence Theorem:
Why Moving to 1PA Directly Increases Advertiser Costs
William Vickrey is the father of modern auction theory, so much so that the sealed-bid second-price auction (introduced above) is commonly referred to as the “Vickrey auction” in his honor. Vickrey’s 1961 paper Counterspeculation, auctions, and competitive sealed tenders not only created the field of auction theory but also established a core concept called “revenue equivalence”. Vickrey used tools from game theory to demonstrate that if certain conditions were met, four standard auction types would generate equivalent revenues for sellers (and prices paid by auction winners):
- Second price single-bid sealed-bid auctions (Vickrey - classic 2PA)
- First price single-bid sealed-bid auctions (Blind - 1PA)
- First price ascending-bid open auctions (English)
- First price descending-bid open auctions (Dutch)
The Vickrey auction and Blind auction are commonly analogized to programmatic advertising’s 2PA and 1PA auctions, respectively. This is a mistake, because programmatic 2PAs deviate from the classic design in a way that violates the revenue equivalence requirements. Vickrey’s revenue equivalence theorem requires the following conditions to be met to apply to a given auction design:
- The auction mechanism must always allocate the item to the highest bidder
- Bidders must be risk-neutral
- Bidders have independent, uncorrelated values for the item
Programmatic second-price auctions, as shown in Fig. 1 above, do not satisfy the first criterion: the impression does not always go to the highest bidder. Sometimes, a lower-bidding advertiser can win. Therefore, programmatic 2PAs and 1PAs cannot be revenue-equivalent7.
We can take this a step further than merely saying the two forms are not revenue-equivalent. Since the programmatic 2PA sometimes allocates impressions to non-high bidders, it is a mathematical certainty that average clearing prices will be higher in a 1PA auction form8.
Dominant and dominated strategies in first- and second-price auctions:
Why it is impossible to construct a dominant bid strategy in programmatic 1PA
Constructing a dominant strategy in a second-price auction is easy: a classic 2PA is fully incentive-compatible. All a bidder must do is be aware of his private value and then bid it. The auction form takes care of the rest, ensuring that the high bidder captures the maximum possible buyer surplus9.
Sealed-bid first price auctions call for a more complex strategy that is rooted equally in probability and the bidder’s private valuation of an item. After all, if a bidder values a media vehicle at a $10 CPM, bids $10, and wins at $10, no surplus is captured. Bidders therefore need to construct a strategy that includes “bid shading” – the reduction of their private value to a bid that captures a surplus, while still beating competing bidders with lower private values. Put simply, the more competition you face, the smaller the gap between your private value and your optimal bid. The underlying math is presented in depth in a blog post at Mathalicious, but the takeaway is straightforward: against n opponents, the dominant strategy is to bid n/(n+1) of one’s private value. The visual below illustrates how this plays out in a case where a bidder’s private value is $5:
Unfortunately, this ideal approach to bid shading is technically impossible to execute in the world of programmatic advertising due to information deficiencies on the advertiser side. While advertisers do have access to certain metrics related to the auction like submitted bid values, clearing prices, and win rates, one thing they never see is the number of competing bidders. In fact, no single party in the programmatic supply chain sees all bids:
- An advertiser can see their own bids, but not those of other advertisers
- A DSP can track all bids it submits for a given impression, but cannot see those of other DSPs.
- An SSP can see all bids that it receives for a given bid request, but not those received by other SSPs.
- A publisher does not see all bids, only the subset that survive each SSP’s own 1PA.
Constructing a dominant 1PA bid strategy requires information that no one in programmatic advertising has: the number of competitors a given advertiser has in each auction. Without this knowledge, constructing a dominant bid strategy for 1PA is impossible10. Bidders will necessarily realize suboptimal outcomes compared to what they could achieve with a classic 2PA, which does not require knowledge of competitor bidder volume.
Auction theory predicts that the move from the programmatic 2PA to 1PAs will be negative for advertisers. Irrespective of any changes advertisers make to their bid strategy, it is a mathematical certainty that clearing prices will increase. Separately, the information-poor auction environment that bidders encounter makes it impossible to implement a dominant 1PA bid strategy.
We set out to see if the predictions of auction theory would be supported by field tests in the current programmatic auction environment.
Acknowledging that: in a two-stage auction, using a second price auction in the first stage can lead to undesirable losses due to an uncompetitive second price being passed to the final first-price auction
We hypothesize that these losses are nevertheless outweighed by the superior buyer surplus produced by the second price auction format.
Our test was designed to be as favorable to 1PAs as possible. Of our three cells, only one was 2PA. 1PAs were granted the benefit of a cell with an auto-bidder. We assumed that if our predictions about the effects of changing auction types were inaccurate, 1PAs using a DSP’s bidding algorithm would easily outperform 2PAs using a simple flat bid strategy.
To construct the test cells, we relied on SSPs’ own reports of their auction type, our DSP’s guidance, and confirming coverage in the advertising trade press. The 1PA test cells only utilized SSPs that have publicly switched to this auction format. For our 2PA test cell, all media ran through a single SSP that is widely known to only conduct 2PAs.
The test cells ran concurrently through a single DSP, all with a simple demographic target serving across the United States. All media was bought exclusively from authorized sellers as declared by publisher ads.txt files, so we know that none of the inventory was “spoofed”. Each test cell also had the same daily rate of spend in order to control for volume of demand as a confounding factor.
Conscious of the complexity here, we took steps to control for certain factors when conducting our analysis of performance:
- We restricted our analysis to the publishers where we had significant delivery across all three test cells
- Since different ad units (300x250, 728x90, etc.) vary in price we controlled for ad unit type
We found that our 2PA test cell with flat bids outperformed both 1PA cells. While the use of an auto-bidder reduced clearing prices relative to the 1PA cell with flat bids, these prices were still higher than those returned by the 2PA auction cell.
There was no significant variation in either viewability rates or in-demo delivery accuracy across our test cells, which tells us that we were serving in inventory of a consistent quality and reaching the same types of users. That is important, because it contradicts any claims that 1PAs provide access to “more premium” inventory.
We further saw that these patterns were repeated publisher by publisher, even when we controlled for variations in ad unit composition:
In 14 out of 15 publishers we analyzed (these were the publishers with significant delivery across all three test cells), we saw that our 2PA cell with flat bids out-performed the 1PA cell with the DSP auto-bidder. It is likely that if we had run a fourth cell, using 2PA combined with the DSP auto-bidder, the relative performance of 2PA would have been better still11.
Avenues for future research
It is possible that improvements in DSP bidding algorithms could shrink the gap between 2PA and 1PA clearing prices. If the difference were to become small enough (<2%) arguments about auction mechanism design would be relegated to the realm of the academic. We will continue evaluating DSP bid algorithms as they improve.
SSPs are also getting into the bid optimization game: certain SSPs offer a bid-shading algorithm at no additional cost, which our preliminary testing shows reduces 1PA clearing prices below the levels described above. 2PAs are still cheaper than 1PAs when we use SSP bid shading, but the approach bears further exploration.
Recommendation for advertisers
Theory and our experimental data both point in the same direction: 1PAs are broadly bad for advertisers. Whenever the same inventory is accessible via both 1PA and 2PA types, advertisers should generally prioritize 2PA. To be clear, our recommendation is notto avoid 1PAs at all costs and in all circumstances, because there is trade-off to the two-stage 2PA auction form: it delivers lower prices, but the highest bidder will sometimes lose the auction. As a result, advertisers should calibrate their inventory strategy to the needs of their campaigns.
The guidance to prioritize 2PAs is most applicable to large-scale, broadly targeted buys that aim to acquire only a small fraction of the relevant inventory. In circumstances where winning as many auctions as possible is essential, advertisers should be more open to bidding in 1PAs as long as they have bid-shading active in their DSP. For example, a local-market campaign aiming to saturate a target audience, an ecommerce retargeting campaign, or a campaign targeting a small CRM list of potential customers would all be ill-served by eschewing 1PAs entirely.
Private marketplace deals (PMPs) offer a way out of the programmatic auction quagmire, with some limitations. It is possible for advertisers or their agencies to negotiate fixed-price PMP deals, wherein the buying party is guaranteed a stable clearing price through time. The trade-off is that each deal must be negotiated and monitored individually: the onus is on the buying party to ensure the price is fair and the deal is liquid. After all, a deal that is more expensive than open market or unable to deliver any volume is no deal at all. Ultimately the utility of these PMP arrangements is limited both by an advertiser’s negotiation and administrative skills, as well as the willingness of publishers to make favorable terms.
Most advertisers will need to participate in some quantity of 1PAs, so they should undertake experiments such as those described in this paper to determine what combination of DSP and SSP bid shading methods yields the best results for their campaigns. Advertisers should also direct investment to prudently negotiated and administered fixed-price PMP deals. For most advertisers, it would be advantageous to retain an agency such as Hearts and Science to design, implement, and evaluate the results of such a bid strategy testing regime, as well as to negotiate and administer PMP deals.
Proposal for the future development of the auction system
The adoption of header bidding and 1PA has come with benefits, but left us with a profoundly flawed auction system. From a bid strategy perspective, programmatic auctions are advertiser-hostile when compared with their search and social counterparts. This article does not suggest the current programmatic 2PA system is the answer. On the contrary, the specific claim is merely that for advertisers, the existing programmatic 2PA is generally superior to the 1PA system presently being promulgated by various interested parties.
Going forward, we should work as an industry to migrate to a single-stage 2PA. Facebook and Google figured this out years ago and constructed their auction systems accordingly. In Aggregating all advertiser bids into a single second-price auction is the optimal design for online ad sales because such a system:
1. Always allocates a given impression to the highest bidder
2. Allows a simple dominant bid strategy
3. Only requires a single auction (low latency)
4. Maximizes publisher yield because
a. All bids are surfaced to the publisher – there is no bid attenuation
b. Revenues rise as more bidders join an auction12
For this move to be publisher-friendly, it is critical that all advertiser bids are being transmitted to the publisher before the change to a true 2PA is implemented. Moving to 2PA without making this change first would result in depressed publisher yields due to the current environment of artificial bid sparseness.
Restructuring the auction system will be a significant undertaking. There are also multiple technical means of achieving the goal of a unified single-stage 2PA that captures all advertiser bids. A future paper will explore the mechanics of how this could be achieved from a technical perspective.
Even if we bring that structural change about, however, mirroring auction design alone isn’t enough: the programmatic ad auction must also become as trusted and predictable as those conducted by Google and Facebook. To establish the credibility of this new system and escape the long shadow of past misbehavior by various stakeholders, publishers should submit to an auction audit conducted by a disinterested third party such as the MRC.
Auctions are the beating heart of modern digital advertising. Every day, billions of impressions costing hundreds of millions of dollars13 are auctioned, sold, and delivered across the United States. Auction design therefore has commensurate economic importance to all digital advertising stakeholders. Somehow, though, auctions have managed to avoid the close scrutiny – and accompanying calls for standardization and transparency – engendered by topics like brand safety and viewability.
That needs to change. Right now, advertiser resources - which fund the entire ecosystem- are being squandered on the needless opacity and complexity of the current programmatic auction system. While advertisers can and should take measures like bid shading to cope with the present fragmentation, it is in their long-term interest to demand a transparent, independently verified auction system that supports straightforward bid strategies. Hearts and Science looks forward to working with its clients and partners to make this a reality.
Elijah Halpern, Sam Hochman, and Ryan Verklin on the Hearts Digital Activation team were invaluable in designing, running, and analyzing our test of different auction types. Without their contributions this article’s arguments would lack empirical heft.
Chris Principe, Sr. Director of Digital Activation at Hearts, patiently educated me on the evolution of waterfalled SSPs to header bidding. Any coherence in that section of this article is solely attributable to him. Chris was also indispensable to the review and integration of feedback on the first draft.
Greg Dreifus and Mark Oster at Resolution Performance Media provided extensive feedback on this paper, as well as further insight on the technical underpinnings of the auction system. Our conversation about the trade-offs between the different auction systems shaped the “recommendation for advertisers” section.
Megan Pagliuca, Chief Data Officer at Hearts, offered steady support, encouragement, and patience.
Michael Griffiths lent this project his inimitable critical eye, spotting awkward phrases and tracking down intellectual forerunners.
Julie Van Ullen wrote the first article I am aware of recommending that programmatic advertising move to an entirely new auction system, with all advertiser bids flowing into a classic second-price publisher auction. This paper’s recommendations on reforms of the auction system are materially similar.
- Another way of putting this is that a risk-neutral bidder should adopt a profit-maximizing bid strategy. Risk-neutral means that a bidder values a 100% probability of a $0.50 profit the same as a 50% probability of a $1 profit. The profit in this case refers to the buyer surplus. The concept is fully applicable to agents working on an advertiser’s behalf, as even in a fully transparent model the agent should be attempting to maximize the principal’s buyer surplus.
- Paul Klemperer’s “Auction Theory: A Guide to the Literature” explains this in greater detail on pp. 6-7
- Digiday’s “WTF is header bidding?” has a detailed explanation
- SSPs’ costs of conducting an internal auction and submitting to the publisher are fixed but their revenues vary with their win rate and their average bid. Therefore, all else being equal, SSP profitability will decline in tandem with win rates. In a first price model, SSPs gain greater ability to predict whether their highest bid will result in an auction win. SSPs are therefore able to constrain their costs by opting not to participate in auctions they are unlikely to win.
- This is an open secret in the advertising industry. Ari Paparo’s article “Waterfalling – a New Hope” contains a great summary.
- Another benefit of 1PAs for advertisers is that their bids become more competitive with direct-sold buys. If their bid values were being reduced by a 2PA they might lose an auction to another advertiser with a direct-sold campaign at a fixed CPM
- Paul Klemperer’s “Auction Theory: A Guide to the Literature” contains a detailed proof of the Revenue Equivalence Theorem, pp. 35-39
- 1PAs always allocate inventory to the highest bidder. Programmatic 2PAs sometimes do and sometimes don not. That means that even if bidders in both auction forms were pursuing an optimal bid strategy, 1PAs would average higher clearing prices.
- This assumes that the other bidders have read enough auction theory to understand that bidding their true value is the dominant strategy in classic 2PA. If competing bidders overbid they will erode the winner’s buyer surplus, however this is true of any auction form.
- Erik Ekstrom explores a related scenario in his paper “Sealed-bid first-price auctions with an unknown number of bidders”. Ekstrom makes the stylized assumption that all bidders have a common valuation of the item being auctioned; this is not descriptive of dynamics in an ad auction since advertisers place different values on a given user impression. Even granting this assumption, however, Ekstrom finds no optimal bid strategy in a 1PA environment where the number of competitor bidders is unknown.
- Astute observers will note that our clearing prices in the 1PA flat bid cell fluctuate across publishers, which shouldn’t be the case if we were bidding a consistent value into a true 1PA. This is because even though we took all available measures to participate in 1PAs exclusively, some 2PAs slipped into this test cell. Ironically this reinforces this paper’s recommendation for audit-validated auction transparency.
- Stated more formally: Assume that private values follow a normal distribution around a stable median and that truthful bidding is the prevailing strategy in a 2PA. Clearing prices in a 2PA will rise as the number of participating bidders increases.
- Statista page on US digital ad spending