Building a Production-Grade Ad Tech Stack — RTB, Ad Exchange, DSP/SSP, Targeting, Auctions & Attribution at Scale
1. Introduction & Why Ad Serving is Hard
Digital advertising is the economic backbone of the open internet. Global digital ad spending exceeded billion in 2025, and every single dollar flows through a complex ecosystem of ad servers, exchanges, demand-side platforms (DSPs), supply-side platforms (SSPs), and data management platforms (DMPs). When a user loads a web page or opens a mobile app, an invisible auction happens in under 100 milliseconds, deciding which advertiser gets to show an impression, how much they pay, and what creative gets rendered.
Designing an ad serving platform is one of the hardest problems in distributed systems engineering. It combines ultra-low-latency decisioning (the entire bid-response pipeline must complete in under 100ms), massive scale (billions of impressions per day), complex real-time bidding protocols (OpenRTB), intricate auction theory, financial accounting, privacy compliance (GDPR, CCPA), brand safety, fraud detection, and multi-sided marketplace dynamics. A single system must serve publishers who want to maximize revenue, advertisers who want to maximize ROI, and users who should see relevant, non-intrusive ads.
The Ad Tech Ecosystem
Before diving into architecture, it is essential to understand the key players:
- Publishers — Websites and apps that have ad inventory (e.g., NYTimes, YouTube, CNN).
- Advertisers / Brands — Companies that want to show ads (e.g., Nike, Coca-Cola, Toyota).
- Demand-Side Platforms (DSPs) — Platforms that let advertisers buy ad inventory programmatically (e.g., Google DV360, The Trade Desk, Amazon DSP).
- Supply-Side Platforms (SSPs) — Platforms that let publishers sell their inventory programmatically (e.g., Google Ad Manager, Magnite, PubMatic).
- Ad Exchanges — Marketplaces that connect DSPs and SSPs, facilitating real-time auctions (e.g., Google AdX, OpenX).
- Ad Servers — Systems that store, deliver, and track ads (e.g., Google Campaign Manager, Sizmek).
- Data Management Platforms (DMPs) — Systems that collect, organize, and activate audience data for targeting.
- Customer Data Platforms (CDPs) — Unified customer databases that combine first-party data across channels.
- Verification Providers — Third parties that verify viewability, brand safety, and fraud (e.g., IAS, DoubleVerify, MOAT).
In this deep dive, we will design an end-to-end ad serving platform that covers the complete lifecycle: from the moment a publisher's page loads and sends an ad request, through the real-time auction, bid evaluation, creative serving, impression tracking, conversion attribution, and reporting. We will build this from first principles, addressing each subsystem in detail.
Historical Context
The ad tech industry evolved from simple tag-based ad serving in the early 2000s (where publishers would hard-code ad tags into their pages) to the sophisticated programmatic ecosystem we have today. The introduction of real-time bidding (RTB) around 2009-2010 was a paradigm shift: instead of negotiating deals directly between advertisers and publishers, inventory could be auctioned in real-time on a per-impression basis. This created the need for sub-100ms decisioning pipelines, probabilistic data structures for frequency capping, and massive-scale logging infrastructure.
Today, the ecosystem faces new challenges: the deprecation of third-party cookies, increasing privacy regulations, the rise of walled gardens (Google, Meta, Amazon), header bidding, programmatic guaranteed deals, and the constant battle against ad fraud. A modern ad serving platform must navigate all of these while maintaining sub-100ms latency and financial accuracy.
2. Functional & Non-Functional Requirements
Functional Requirements
- Ad Serving — Serve the right ad to the right user at the right time, based on targeting criteria.
- Real-Time Bidding — Accept bids from multiple DSPs within a tight latency budget (under 100ms).
- Auction Engine — Run first-price and second-price auctions with proper winner determination.
- Targeting — Support demographic, behavioral, contextual, geographic, and first-party audience targeting.
- Frequency Capping — Limit how many times a user sees a particular ad within a given time window.
- Creative Management — Upload, store, validate, and serve ad creatives in various formats (banner, video, native, interstitial).
- Impression & Click Tracking — Track impressions via pixels and clicks via redirect URLs with deduplication.
- Conversion Tracking — Track post-click and post-view conversions via pixel firing on advertiser confirmation pages.
- Attribution — Support last-click, first-click, linear, time-decay, and position-based attribution models.
- Budget Management — Enforce daily and lifetime budgets with pacing algorithms to spend evenly.
- Brand Safety — Block ads from appearing on inappropriate content using keyword lists, category blocking, and third-party verification.
- Fraud Detection — Detect and filter invalid traffic (IVT) including bot traffic, click farms, pixel stuffing, and ad stacking.
- Viewability Measurement — Measure whether an ad was actually viewable using MRC standards.
- Reporting & Analytics — Provide real-time and batch reporting on impressions, clicks, conversions, spend, and performance metrics.
- A/B Testing — Allow split testing of ad creatives with statistical significance measurement.
- Privacy Compliance — Support GDPR consent management, CCPA opt-out, cookie deprecation, and Google Topics API.
Non-Functional Requirements
| Attribute | Target | Rationale |
|---|---|---|
| Latency (Ad Response) | < 100ms p99 | User experience degrades; publishers switch to faster ad servers |
| Throughput | 500,000+ QPS | Peak traffic during prime-time events can spike 10x normal |
| Availability | 99.99% | Every millisecond of downtime costs millions in lost ad revenue |
| Data Durability | 99.999999% | Every impression must be logged for billing reconciliation |
| Consistency | Eventual (reporting), Strong (billing) | Reporting can lag; financial data cannot be lost or duplicated |
| Scalability | 10B+ impressions/day | Global ad serving at internet scale |
| Privacy | GDPR, CCPA compliant | Regulatory fines can reach 4% of global revenue |
3. Capacity Estimation & SLA Targets
Traffic Estimates
| Metric | Estimate | Calculation |
|---|---|---|
| Daily impressions | 10 billion | 1M publishers x avg 10K pages/day x avg 1 ad slot/page |
| Peak QPS | 500K | 10B / 86400 x 4x peak multiplier |
| Average QPS | 116K | 10B / 86400 |
| Bid requests/day | 10 billion | One per impression |
| Bid responses/day | 30 billion | Avg 3 bids per request (10 DSPs contacted, ~30% respond) |
| Click events/day | 100 million | ~1% CTR |
| Conversion events/day | 1 million | ~1% conversion rate from clicks |
| Data volume (impression logs) | ~5 TB/day | ~500 bytes per impression log x 10B |
| Data volume (click logs) | ~50 GB/day | ~500 bytes per click x 100M |
| Creative assets storage | ~10 TB total | 1M creatives x avg 10KB per asset |
Storage Estimates (5-Year Retention)
| Data Type | Daily Size | 5-Year Total | Storage Tier |
|---|---|---|---|
| Impression logs | 5 TB | ~9 PB | Hot: 30 days / Warm: 1 year / Cold: S3 Glacier |
| Click logs | 50 GB | ~90 TB | Hot: 90 days / Warm: 1 year / Cold: 5 years |
| Conversion logs | 1 GB | ~2 TB | Hot: 1 year / Warm: 5 years |
| Auction logs | 2 TB | ~3.6 PB | Hot: 7 days / Warm: 90 days / Cold: 1 year |
| User profiles (DMP) | 50 GB updates | ~2 TB delta | Hot: always (Redis) + persistent store |
| Creative assets | 1 GB delta | ~10 TB | CDN + object storage |
Bandwidth Estimates
At peak QPS of 500K, with an average bid request size of 2KB and bid response of 5KB, the ad server must handle:
- Inbound (bid requests): 500K x 2KB = 1 GB/s
- Outbound (to DSPs): 500K x 10 DSPs x 2KB = 10 GB/s
- Inbound (bid responses): 500K x 10 DSPs x 30% response rate x 5KB = 7.5 GB/s
- Impression pixel fires: 116K QPS x 1KB = 116 MB/s
4. High-Level Architecture Overview
Core Components
The architecture is organized into five major subsystems:
- Publisher Integration Layer — Handles ad tag delivery, header bidding wrappers, and server-side ad insertion (SSAI). This is the entry point for all ad requests and must be extremely fast and resilient.
- Ad Selection & Auction Core — The brain of the system. Receives ad requests, evaluates targeting criteria, runs frequency capping checks, contacts DSPs for bids, runs the auction, applies brand safety filters, enforces budget pacing, and selects the winning creative.
- Exchange & Demand Layer — Manages connections to DSPs via OpenRTB, handles header bidding adapters, and manages deal IDs for private marketplace (PMP) and programmatic guaranteed (PG) deals.
- Tracking & Attribution Layer — Fires impression pixels, handles click redirects, tracks conversions, and runs attribution models. This layer generates the financial data that drives billing.
- Data & Analytics Layer — Ingests event streams from Kafka, stores them in ClickHouse for real-time analytics, and feeds the reporting dashboard. Also powers the DMP for audience segmentation.
Technology Stack
| Component | Technology | Rationale |
|---|---|---|
| Ad Server | C# / .NET 8, ASP.NET Core | High-performance async I/O, strong typing for financial calculations |
| Load Balancer | Envoy / NGINX + GeoDNS | L7 load balancing with latency-aware routing |
| Cache | Redis Cluster (6-node) | Sub-millisecond reads for targeting data, frequency caps, budgets |
| Message Queue | Apache Kafka | Durability for impression/click event streams, exactly-once semantics |
| OLAP Database | ClickHouse | Columnar storage for fast aggregations on billions of rows |
| Object Storage | AWS S3 / CloudFront CDN | Creative asset storage with global CDN distribution |
| Service Mesh | Kubernetes + Istio | Auto-scaling, circuit breaking, mTLS between services |
| Monitoring | Prometheus + Grafana + PagerDuty | Industry standard for metrics, dashboards, alerting |
5. Ad Request Flow & OpenRTB Protocol
The ad request flow is the most latency-sensitive path in the entire system. When a user loads a publisher's page, a sequence of events must complete in under 100 milliseconds to return a valid ad response. Understanding the OpenRTB protocol is essential because it is the lingua franca of programmatic advertising — every major DSP and SSP communicates using this standard.
OpenRTB Bid Request Structure
The OpenRTB 2.6 specification defines a JSON-based protocol for communication between exchanges and DSPs. Here is a simplified bid request:
JSON
{
"id": "bid-request-uuid-12345",
"at": 1,
"imp": [
{
"id": "imp-001",
"banner": {
"w": 300,
"h": 250,
"fmt": [{"w": 300, "h": 250}, {"w": 728, "h": 90}],
"pos": 1
},
"bidfloor": 0.50,
"bidfloorcur": "USD",
"tagid": "slot-homepage-top",
"secure": 1,
"pmp": {
"private_auction": 0,
"deals": [
{
"id": "deal-premium-001",
"bidfloor": 2.00,
"bidfloorcur": "USD",
"wseat": ["dsp-premium-1"]
}
]
}
}
],
"site": {
"id": "site-nytimes-001",
"domain": "nytimes.com",
"page": "https://www.nytimes.com/article/breaking-news",
"cat": ["IAB12", "IAB12-1"],
"content": {
"title": "Breaking: Major Economic Report Released",
"lang": "en",
"sitetitle": "The New York Times"
}
},
"device": {
"ua": "Mozilla/5.0 ...",
"ip": "198.51.100.42",
"devicetype": 2,
"os": "Windows",
"osv": "11",
"make": "Dell",
"model": "XPS",
"language": "en",
"geo": {
"country": "USA",
"region": "NY",
"city": "New York",
"lat": 40.7128,
"lon": -74.0060,
"zip": "10001"
}
},
"user": {
"id": "user-encrypted-id-abc",
"buyeruid": "dsp-buyer-id-xyz",
"data": [
{
"name": "dmp-provider",
"segment": [
{"id": "segment-auto-intenders"},
{"id": "segment-luxury-shoppers"}
]
}
]
},
"regs": {
"gdpr": 1,
"ext": {"us_privacy": "1YNN"}
},
"ext": {
"prebid": {
"bidder": {
"adapter-version": "2.0"
}
}
}
}OpenRTB Bid Response Structure
A DSP responds with one or more bid objects. The exchange evaluates all responses and determines the winner:
JSON
{
"id": "bid-request-uuid-12345",
"seatbid": [
{
"bid": [
{
"id": "bid-001",
"impid": "imp-001",
"price": 2.50,
"adid": "ad-campaign-nike-001",
"adm": "<script src='https://cdn.adserver.com/nike-banner.js'></script>",
"adomain": ["nike.com"],
"cat": ["IAB3"],
"crid": "creative-nike-airmax-001",
"w": 300,
"h": 250,
"ext": {
"dealid": "deal-premium-001",
"advertiser_domain": "nike.com"
}
}
],
"seat": "dsp-premium-1"
}
],
"cur": "USD",
"bidid": "bid-result-uuid-67890"
}Server-Side vs Client-Side Ad Request
Modern ad platforms support two primary ad request mechanisms:
- Client-Side (JavaScript Tag): The publisher's page loads a JavaScript tag that makes ad requests directly from the browser. This provides access to cookie data and browser signals but is vulnerable to ad blockers and has higher latency due to client-side rendering.
- Server-Side (API-based): The publisher's server makes the ad request to the ad server on behalf of the user. This is faster (no client-side round trip), more reliable (not blocked by ad blockers), and provides better privacy control. Google's Secure Ad Serving and OpenRTB server-to-server (S2S) are examples.
- Hybrid (Header Bidding + S2S): Most modern publishers use header bidding (prebid.js) for the initial auction pass, falling back to server-side for latency-sensitive scenarios. This hybrid approach maximizes yield while maintaining acceptable page load times.
6. Real-Time Bidding (RTB) in Under 100ms
Real-time bidding is the core innovation that transformed ad tech from a manual, relationship-driven business into a programmatic, data-driven marketplace. RTB enables advertisers to bid on individual impressions in real-time, paying only for the impressions they actually win. The entire bid request, auction, and response cycle must complete within 100 milliseconds — or the impression is lost.
RTB Timing Budget Breakdown
| Stage | Budget (ms) | Description |
|---|---|---|
| Request parsing & validation | 2-3ms | Parse JSON, validate schema, extract user/device/site|
| User profile lookup | 3-5ms | Redis lookup for targeting data, audience segments, frequency caps|
| Targeting evaluation | 2-3ms | Match campaign targeting criteria against user/device/site attributes|
| Bid request construction | 2-3ms | Build OpenRTB bid request, sign with ad request ID|
| Parallel DSP callout | 40-60ms | Send bid requests to multiple DSPs, collect responses with timeout|
| Auction execution | 2-5ms | Run auction algorithm, determine winner, calculate price|
| Budget & pacing check | 2-3ms | Verify advertiser has budget remaining, check pacing|
| Brand safety & fraud check | 2-3ms | Verify ad is safe for the page context, check for fraud signals|
| Response assembly | 2-3ms | Build ad response with tracking pixels, win notification URL|
| Total | 60-88ms | Must complete within 100ms p99 |
Parallel Callout Architecture
The DSP callout is by far the largest contributor to total latency. The ad server sends bid requests to multiple DSPs in parallel and collects responses within a configurable timeout window (typically 60-80ms). DSPs that fail to respond within this window are automatically excluded.
C#
public class ParallelBidCollector
{
private readonly HttpClient _httpClient;
private readonly IDspRegistry _dspRegistry;
private readonly TimeSpan _defaultTimeout = TimeSpan.FromMilliseconds(65);
public async Task<IReadOnlyList<BidResponse>> CollectBidsAsync(
BidRequest bidRequest, CancellationToken ct)
{
var eligibleDsps = _dspRegistry
.GetEligibleDspIds(bidRequest.Impressions.First().TagId);
var bidTasks = eligibleDsps.Select(dspId =>
SendBidRequestWithTimeout(bidRequest, dspId, _defaultTimeout, ct));
var responses = await Task.WhenAll(bidTasks);
return responses
.Where(r => r != null && r.HasValidBid)
.ToList();
}
private async Task<BidResponse?> SendBidRequestWithTimeout(
BidRequest request, string dspId, TimeSpan timeout, CancellationToken ct)
{
using var timeoutCts = CancellationTokenSource
.CreateLinkedTokenSource(ct);
timeoutCts.CancelAfter(timeout);
try
{
var dspEndpoint = _dspRegistry.GetEndpoint(dspId);
var content = new StringContent(
SerializeBidRequest(request),
Encoding.UTF8,
"application/json");
var response = await _httpClient
.PostAsync(dspEndpoint, content, timeoutCts.Token);
response.EnsureSuccessStatusCode();
var body = await response.Content.ReadAsStringAsync();
return DeserializeBidResponse(body);
}
catch (OperationCanceledException)
{
return null; // Timeout — skip this DSP
}
}
}Server-Side RTB vs Client-Side Header Bidding
Header bidding moved the RTB auction from the ad server to the browser, giving publishers direct access to DSP demand. However, client-side header bidding adds 500-2000ms to page load time because the browser must execute JavaScript, collect bids, and then pass them to the ad server. Server-side header bidding (S2S) moves this process server-to-server, reducing latency to under 100ms but losing access to browser-level user signals (cookies, fingerprinting).
The industry is converging on a hybrid approach: Google's Open Bidding (formerly Exchange Bidding) runs the auction server-side but within Google's infrastructure, providing access to user signals through their cookie graph. Prebid Server provides an open-source alternative for server-side header bidding.
7. DSP & SSP Architecture
The DSP (Demand-Side Platform) and SSP (Supply-Side Platform) are the two sides of the programmatic advertising marketplace. While they serve different stakeholders — advertisers and publishers respectively — they share many architectural patterns and face similar challenges around latency, scale, and optimization.
DSP Architecture Deep Dive
A DSP is essentially a real-time optimization engine that evaluates millions of bid opportunities per second and decides which ones to bid on, at what price, and with which creative. The key subsystems of a DSP include:
The DSP bid optimization pipeline works as follows: When a bid request arrives, the DSP first checks whether the impression is eligible for any active campaigns (audience matching). It then evaluates the expected value of bidding on this impression using a bid optimization model that considers the user's predicted conversion probability, the campaign's target CPA/ROAS, the competitive landscape, and the inventory quality. The bid price is calculated using a formula like: bid = pConversion x targetCPA x qualityMultiplier.
SSP Architecture Deep Dive
The SSP's job is to maximize publisher revenue by connecting them to as many demand sources as possible and running efficient auctions. Key SSP components include:
C#
public class SspYieldOptimizer
{
private readonly IAdExchangeClient _exchangeClient;
private readonly IPublisherProfileStore _publisherStore;
private readonly IBidResponseAggregator _aggregator;
public async Task<AuctionResult> RunYieldOptimizationAsync(
AdRequest request, CancellationToken ct)
{
// Step 1: Determine floor prices per impression opportunity
var floorPrices = await _publisherStore
.GetFloorPricesAsync(request.PublisherId, request.SlotId);
// Step 2: Apply dynamic floor pricing based on historical data
var dynamicFloors = _floorPricer.CalculateDynamicFloors(
request.PublisherId,
request.SlotId,
floorPrices.MinFloor,
request.UserSegments);
// Step 3: Send bid requests to all connected exchanges with floors
var allBids = await _exchangeClient
.SendBidRequestsToExchangesAsync(request, dynamicFloors, ct);
// Step 4: Filter invalid bids (below floor, non-compliant, fraud)
var validBids = allBids
.Where(b => b.Price >= dynamicFloors.MinFloor)
.Where(b => _complianceChecker.IsCompliant(b))
.Where(b => !_fraudDetector.IsSuspicious(b))
.ToList();
// Step 5: Run auction (header bidding waterfall + unified auction)
var auctionResult = _aggregator.RunAuction(
validBids,
request.AuctionType); // FirstPrice or SecondPrice
return auctionResult;
}
}Dynamic Floor Pricing
Floor prices are the minimum bid price a publisher will accept for an impression. Setting floors too high reduces fill rate and total revenue; setting them too low leaves money on the table. Dynamic floor pricing uses machine learning to optimize floor prices based on historical auction data, user value, time of day, and demand patterns. The algorithm typically works by predicting the probability of receiving a bid at each price point and finding the price that maximizes expected revenue: expectedRevenue = pBidAboveFloor x bidPrice.
8. Ad Exchange Design
The ad exchange is the marketplace that connects supply (SSPs/publishers) with demand (DSPs/advertisers). It facilitates real-time auctions, manages deal inventory, handles billing reconciliation, and provides market transparency. Designing an ad exchange is essentially designing a real-time auction system with financial accuracy guarantees.
Exchange Core Responsibilities
- Auction Management — Collect bids from multiple DSPs, run the auction algorithm, determine the winner, and notify all participants.
- Deal Management — Handle private marketplace (PMP) deals, programmatic guaranteed (PG) deals, and preferred deals between specific buyers and sellers.
- Billing & Reconciliation — Track won impressions, calculate charges, generate invoices, and reconcile with DSPs and publishers.
- Traffic Quality — Validate bid requests, filter invalid traffic, enforce IAB standards, and ensure brand safety.
- Reporting — Provide real-time and historical reporting on auction activity, fill rates, CPM trends, and revenue metrics.
Auction Flow in the Exchange
Publisher paid .81 minus exchange fee
Exchange Fee Structure
| Deal Type | Typical Exchange Fee | SSP Fee | Total Take Rate |
|---|---|---|---|
| Open Auction (RTB) | 5-10% | 10-20% | 15-30% |
| Private Marketplace (PMP) | 5-10% | 5-15% | 10-25% |
| Programmatic Guaranteed | 3-7% | 5-10% | 8-17% |
| Preferred Deal | 0-5% | 5-10% | 5-15% |
The exchange fee (also called the exchange take rate) is deducted from the winning bid price before the publisher is paid. For example, if a DSP bids .50 and wins, the exchange might take 8% (.20), the SSP takes 12% (.30), and the publisher receives .00. This fee structure is critical to the exchange's business model and must be accurately tracked and reconciled.
9. Targeting Engine (Demographic, Behavioral, Contextual, Geo)
The targeting engine is what makes advertising relevant. Without targeting, every user would see the same generic ads, leading to low engagement, poor advertiser ROI, and a terrible user experience. The targeting engine evaluates every incoming ad request against the targeting criteria of all eligible campaigns and returns a filtered list of campaigns that match.
Targeting Dimensions
| Dimension | Data Source | Examples | Latency Impact |
|---|---|---|---|
| Demographic | DMP / registration data | Age: 25-34, Gender: Female, Income: + | Low (pre-computed segments) |
| Behavioral | Browsing history, search queries | Auto intenders, travel planners, fashion enthusiasts | Medium (segment lookup required) |
| Contextual | Page content analysis | Technology articles, sports content, recipe pages | Low (pre-tagged by NLP) |
| Geographic | IP geolocation, GPS (mobile) | Country: US, City: NYC, Zip: 10001, Radius: 5mi | Low (IP lookup is fast) |
| Device | User agent, device fingerprint | iOS 17, Chrome on Windows, mobile vs desktop | Very Low (in request) |
| Temporal | Time of day, day of week | Weekday mornings, weekend evenings, holiday season | Very Low (from timestamp) |
| Retargeting | First-party cookies, CRM data | Visited product page, abandoned cart, past purchaser | Medium (cookie/ID lookup) |
| Lookalike/Similar | ML model on seed audience | Similar to top 10% of converters | Medium-High (vector similarity) |
Targeting Evaluation Engine
The targeting engine evaluates campaigns using a conjunction of targeting criteria. Each campaign defines its targeting as a set of AND conditions across dimensions (e.g., age 25-34 AND male AND interested in cars AND in New York AND on mobile). The engine must evaluate these conditions against the bid request attributes and return matching campaigns.
C#
public class TargetingEngine
{
private readonly ICampaignStore _campaignStore;
private readonly IAudienceSegmentLookup _audienceLookup;
private readonly IContextualClassifier _contextClassifier;
public IReadOnlyList<TargetedCampaign> EvaluateTargeting(
AdRequest request, DateTime timestamp)
{
var campaigns = _campaignStore.GetActiveCampaigns();
var matched = new List<TargetedCampaign>();
foreach (var campaign in campaigns)
{
if (EvaluateCampaign(campaign, request, timestamp))
{
matched.Add(new TargetedCampaign
{
Campaign = campaign,
MatchedSegments = GetMatchedSegments(campaign, request),
Priority = campaign.Priority,
Weight = campaign.TargetingWeight
});
}
}
return matched.OrderByDescending(m => m.Priority).ToList();
}
private bool EvaluateCampaign(
Campaign campaign, AdRequest request, DateTime timestamp)
{
var targeting = campaign.TargetingCriteria;
// Demographic targeting
if (!targeting.AgeRanges.Any(r => r.Contains(request.User.Age)))
return false;
if (targeting.Gender != null && targeting.Gender != request.User.Gender)
return false;
// Geographic targeting
if (!targeting.Countries.Contains(request.Device.Geo.Country))
return false;
if (targeting.Regions?.Any() == true &&
!targeting.Regions.Contains(request.Device.Geo.Region))
return false;
if (targeting.Cities?.Any() == true &&
!targeting.Cities.Contains(request.Device.Geo.City))
return false;
if (targeting.RadiusMiles != null)
{
var distance = GeoUtils.HaversineDistance(
targeting.CenterLat, targeting.CenterLon,
request.Device.Geo.Lat, request.Device.Geo.Lon);
if (distance > targeting.RadiusMiles)
return false;
}
// Behavioral targeting (audience segments)
if (targeting.RequiredSegments?.Any() == true)
{
var userSegments = _audienceLookup
.GetUserSegments(request.User.Id);
if (!targeting.RequiredSegments
.All(rs => userSegments.Contains(rs)))
return false;
}
// Contextual targeting
if (targeting.ContextualCategories?.Any() == true)
{
var pageCategories = _contextClassifier
.Classify(request.Site.Page, request.Site.Content);
if (!targeting.ContextualCategories
.Any(tc => pageCategories.Contains(tc)))
return false;
}
// Device targeting
if (targeting.DeviceTypes?.Any() == true &&
!targeting.DeviceTypes.Contains(request.Device.Type))
return false;
// Operating system targeting
if (targeting.OperatingSystems?.Any() == true &&
!targeting.OperatingSystems.Contains(request.Device.OS))
return false;
// Temporal targeting
if (targeting.DayParting?.Any() == true)
{
var hour = timestamp.Hour;
if (!targeting.DayParting.Any(dp =>
dp.StartHour <= hour && hour <= dp.EndHour))
return false;
}
return true;
}
}Contextual Targeting with NLP
Contextual targeting analyzes the content of the page where the ad will appear and matches it against the advertiser's desired context. Modern contextual targeting uses NLP models to classify pages into IAB Content Taxonomy categories (e.g., IAB19-18 for "Tech News", IAB17-1 for "Sports News"). This is increasingly important as third-party cookie deprecation makes behavioral targeting less reliable.
10. Auction Mechanics (Second-Price, First-Price, Header Bidding)
Auction theory is the mathematical backbone of ad serving. The auction mechanism determines how much the winner pays, which directly impacts publisher revenue, advertiser ROI, and the overall health of the marketplace. The choice of auction type has profound strategic implications for all participants.
Auction Types Compared
| Auction Type | Winner Pays | Strategic Behavior | Prevalence (2025) |
|---|---|---|---|
| Second-Price (GSP) | Second-highest bid + .01 | Bid true value (truthful in theory) | ~20% (declining) |
| First-Price | Exact bid amount | Bid shading, bid below true value | ~75% (dominant) |
| Fixed-Price (Guaranteed) | Negotiated CPM | No bidding — direct deal | ~5% (PG deals) |
Second-Price Auction (GSP)
In a second-price auction (also called Generalized Second-Price or GSP), the winner pays one cent more than the second-highest bid. The theoretical property of this mechanism is that it is incentive-compatible for the winner — meaning the optimal strategy is to bid your true value. If you value an impression at .00 and the second-highest bid is .00, you win and pay .01. Bidding higher or lower than your true value does not improve your outcome.
In practice, second-price auctions were the standard in ad tech until 2019. The problem was that many platforms were adding hidden fees and adjustments on top of the second-price formula, which undermined the truthfulness guarantee and led to a major industry shift toward first-price auctions.
First-Price Auction & Bid Shading
In a first-price auction, the winner pays exactly what they bid. This is simpler to understand but introduces a strategic problem: if an advertiser values an impression at .00 and bids .00, they overpay when the second-highest bid is only .00. Rational advertisers respond by bid shading — deliberately bidding below their true value — which creates a complex game-theoretic optimization problem.
C#
public class BidShadingOptimizer
{
// Estimates the probability of winning at each bid price
// based on historical auction data
public double CalculateOptimalBid(
double trueValue,
AuctionHistory history,
string dspId)
{
// Step 1: Build distribution of competing bids
var competingBids = history.GetCompetingBids(
dspId,
lastN: 10000);
// Step 2: Calculate expected value for each bid price
double bestExpectedValue = 0;
double optimalBid = 0;
for (double bid = 0.01; bid <= trueValue; bid += 0.01)
{
// P(win at this price)
double pWin = competingBids
.Count(b => b > bid) == 0 ? 1.0 :
1.0 - (double)competingBids
.Count(b => b >= bid) / competingBids.Count;
// Expected profit = P(win) x (trueValue - bid)
double expectedValue = pWin * (trueValue - bid);
if (expectedValue > bestExpectedValue)
{
bestExpectedValue = expectedValue;
optimalBid = bid;
}
}
return optimalBid;
}
}Header Bidding Architecture
Header bidding allows publishers to offer their inventory to multiple ad exchanges simultaneously before the ad server makes its own decision. This creates a unified auction across all demand sources, maximizing publisher yield.
In client-side header bidding, Prebid.js runs in the browser and sends bid requests to multiple SSPs/exchanges in parallel. The highest bid is passed to Google Ad Manager (or another ad server) as a key-value pair (e.g., hb_pb=2.50), which is matched against price-priority line items in the ad server. The ad server then compares this header-bidding price against direct-sold campaigns to determine the final winner.
11. Frequency Capping & Pacing
Frequency capping limits how many times a particular user sees a particular ad within a given time window. Without frequency capping, a user might see the same Nike shoe ad 50 times in a single browsing session, leading to ad fatigue, negative brand perception, and wasted advertiser budget. Frequency capping is both a user experience feature and a budget optimization tool.
Frequency Cap Types
| Cap Type | Scope | Example |
|---|---|---|
| Campaign-level cap | User sees a specific campaign N times per day | Max 5 impressions per user per day for Campaign A |
| Creative-level cap | User sees a specific creative N times per day | Max 3 impressions of Creative #1 per user per day |
| Advertiser-level cap | User sees any ad from Brand X N times per day | Max 10 impressions across all Nike campaigns |
| Category-level cap | User sees N ads in a category per day | Max 3 auto insurance ads per day |
| Global cap | Max total ads shown to user across all campaigns | Max 50 ads per user per day (site-wide) |
Counting Algorithms
Accurate frequency counting at scale requires probabilistic data structures. At 10 billion impressions per day, storing exact counts for every user-campaign pair would require petabytes of storage. Instead, production systems use one of these approaches:
C#
// Option 1: Redis HyperLogLog for approximate counting
public class RedisFrequencyCapper
{
private readonly IDatabase _redis;
public async Task<bool> CheckFrequencyCapAsync(
string userId, string campaignId, int maxImpressions)
{
var key = $"freq:{campaignId}:{userId}";
// Use a sliding window counter with Redis sorted sets
var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
var windowStart = now - 86400000; // 24-hour window
// Remove expired entries
await _redis.SortedSetRemoveRangeByScoreAsync(
key, 0, windowStart);
// Count current impressions in window
var count = await _redis.SortedSetLengthAsync(key);
return count < maxImpressions;
}
public async Task RecordImpressionAsync(
string userId, string campaignId)
{
var key = $"freq:{campaignId}:{userId}";
var now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
await _redis.SortedSetAddAsync(key, now, now);
// Set TTL for auto-cleanup
await _redis.KeyExpireAsync(key, TimeSpan.FromHours(25));
}
}
// Option 2: Local Bloom Filter for ultra-low-latency pre-check
public class BloomFilterFrequencyGuard
{
private readonly BloomFilter _bloomFilter;
private readonly IDatabase _redis;
// Bloom filter in local memory for fast negative checks
public async Task<bool> MayExceedCapAsync(
string userId, string campaignId, int cap)
{
var key = $"{userId}:{campaignId}";
// Bloom filter says "definitely not seen" → allow
if (!_bloomFilter.MightContain(key))
return false;
// Bloom filter says "maybe seen" → check Redis for accuracy
return await CheckRedisAsync(userId, campaignId, cap);
}
}Pacing Algorithms
Pacing ensures that an ad campaign's budget is spent evenly throughout the day rather than exhausting it in the first hour. Without pacing, a campaign with a ,000 daily budget and a high bid might spend the entire budget during the 9 AM hour when competition is fierce, leaving no budget for the remaining 23 hours.
The most common pacing algorithm is pacing-by-spend-ratio: at each moment in time, the algorithm checks whether the campaign has spent a proportionate share of its budget. If 50% of the day has passed and only 30% of the budget is spent, the campaign increases its bid frequency. If 50% of the day has passed and 70% is spent, it throttles back.
C#
public class PacingEngine
{
public double CalculatePacingMultiplier(
Campaign campaign, DateTime now)
{
var dayProgress = (now - now.Date).TotalHours / 24.0;
var budgetProgress = campaign.SpendToday / campaign.DailyBudget;
// Ideal: spendProgress == timeProgress
var ratio = budgetProgress / Math.Max(dayProgress, 0.01);
// Smooth the multiplier to avoid oscillation
// ratio < 1 means behind schedule, ratio > 1 means ahead
if (ratio < 0.8)
return 1.2; // Increase frequency by 20%
else if (ratio v 1.2)
return 0.7; // Decrease frequency by 30%
else
return 1.0; // On track, maintain pace
// More sophisticated: exponential smoothing
// multiplier = Math.Exp(-(ratio - 1.0) * smoothingFactor);
}
}12. Ad Creative Management & Tagging
Ad creatives are the actual visual assets (banners, videos, native ads) that users see. Managing creatives at scale requires a robust system for uploading, validating, storing, serving, and tracking millions of ad assets across multiple formats, sizes, and publishers.
Creative Formats & IAB Standards
| Format | IAB Standard Sizes | File Types | Max Size |
|---|---|---|---|
| Display Banner | 300x250, 728x90, 160x600, 320x50 | JPG, PNG, GIF, HTML5 (ZIP) | 150KB (static), 300KB (rich) |
| Video | 640x480, 1920x1080, 1x1 (VAST) | MP4 (H.264/H.265) | 5MB (pre-roll), 25MB (long-form) |
| Native | Flexible (API-defined) | JSON + image URLs | N/A |
| Interstitial | 320x480, 768x1024 | JPG, PNG, HTML5 | 500KB |
| Rich Media | Expandable, pull-down, overlay | HTML5 (ZIP), JavaScript | 500KB |
| Audio | Programmatic Audio | MP3, WAV | 500KB |
Creative Validation Pipeline
Before a creative can be served, it must pass through a validation pipeline that checks for compliance, safety, and technical correctness:
C#
public class CreativeValidator
{
private readonly IBrandSafetyChecker _brandSafety;
private readonly IFileSizeValidator _fileValidator;
private readonly IHtmlSanitizer _htmlSanitizer;
private readonly IMalwareScanner _malwareScanner;
public async Task<ValidationResult> ValidateCreativeAsync(
CreativeUpload creative)
{
var results = new List<ValidationCheck>();
// 1. File size validation
results.Add(_fileValidator.ValidateSize(
creative.FileContent, creative.Format));
// 2. Format validation (correct dimensions, aspect ratio)
results.Add(ValidateDimensions(creative));
// 3. HTML5 creative security scan
if (creative.Format == CreativeFormat.Html5)
{
var sanitized = _htmlSanitizer.Sanitize(creative.HtmlContent);
results.Add(new ValidationCheck(
"Sanitization", sanitized.Warnings));
// 4. Malware scanning
var malwareResult = await _malwareScanner
.ScanAsync(creative.ZipContent);
results.Add(malwareResult);
}
// 5. Brand safety — check ad content against blocked categories
var brandCheck = await _brandSafety
.CheckCreativeContentAsync(creative);
results.Add(brandCheck);
// 6. Landing page validation
if (creative.LandingPageUrl != null)
{
var urlCheck = await ValidateLandingPage(
creative.LandingPageUrl);
results.Add(urlCheck);
}
// 7. Privacy compliance — check for tracking cookies without consent
var privacyCheck = CheckTrackingCompliance(creative);
results.Add(privacyCheck);
return new ValidationResult
{
Approved = results.All(r => r.Passed),
Checks = results
};
}
}Ad Tag Structure
An ad tag is a snippet of HTML/JavaScript that publishers embed in their pages to request and display ads. Modern ad tags are sophisticated pieces of code that handle ad requesting, rendering, impression tracking, click tracking, viewability measurement, and fallback behavior:
HTML/JavaScript
<!-- Ad Server Tag Example -->
<div id="ad-slot-12345" class="ad-container" style="width:300px;height:250px;">
<script>
(function() {
var adSlot = {
slotId: 'slot-12345',
size: [300, 250],
adUnitPath: '/12345/homepage/top',
targeting: {
section: 'technology',
article: 'breaking-news'
},
fallback: {
type: 'house_ad',
creativeId: 'house-ad-001'
},
timeout: 800
};
// Load the ad server library
var script = document.createElement('script');
script.src = 'https://adserver.platform.com/adlib.js';
script.onload = function() {
AdLib.requestAd(adSlot, function(response) {
if (response.hasAd) {
document.getElementById('ad-slot-12345')
.innerHTML = response.creativeMarkup;
// Fire impression pixel
new Image().src = response.impressionUrl;
// Set up click tracking
setupClickTracker('ad-slot-12345', response.clickUrl);
// Set up viewability tracking
observeViewability('ad-slot-12345', response.viewabilityUrl);
} else if (adSlot.fallback) {
loadFallback('ad-slot-12345', adSlot.fallback);
}
});
};
document.head.appendChild(script);
})();
</script>
</div>max-age=86400). Rich media creatives (HTML5 ZIP files) are unzipped and served as individual files to reduce payload size.
13. Pixel Tracking & Conversion Tracking
Impression tracking and conversion tracking are the financial backbone of digital advertising. Without accurate tracking, advertisers cannot verify that their ads were shown, clicked, or led to conversions. Publishers cannot prove they delivered impressions. And the entire billing and reconciliation process breaks down.
Impression Tracking Mechanism
Impression tracking works by embedding a 1x1 pixel image (called a tracking pixel) in the ad response. When the user's browser renders the ad and loads this pixel, it sends a request to the ad server, which logs the impression event. The pixel response is typically a transparent GIF or a no-cache response to prevent caching.
for reporting dashboard
C#
[ApiController]
[Route("tracking")]
public class TrackingPixelController : ControllerBase
{
private readonly IKafkaProducer _kafka;
private readonly IImpressionDeduplicator _deduplicator;
private static readonly byte[] TransparentGif = Convert
.FromBase64String("R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7");
[HttpGet("pixel.gif")]
public async Task<IActionResult> TrackImpression(
[FromQuery] string impId,
[FromQuery] string campaignId,
[FromQuery] string adId,
[FromQuery] string dspId,
[FromQuery] string winPrice,
[FromQuery] string uid)
{
// Check for duplicate impression within 24-hour window
if (await _deduplicator.IsDuplicateAsync(impId))
{
return File(TransparentGif, "image/gif");
}
// Build impression event
var impressionEvent = new ImpressionEvent
{
ImpressionId = impId,
CampaignId = campaignId,
AdId = adId,
DspId = dspId,
WinPrice = decimal.Parse(winPrice),
UserId = uid,
Timestamp = DateTime.UtcNow,
UserAgent = Request.Headers.UserAgent.ToString(),
IpAddress = HttpContext.Connection.RemoteIpAddress?.ToString(),
Referer = Request.Headers.Referer.ToString(),
Country = ResolveCountry(HttpContext.Connection.RemoteIpAddress)
};
// Send to Kafka (non-blocking)
await _kafka.ProduceAsync("impressions", impressionEvent);
// Mark as seen (prevent duplicates)
await _deduplicator.MarkSeenAsync(impId, TimeSpan.FromHours(24));
// Return transparent GIF with no-cache headers
Response.Headers["Cache-Control"] = "no-cache, no-store, must-revalidate";
Response.Headers["Pragma"] = "no-cache";
Response.Headers["Expires"] = "0";
return File(TransparentGif, "image/gif");
}
}Click Tracking
Click tracking works by wrapping the advertiser's landing page URL in a redirect. When the user clicks the ad, the browser hits the ad server's click tracking URL, which logs the click event and then redirects (HTTP 302) to the actual advertiser's landing page. The redirect adds minimal latency (one additional HTTP round-trip of ~20-50ms) but provides the ad server with complete visibility into click events.
Conversion Tracking
Conversion tracking measures the downstream actions that advertisers care about: purchases, sign-ups, app installs, form submissions. The advertiser places a conversion pixel on their confirmation/thank-you page. When a user who previously saw or clicked an ad arrives at this page, the pixel fires and the ad server attributes the conversion to the ad.
C#
[HttpGet("conversion.gif")]
public async Task<IActionResult> TrackConversion(
[FromQuery] string convId,
[FromQuery] string advId,
[FromQuery] string orderId,
[FromQuery] decimal revenue,
[FromQuery] string currency)
{
// Look up the user's impression/click history
var userHistory = await _attributionStore
.GetUserAdHistoryAsync(GetUserIdFromCookie());
var conversionEvent = new ConversionEvent
{
ConversionId = convId,
AdvertiserId = advId,
OrderId = orderId,
Revenue = revenue,
Currency = currency,
Timestamp = DateTime.UtcNow,
// Attribution: find the most recent click/view
AttributedClickId = userHistory.LastClick?.ImpressionId,
AttributedViewId = userHistory.LastView?.ImpressionId,
AttributionModel = AttributionModel.LastClick,
LookbackWindow = TimeSpan.FromDays(30)
};
await _kafka.ProduceAsync("conversions", conversionEvent);
await _deduplicator.MarkSeenAsync(convId, TimeSpan.FromDays(90));
Response.Headers["Cache-Control"] = "no-cache, no-store";
return File(TransparentGif, "image/gif");
}14. Attribution Models (Last-Click, Multi-Touch)
Attribution answers the fundamental question: which ad interaction should be credited with driving a conversion? When a user sees 5 different ads across 3 different platforms before making a purchase, how do you allocate credit? The attribution model determines this, and the choice of model has enormous financial implications — it determines which campaigns get credited, which get optimized, and which get cut.
Attribution Models Compared
| Model | How It Works | Pros | Cons |
|---|---|---|---|
| Last-Click | 100% credit to last ad clicked before conversion | Simple, easy to implement | Biases toward bottom-funnel, ignores awareness |
| First-Click | 100% credit to first ad in the journey | Values awareness/discovery | Biases toward top-funnel, ignores nurturing |
| Linear | Equal credit to all touchpoints | Fair, considers full journey | Oversimplifies — treats all touchpoints equally |
| Time-Decay | More credit to recent touchpoints | Recency bias is often reasonable | Arbitrary decay function, hard to tune |
| Position-Based (U-Shaped) | 40% first, 40% last, 20% middle | Values entry and conversion points | Arbitrary split, complex multi-touch |
| Algorithmic / Data-Driven | ML model learns optimal credit allocation | Most accurate, data-driven | Black box, requires large data volume |
C#
public class AttributionEngine
{
public AttributionResult Attribute(
ConversionEvent conversion,
IReadOnlyList<AdInteraction> interactions,
AttributionModel model)
{
var eligibleInteractions = interactions
.Where(i => i.Timestamp >= conversion.Timestamp
- TimeSpan.FromDays(30)) // Lookback window
.Where(i => i.Timestamp <= conversion.Timestamp)
.OrderBy(i => i.Timestamp)
.ToList();
return model switch
{
AttributionModel.LastClick => LastClick(eligibleInteractions),
AttributionModel.FirstClick => FirstClick(eligibleInteractions),
AttributionModel.Linear => Linear(eligibleInteractions),
AttributionModel.TimeDecay => TimeDecay(eligibleInteractions),
AttributionModel.PositionBased => PositionBased(eligibleInteractions),
_ => throw new ArgumentException($"Unknown model: {model}")
};
}
private AttributionResult TimeDecay(
IReadOnlyList<AdInteraction> interactions)
{
var halfLife = TimeSpan.FromDays(7); // 7-day half-life
var totalWeight = 0.0;
var weightedCredits = new Dictionary<string, double>();
foreach (var interaction in interactions)
{
var timeDiff = interactions.Last().Timestamp
.Subtract(interaction.Timestamp);
var weight = Math.Pow(0.5,
timeDiff.TotalDays / halfLife.TotalDays);
totalWeight += weight;
var key = interaction.CampaignId;
if (!weightedCredits.ContainsKey(key))
weightedCredits[key] = 0;
weightedCredits[key] += weight;
}
// Normalize to get credit percentages
var credits = weightedCredits.ToDictionary(
kvp => kvp.Key,
kvp => kvp.Value / totalWeight);
return new AttributionResult
{
Model = AttributionModel.TimeDecay,
Credits = credits
};
}
}15. Budget Management & Spend Pacing
Budget management is one of the most critical subsystems in an ad serving platform. Advertisers set daily and lifetime budgets for their campaigns, and the system must enforce these limits with high accuracy while optimizing for performance. A system that overspends a budget by even a small percentage can lead to massive financial liability and loss of advertiser trust.
Budget Enforcement Architecture
Budget enforcement operates at multiple levels with different consistency requirements:
Two-Phase Budget Enforcement
Production ad servers use a two-phase budget enforcement pattern to balance latency and accuracy:
C#
public class BudgetManager
{
private readonly IDatabase _redis;
private readonly IKafkaProducer _kafka;
// Phase 1: Fast pre-check (sub-millisecond, approximate)
public async Task<BudgetCheckResult> PreCheckBudgetAsync(
string campaignId)
{
var remaining = await _redis.StringGetAsync(
$"budget:{campaignId}:remaining");
if (!remaining.HasValue)
return BudgetCheckResult.Unknown; // No data → allow cautiously
var value = decimal.Parse(remaining!);
if (value <= 0)
return BudgetCheckResult.Exhausted;
if (value < 10.0m) // Less than remaining → conservative
return BudgetCheckResult.LowBudget;
return BudgetCheckResult.Available;
}
// Called after impression is served — atomically decrement budget
public async Task<bool> DeductBudgetAsync(
string campaignId, decimal cost)
{
var key = $"budget:{campaignId}:remaining";
// Atomic decrement using Lua script in Redis
var script = @"
local current = tonumber(redis.call('GET', KEYS[1]) or '0')
local cost = tonumber(ARGV[1])
if current >= cost then
redis.call('DECRBY', KEYS[1], math.floor(cost * 10000))
return 1
else
return 0
end";
var result = await _redis.ScriptEvaluateAsync(script,
new RedisKey[] { key },
new RedisValue[] { (int)(cost * 10000) });
return (int)result == 1;
}
// Phase 2: Async reconciliation (every 30 seconds)
public async Task ReconcileBudgetsAsync()
{
var recentSpend = await _kafka.ConsumeBatchAsync(
"impressions", TimeSpan.FromSeconds(30));
var spendByCampaign = recentSpend
.GroupBy(e => e.CampaignId)
.ToDictionary(
g => g.Key,
g => g.Sum(e => e.WinPrice));
foreach (var (campaignId, totalSpend) in spendByCampaign)
{
// Re-sync Redis with source of truth
var dbBudget = await _campaignStore
.GetRemainingBudgetAsync(campaignId);
await _redis.StringSetAsync(
$"budget:{campaignId}:remaining",
dbBudget.ToString());
}
}
}Budget Types
| Budget Type | Scope | Enforcement | Example |
|---|---|---|---|
| Daily Budget | Per campaign per day (advertiser timezone) | Reset at midnight local time | ,000/day |
| Lifetime Budget | Total budget for campaign duration | Decrement on each impression | ,000 total |
| Flight Budget | Budget per flight/line item | Within campaign, per line item | ,000 for summer flight |
| Pacing Budget | Smooth spending over time period | Adjusts bid frequency based on schedule | Spend over 7 days evenly |
| Portfolio Budget | Shared budget across campaigns | Allocate dynamically to best performers | across 5 campaigns |
16. Brand Safety & Fraud Detection
Brand safety and fraud detection are two of the most important trust mechanisms in digital advertising. Brand safety ensures that ads appear alongside appropriate content, while fraud detection ensures that impressions and clicks are generated by real humans rather than bots or malicious actors.
Brand Safety Mechanisms
- Keyword Blocking: Advertisers specify keywords or phrases that should block their ads from appearing (e.g., "shooting", "bomb", "pornography"). The ad server scans page content for these keywords before serving.
- Category Blocking: Using IAB Content Taxonomy, advertisers block entire categories of content (e.g., IAB19-8 for "Crime", IAB15 for "Adult Content").
- Domain/URL Blacklisting: Maintaining lists of domains where ads should never appear (known low-quality or dangerous sites).
- Page-Level Analysis: Using NLP and ML models to analyze page sentiment, topic, and content quality in real-time.
- Third-Party Verification: Integrating with IAS, DoubleVerify, or MOAT for real-time brand safety scanning via their APIs.
Ad Fraud Types & Detection
| Fraud Type | Description | Detection Method |
|---|---|---|
| Bot Traffic | Software simulating human browsing | Behavioral analysis, fingerprinting, CAPTCHA challenges |
| Click Farms | Low-wage workers clicking ads manually | Geographic patterns, device clustering, click velocity |
| Pixel Stuffing | Ads rendered in 1x1 pixel frames (invisible to users) | Viewability measurement, iframe analysis |
| Ad Stacking | Multiple ads layered on top of each other | Layer analysis, z-index detection |
| Domain Spoofing | Fake sites impersonating premium publishers | Ads.txt, sellers.json, domain verification |
| Click Injection | Mobile apps hijacking ad clicks | Install referrer validation, timestamp analysis |
| SDK Spoofing | Fake ad SDK requests from server farms | Device attestation, supply chain verification |
| Ad Fraud Bots | Sophisticated bots mimicking human ad engagement | Machine learning classifiers, mouse movement analysis |
C#
public class FraudDetectionEngine
{
private readonly IClickPatternAnalyzer _clickAnalyzer;
private readonly IDeviceFingerprinter _deviceFingerprinter;
private readonly IBehavioralModel _behavioralModel;
public FraudScore EvaluateTraffic(TrafficEvent evt)
{
var score = 0.0;
// Check 1: Click velocity — too many clicks in short time
var recentClicks = _clickAnalyzer.GetRecentClicks(
evt.IpAddress, TimeSpan.FromMinutes(5));
if (recentClicks > 20)
score += 0.3;
// Check 2: Device fingerprint anomalies
var fingerprint = _deviceFingerprinter.GetFingerprint(evt);
if (fingerprint.SuspiciousCharacteristics.Any())
score += 0.2;
// Check 3: Behavioral model — mouse movement, scroll, dwell time
var behavioralScore = _behavioralModel
.AnalyzeBehavior(evt.UserBehavior);
if (behavioralScore < 0.3) // Low human-likeness score
score += 0.25;
// Check 4: Geographic anomalies
if (IsImpossibleTravel(evt.Geo, evt.LastSeenGeo))
score += 0.15;
// Check 5: Known bot signatures
if (_botDatabase.ContainsSignature(evt.UserAgent))
score += 0.4;
return new FraudScore
{
Score = Math.Min(score, 1.0),
Classification = score > 0.7 ? FraudClass.HighConfidence :
score > 0.4 ? FraudClass.Suspicious :
FraudClass.Legitimate,
Signals = GetMatchedSignals(evt)
};
}
}17. Viewability Measurement
Viewability answers a simple but critical question: was the ad actually seen by a human? An impression that is never in the user's viewport is worthless to the advertiser. The Media Rating Council (MRC) defines a viewable display impression as one where at least 50% of the ad's pixels are in the viewable area of the browser for at least 1 continuous second. For video, the standard is 50% of pixels for 2 continuous seconds.
Viewability Measurement Architecture
Viewability is measured client-side using JavaScript libraries that observe the ad element's position relative to the viewport. The measurement script tracks:
- Intersection Observer: Monitors whether the ad element enters/exits the viewport using the browser's IntersectionObserver API.
- Geometry Calculation: Calculates the percentage of ad pixels that are visible based on element position and viewport dimensions.
- Duration Tracking: Measures how long the ad remains in view continuously.
- Tab Visibility: Checks whether the browser tab is in the foreground (Page Visibility API).
- Ad Rendering: Verifies the ad creative actually loaded (not blocked by ad blocker or failed to render).
JavaScript
class ViewabilityTracker {
constructor(config) {
this.adElement = config.adElement;
this.viewabilityUrl = config.viewabilityUrl;
this.threshold = config.threshold || 50; // 50% pixels
this.duration = config.duration || 1000; // 1 second
this.isVisible = false;
this.visibleStartTime = null;
this.isViewable = false;
this.setupIntersectionObserver();
}
setupIntersectionObserver() {
const options = {
root: null, // viewport
rootMargin: '0px',
threshold: [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
};
this.observer = new IntersectionObserver((entries) => {
entries.forEach(entry => {
const visiblePercentage =
(entry.intersectionRatio * 100).toFixed(0);
if (entry.intersectionRatio >= this.threshold / 100
&& document.visibilityState === 'visible') {
if (!this.isVisible) {
this.isVisible = true;
this.visibleStartTime = Date.now();
}
// Check if visible for required duration
const elapsed = Date.now() - this.visibleStartTime;
if (elapsed >= this.duration && !this.isViewable) {
this.markAsViewable(visiblePercentage);
}
} else {
this.isVisible = false;
this.visibleStartTime = null;
}
});
}, options);
this.observer.observe(this.adElement);
}
markAsViewable(visiblePercentage) {
this.isViewable = true;
this.observer.disconnect();
// Fire viewability beacon
new Image().src = ${this.viewabilityUrl} +
&viewable=true +
&visible_pct= +
&visible_duration=;
}
}Viewability by Ad Format
| Format | MRC Standard | Typical Viewability | Measurement Tool |
|---|---|---|---|
| Display (Desktop) | 50% pixels, 1 second | 50-65% | JavaScript Intersection Observer |
| Display (Mobile) | 50% pixels, 1 second | 55-70% | JavaScript + native viewability APIs |
| Video (In-Stream) | 50% pixels, 2 seconds | 70-85% | VPAID / OM SDK |
| Video (Out-Stream) | 50% pixels, 2 seconds | 40-60% | JavaScript player integration |
| Native | 50% pixels, 1 second | 60-75% | Custom measurement per layout |
| Audio | Play initiated | 80-95% | Audio player event tracking |
18. Reporting & Analytics Pipeline
Reporting is the interface through which advertisers, publishers, and operations teams understand the performance of their campaigns. A production ad serving platform must provide both real-time dashboards (for monitoring current campaign performance) and historical analytics (for deep-dive analysis and billing reconciliation). The reporting pipeline must handle billions of events per day and support complex aggregations across multiple dimensions.
OLAP] REDIS_RT[Redis
Real-Time Counters] S3_COLD[S3
Cold Storage] end subgraph "Presentation" DASH[Real-Time Dashboard] REPORT[Batch Reports] EXPORT[Data Export API] end IMP --> KAFKA CLK --> KAFKA CONV --> KAFKA BID --> KAFKA WIN --> KAFKA KAFKA --> FLINK FLINK --> REDIS_RT FLINK --> CLICKHOUSE KAFKA --> S3_COLD REDIS_RT --> DASH CLICKHOUSE --> REPORT CLICKHOUSE --> EXPORT
ClickHouse Schema for Ad Analytics
SQL
-- ClickHouse table for impression analytics
CREATE TABLE ad_impressions
(
impression_id String,
timestamp DateTime,
campaign_id String,
ad_id String,
advertiser_id String,
publisher_id String,
site_domain LowCardinality(String),
country LowCardinality(String),
region LowCardinality(String),
city LowCardinality(String),
device_type LowCardinality(String),
os LowCardinality(String),
browser LowCardinality(String),
ad_format LowCardinality(String),
ad_size LowCardinality(String),
dsp_id LowCardinality(String),
deal_id Nullable(String),
win_price Decimal64(4),
viewability Nullable(UInt8),
fraud_score Nullable(Float32),
auction_type LowCardinality(String),
bid_request_id String
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (campaign_id, timestamp, publisher_id)
TTL timestamp + INTERVAL 5 YEAR;
-- Materialized view for real-time campaign aggregations
CREATE MATERIALIZED VIEW campaign_hourly_agg
ENGINE = SummingMergeTree()
PARTITION BY toYYYYMMDD(hour)
ORDER BY (campaign_id, hour)
AS SELECT
campaign_id,
toStartOfHour(timestamp) AS hour,
count() AS impressions,
sum(win_price) AS total_spend,
uniqExact(user_id) AS unique_users,
avg(viewability) AS avg_viewability
FROM ad_impressions
GROUP BY campaign_id, hour;Real-Time Reporting API
C#
[ApiController]
[Route("api/v1/reporting")]
public class ReportingController : ControllerBase
{
private readonly IClickHouseClient _clickhouse;
private readonly IDistributedCache _cache;
[HttpGet("campaign/{campaignId}/metrics")]
public async Task<CampaignMetrics> GetCampaignMetrics(
string campaignId,
[FromQuery] DateTime startDate,
[FromQuery] DateTime endDate,
[FromQuery] string granularity = "daily")
{
var cacheKey = $"report:{campaignId}:{startDate:yyyyMMdd}" +
$":{endDate:yyyyMMdd}:{granularity}";
var cached = await _cache.GetAsync<CampaignMetrics>(cacheKey);
if (cached != null) return cached;
var query = granularity switch
{
"hourly" => $@"
SELECT
toStartOfHour(timestamp) as period,
count() as impressions,
sum(win_price) as spend,
uniqExact(impression_id) as unique_impressions,
avg(viewability) as avg_viewability
FROM ad_impressions
WHERE campaign_id = '{campaignId}'
AND timestamp >= '{startDate:yyyy-MM-dd}'
AND timestamp < '{endDate.AddDays(1):yyyy-MM-dd}'
GROUP BY period ORDER BY period",
"daily" => $@"
SELECT
toDate(timestamp) as period,
count() as impressions,
sum(win_price) as spend,
uniqExact(impression_id) as unique_impressions,
avg(viewability) as avg_viewability
FROM ad_impressions
WHERE campaign_id = '{campaignId}'
AND timestamp >= '{startDate:yyyy-MM-dd}'
AND timestamp < '{endDate.AddDays(1):yyyy-MM-dd}'
GROUP BY period ORDER BY period",
_ => throw new ArgumentException("Invalid granularity")
};
var result = await _clickhouse.QueryAsync<CampaignMetrics>(query);
await _cache.SetAsync(cacheKey, result, TimeSpan.FromMinutes(5));
return result;
}
}19. A/B Testing Ad Creatives
A/B testing (also called split testing) is the practice of showing different ad creatives to different segments of users and measuring which performs better. This is essential for optimizing campaign performance — advertisers need to know which creative resonates with their target audience, which headline drives more clicks, and which call-to-action generates the most conversions.
Ad Creative A/B Testing Architecture
The A/B testing system in an ad server works by assigning users to test variants deterministically (using consistent hashing) and tracking performance metrics per variant. The system must ensure statistical validity while operating within the latency constraints of real-time ad serving.
C#
public class AdCreativeSplitTester
{
private readonly ICampaignStore _campaignStore;
private readonly IConsistentHasher _hasher;
private readonly IStatisticsEngine _stats;
public async Task<CreativeVariant> SelectVariantAsync(
string campaignId, string userId, AdRequest request)
{
var experiment = await _campaignStore
.GetActiveExperimentAsync(campaignId);
if (experiment == null)
return await _campaignStore.GetDefaultCreativeAsync(campaignId);
// Deterministic assignment based on user ID + experiment ID
var hash = _hasher.Hash($"{userId}:{experiment.Id}");
var bucket = hash % 100; // 0-99
// Traffic allocation (e.g., 50/50 split)
var variant = bucket < experiment.TrafficAllocation
? experiment.ControlVariant
: experiment.TestVariant;
// Track assignment for analytics
await TrackAssignment(experiment.Id, variant.Id, userId);
return variant;
}
public ExperimentResult GetResults(string experimentId)
{
var controlMetrics = _stats.GetMetrics(experimentId, "control");
var testMetrics = _stats.GetMetrics(experimentId, "test");
return new ExperimentResult
{
ControlCTR = controlMetrics.CTR,
TestCTR = testMetrics.CTR,
Lift = (testMetrics.CTR - controlMetrics.CTR)
/ controlMetrics.CTR * 100,
PValue = _stats.CalculatePValue(
controlMetrics, testMetrics),
IsStatisticallySignificant = _stats
.CalculatePValue(controlMetrics, testMetrics) < 0.05,
SampleSize = controlMetrics.SampleSize
+ testMetrics.SampleSize,
ConfidenceInterval = _stats
.CalculateConfidenceInterval(controlMetrics, testMetrics)
};
}
}Multi-Variate Testing
Beyond simple A/B testing, multi-variate testing (MVT) allows advertisers to test multiple creative elements simultaneously (e.g., headline x image x CTA button). This requires a factorial experiment design where every combination of elements is tested. For example, testing 3 headlines x 3 images x 2 CTAs requires 18 variants, each receiving 5-10% of traffic.
Key Metrics for Ad A/B Testing
| Metric | Definition | Typical Benchmark |
|---|---|---|
| CTR (Click-Through Rate) | Clicks / Impressions | 0.5-2% (display), 1-5% (native) |
| View-Through Rate | Viewable Impressions / Total Impressions | 50-70% |
| CVR (Conversion Rate) | Conversions / Clicks | 1-5% |
| CPM (Cost Per Mille) | Spend / (Impressions / 1000) | -15 (varies by inventory) |
| CPC (Cost Per Click) | Spend / Clicks | .50-5.00 |
| CPA (Cost Per Acquisition) | Spend / Conversions | -200 (varies by vertical) |
| ROAS (Return On Ad Spend) | Revenue / Spend | 300-800% |
| Engagement Rate | Engagements / Impressions | 2-8% (video), 1-3% (display) |
20. DMP/CDP Integration & Privacy (GDPR, CCPA, Topics API)
Data Management Platforms (DMPs) and Customer Data Platforms (CDPs) are the data backbone of modern advertising. They collect, organize, and activate audience data that powers targeting, personalization, and attribution. However, the ad tech industry is undergoing a fundamental transformation as third-party cookies are deprecated, privacy regulations tighten, and users demand more control over their data.
DMP vs CDP
| Aspect | DMP | CDP |
|---|---|---|
| Data Types | Third-party + anonymous data | First-party + identified data |
| User Identity | Anonymous (cookie IDs, device IDs) | Known (email, customer ID) |
| Data Retention | 30-90 days (cookie-based) | Permanent (persistent profiles) |
| Primary Use | Targeting, audience segments | Personalization, CRM sync |
| Privacy Impact | Higher (third-party data) | Lower (first-party consented) |
| Cookie Deprecation Impact | Severe (core functionality) | Mitigated (uses first-party IDs) |
Privacy Regulations
- GDPR (General Data Protection Regulation): EU regulation requiring explicit consent for processing personal data. Ad servers must verify that users have given valid consent (via a Consent Management Platform or CMP) before setting cookies, tracking users, or running behavioral targeting. The legal basis for ad tech processing is typically consent (Article 6(1)(a)).
- CCPA/CPRA (California Consumer Privacy Act): California regulation giving consumers the right to know what data is collected, opt out of its sale, and request deletion. Ad platforms must honor Do Not Sell requests and provide a clear opt-out mechanism.
- ePrivacy Directive: EU regulation requiring consent for storing/reading cookies on user devices. This is the regulation most directly impacting ad tech cookie usage.
- Google Privacy Sandbox: Google's initiative to replace third-party cookies with privacy-preserving APIs including Topics API, Attribution Reporting API, and Protected Audiences API (formerly FLEDGE).
Topics API (Successor to FLoC)
Google's Topics API allows browsers to infer a user's interests based on their browsing history and share a limited number of topics (3 topics per week, from a taxonomy of ~350 topics) with ad platforms. This replaces the granular behavioral tracking of third-party cookies with a coarser but more privacy-preserving signal.
C#
// Integrating Topics API signals into targeting
public class TopicsApiTargetingAdapter
{
public IReadOnlyList<TargetedCampaign> ApplyTopicsSignals(
IReadOnlyList<TargetedCampaign> campaigns,
TopicsData topicsData)
{
var userTopics = topicsData.InterestTopics; // e.g., ["Auto", "Finance"]
return campaigns.Where(campaign =>
{
var targeting = campaign.TargetingCriteria;
// If campaign uses Topics-based targeting
if (targeting.TopicsTargeting?.Any() == true)
{
return targeting.TopicsTargeting
.Any(tt => userTopics.Contains(tt));
}
// Map Topics to IAB categories for traditional targeting
var mappedCategories = MapTopicsToIabCategories(userTopics);
if (targeting.ContextualCategories?.Any() == true)
{
return targeting.ContextualCategories
.Any(cc => mappedCategories.Contains(cc));
}
return true; // No Topics targeting required
}).ToList();
}
private Dictionary<string, string[]> TopicToIabMapping = new()
{
["Auto"] = new[] { "IAB3", "IAB3-7" }, // Automotive
["Finance"] = new[] { "IAB13", "IAB13-1" }, // Finance
["Food"] = new[] { "IAB6", "IAB6-8" }, // Food & Drink
["Tech"] = new[] { "IAB19", "IAB19-6" }, // Technology
["Travel"] = new[] { "IAB20", "IAB20-3" } // Travel
};
}Consent Management Integration
The ad server must integrate with a Consent Management Platform (CMP) to verify user consent before processing personal data. The IAB's Transparency and Consent Framework (TCF) v2.2 defines a standard for communicating consent between CMPs, publishers, and ad tech vendors:
C#
public class ConsentValidator
{
public bool HasValidConsent(ConsentData consent, ProcessingPurpose purpose)
{
// Check if user consented to the specific processing purpose
// TCF v2.2 purposes:
// Purpose 1: Store and/or access information on a device
// Purpose 2: Select basic ads
// Purpose 3: Create a personalized ads profile
// Purpose 4: Select personalized ads
// Purpose 5: Create a personalized content profile
// Purpose 6: Select personalized content
// Purpose 7: Measure ad performance
// Purpose 8: Measure content performance
// Purpose 9: Apply market research
// Purpose 10: Develop and improve products
return consent.PurposeConsents.Contains((int)purpose)
&& consent.VendorConsents.Contains(_vendorId)
&& !consent.LegitimateInterestsRejected;
}
public ProcessingResult ProcessWithConsent(
AdRequest request, Func<ProcessingResult> process)
{
var consent = request.ConsentData;
if (consent == null || !consent.HasGdprApplies)
return process(); // Non-EU traffic, no GDPR
if (HasValidConsent(consent, ProcessingPurpose.BasicAds))
return process(); // Consent given, process normally
// No consent: fall back to non-personalized ads only
return new ProcessingResult
{
AdServed = true,
Personalized = false,
TargetingUsed = TargetingType.ContextualOnly
};
}
}21. API Design & IAB Compliance
A well-designed API is critical for an ad serving platform. Publishers, advertisers, agencies, and third-party integrators all interact with the platform through APIs. The API must be RESTful, well-documented, versioned, and designed for high throughput with proper authentication and rate limiting.
Core API Endpoints
| Endpoint | Method | Description | Rate Limit |
|---|---|---|---|
| /api/v1/campaigns | POST | Create a new campaign | 100/min |
| /api/v1/campaigns/{id} | GET | Get campaign details | 1000/min |
| /api/v1/campaigns/{id} | PUT | Update campaign | 100/min |
| /api/v1/campaigns/{id}/pause | POST | Pause a campaign | 100/min |
| /api/v1/creatives | POST | Upload a creative asset | 50/min |
| /api/v1/creatives/{id}/validate | POST | Validate creative before upload | 200/min |
| /api/v1/reporting/campaigns/{id} | GET | Campaign performance report | 60/min |
| /api/v1/reporting/campaigns/{id}/daily | GET | Daily breakdown report | 60/min |
| /api/v1/audiences/segments | POST | Create audience segment | 100/min |
| /api/v1/pixels | POST | Register conversion pixel | 100/min |
| /api/v1/budgets/{id}/status | GET | Check budget status | 1000/min |
OpenRTB Compliance
Any ad serving platform that participates in the programmatic ecosystem must comply with IAB OpenRTB specifications. Key compliance requirements include:
- OpenRTB 2.6: The latest RTB protocol supporting video, native, audio, and DOOH (digital out-of-home) ad formats. All bid requests and responses must conform to the spec.
- OpenRTB 2.5: Still widely used. Platforms should support both 2.5 and 2.6 for maximum compatibility.
- VAST 4.2: Video Ad Serving Template for video ad delivery. Defines the XML structure for video ad requests, responses, and tracking events.
- VPAID 2.0: Video Player-Ad Interface Definition for interactive video ads (being deprecated in favor of OMID).
- ads.txt 1.1: Authorized Digital Sellers for preventing unauthorized inventory reselling.
- sellers.json 1.1: Supply chain object for transparency into who is selling inventory.
- Open Measurement (OM) 1.x: Standard for viewability and measurement across all ad formats.
- Ads.cert 2.0: Cryptographic signing of bid requests to prevent spoofing and manipulation.
C#
[ApiController]
[Route("api/v1")]
[Authorize]
public class CampaignApiController : ControllerBase
{
private readonly ICampaignService _campaignService;
private readonly IRateLimiter _rateLimiter;
private readonly IAuditLogger _auditLogger;
[HttpPost("campaigns")]
[ProducesResponseType(typeof(CampaignResponse), 201)]
[ProducesResponseType(typeof(ErrorResponse), 400)]
[ProducesResponseType(typeof(ErrorResponse), 429)]
public async Task<IActionResult> CreateCampaign(
[FromBody] CreateCampaignRequest request)
{
// Rate limiting
var userId = GetUserId();
if (!await _rateLimiter.AllowAsync($"create_campaign:{userId}", 100, TimeSpan.FromMinutes(1)))
return StatusCode(429, new ErrorResponse
{
Error = "rate_limit_exceeded",
Message = "Maximum 100 campaign creations per minute",
RetryAfterSeconds = 60
});
// Validation
if (!ModelState.IsValid)
return BadRequest(new ErrorResponse
{
Error = "validation_error",
Details = ModelState.Values
.SelectMany(v => v.Errors)
.Select(e => e.ErrorMessage)
});
// Create campaign
var campaign = await _campaignService
.CreateCampaignAsync(request, userId);
// Audit log
await _auditLogger.LogAsync("campaign_created",
userId, campaign.Id, request);
return CreatedAtAction(
nameof(GetCampaign),
new { id = campaign.Id },
CampaignResponse.FromDomain(campaign));
}
[HttpGet("campaigns/{id}")]
[ProducesResponseType(typeof(CampaignResponse), 200)]
public async Task<IActionResult> GetCampaign(string id)
{
var campaign = await _campaignService.GetCampaignAsync(id);
if (campaign == null)
return NotFound();
return Ok(CampaignResponse.FromDomain(campaign));
}
}22. Monitoring, Cost Estimation & Testing Strategy
Monitoring & Alerting
Monitoring an ad serving platform requires tracking metrics across multiple dimensions: system health, business performance, financial accuracy, and latency. The monitoring stack must detect anomalies in real-time because every minute of degraded performance translates to lost revenue.
| Metric Category | Key Metrics | Alert Threshold |
|---|---|---|
| Latency | p50, p95, p99 ad response time | p99 > 100ms (critical), > 80ms (warning) |
| Throughput | QPS, requests/second, bids/second | Drop > 30% from baseline |
| Error Rate | HTTP 5xx rate, timeout rate, bid error rate | Error rate > 0.1% |
| Fill Rate | Impressions served / ad requests | Fill rate drops > 10% from baseline |
| Revenue | Revenue per hour, CPM trends, spend pacing | Revenue drop > 20% from hourly forecast |
| Auction Health | Bid rate, win rate, average CPM | Bid rate drops > 15% |
| System Resources | CPU, memory, disk, network utilization | CPU > 80%, memory > 85% |
| Data Pipeline | Kafka lag, ClickHouse query latency | Kafka lag > 100K messages |
| Financial | Spend vs. budget accuracy, reconciliation gaps | Discrepancy > 0.5% |
Cost Estimation
Running an ad serving platform at scale involves significant infrastructure costs. Here is a rough monthly cost estimate for serving 10 billion impressions per month:
| Cost Category | Monthly Estimate | Notes |
|---|---|---|
| Compute (Kubernetes clusters) | ,000 - ,000 | 500+ pods across 3 regions, c6i.4xlarge instances |
| Redis Cluster (ElastiCache) | ,000 - ,000 | 6-node cluster with 256GB RAM total |
| Kafka (MSK) | ,000 - ,000 | 6-broker cluster, 10TB storage |
| ClickHouse (cloud) | ,000 - ,000 | 3-node cluster, 50TB SSD storage |
| Object Storage (S3) | ,000 - ,000 | 10TB creative assets + 50TB data archive |
| CDN (CloudFront) | ,000 - ,000 | 500TB/month transfer, 100+ edge locations |
| Data Transfer (inter-region) | ,000 - ,000 | Cross-region replication, DSP connections |
| Monitoring (Datadog/SaaS) | ,000 - ,000 | Metrics, logs, APM, synthetic monitoring |
| Total | ,000 - ,000 | For 10B impressions/month |
Testing Strategy
Testing an ad serving platform requires a multi-layered approach due to the real-time, distributed nature of the system:
- Unit Tests: Test individual components in isolation — targeting evaluation logic, auction algorithms, pacing calculations, attribution models. Aim for 90%+ code coverage on business logic.
- Integration Tests: Test service interactions — ad request to bid response flow, Kafka event processing, ClickHouse query correctness. Use testcontainers for Redis and Kafka.
- Load Tests: Simulate production traffic levels (500K QPS) to validate latency targets and auto-scaling behavior. Use tools like k6, Gatling, or custom load generators.
- Auction Simulation Tests: Run simulated auctions with synthetic bid distributions to validate auction fairness, revenue accuracy, and pacing behavior.
- Shadow/Canary Testing: Deploy new versions to a subset of traffic and compare metrics against the current production version before full rollout.
- Chaos Engineering: Inject failures (DSP timeout, Redis partition, Kafka broker failure) to validate graceful degradation behavior.
C#
// Example: Auction engine unit test
[TestClass]
public class AuctionEngineTests
{
[TestMethod]
public void SecondPriceAuction_ShouldChargeSecondHighestPlusPenny()
{
var engine = new AuctionEngine();
var bids = new List<Bid>
{
new Bid { DspId = "dsp1", Price = 3.00m, ImpId = "imp1" },
new Bid { DspId = "dsp2", Price = 2.50m, ImpId = "imp1" },
new Bid { DspId = "dsp3", Price = 1.80m, ImpId = "imp1" },
};
var result = engine.RunAuction(bids, AuctionType.SecondPrice);
Assert.AreEqual("dsp1", result.WinnerDspId);
Assert.AreEqual(2.51m, result.ClearingPrice); // Second + .01
}
[TestMethod]
public void FirstPriceAuction_ShouldChargeExactBid()
{
var engine = new AuctionEngine();
var bids = new List<Bid>
{
new Bid { DspId = "dsp1", Price = 3.00m, ImpId = "imp1" },
new Bid { DspId = "dsp2", Price = 2.50m, ImpId = "imp1" },
};
var result = engine.RunAuction(bids, AuctionType.FirstPrice);
Assert.AreEqual("dsp1", result.WinnerDspId);
Assert.AreEqual(3.00m, result.ClearingPrice);
}
[TestMethod]
public void PacingEngine_ShouldThrottleAheadOfSchedule()
{
var pacer = new PacingEngine();
var campaign = new Campaign
{
DailyBudget = 10000m,
SpendToday = 7000m // 70% spent
};
var timeOfDay = new DateTime(2025, 1, 1, 12, 0, 0); // Noon = 50%
var multiplier = pacer.CalculatePacingMultiplier(
campaign, timeOfDay);
// 70% spent at 50% of day → should throttle down
Assert.IsTrue(multiplier < 1.0,
$"Expected throttling but got multiplier: {multiplier}");
}
[TestMethod]
[DataRow("age:25-34,gender:male,geo:US", "age:30,gender:male,geo:US", true)]
[DataRow("age:25-34,gender:male,geo:US", "age:40,gender:male,geo:US", false)]
[DataRow("age:25-34,gender:male,geo:US", "age:30,gender:female,geo:US", false)]
public void TargetingEngine_MatchesCorrectly(
string targetingStr, string userStr, bool expectedMatch)
{
var engine = new TargetingEngine();
var targeting = ParseTargeting(targetingStr);
var user = ParseUser(userStr);
var matches = engine.EvaluateSingleTargeting(targeting, user);
Assert.AreEqual(expectedMatch, matches);
}
}23. Interview Q&A Deep Dive
The following questions and answers cover the most commonly asked ad serving system design interview questions at senior and staff engineer levels. Each answer includes architectural reasoning, trade-offs, and production-relevant details.
Q1: How would you design the ad serving platform to handle 500K QPS with sub-100ms latency?
Answer: The key to achieving sub-100ms latency at 500K QPS is separating the synchronous ad serving path from the asynchronous data processing path. The synchronous path (request to response) uses Redis for all lookups (user profiles, frequency caps, budgets) with sub-millisecond read times. The targeting engine uses pre-computed segment lists refreshed every 5-15 minutes rather than querying a database on every request. DSP callouts happen in parallel with a 60-70ms timeout, using connection pooling and HTTP/2 multiplexing. All logging, analytics, and billing reconciliation happen asynchronously via Kafka, completely decoupled from the serving path. Geographic distribution via GeoDNS and CDN ensures requests hit the nearest ad server cluster. Auto-scaling on Kubernetes with pre-warmed pods handles traffic spikes without cold-start delays.
Q2: How do you prevent budget overspend in a distributed system?
Answer: Budget enforcement uses a two-phase approach. Phase 1: A fast pre-check against Redis counters (sub-millisecond) determines whether a campaign has budget remaining. If the counter is below a threshold, the campaign is throttled or blocked. Phase 2: After serving an impression, the cost is atomically deducted from Redis using a Lua script. A background reconciler runs every 30 seconds, syncing Redis counters with the source-of-truth database to correct any drift caused by failed deductions or Redis evictions. Additionally, a hard budget cap is enforced at the database level — even if Redis allows an impression through, the billing system will flag and reverse any charge that exceeds the campaign's budget. For daily budgets, the system accounts for timezone differences and resets at midnight in the advertiser's timezone, not the server's timezone.
Q3: How does frequency capping work at scale without storing exact counts for every user-campaign pair?
Answer: At 10 billion impressions per day with 500 million unique users and 100,000 active campaigns, storing exact counts for every user-campaign pair would require hundreds of terabytes of storage. Instead, we use Redis sorted sets with timestamps, which allows efficient sliding window counting. Each impression is stored as a sorted set entry with the timestamp as both the key and score. To check a 24-hour frequency cap, we remove entries older than 24 hours using ZREMRANGEBYSCORE, then count remaining entries with ZCARD. This operation is O(log(N + M)) where N is the number of entries and M is the number removed. For campaigns with very high frequency caps (>100/day), we use HyperLogLog for approximate counting, accepting a ~1% error rate in exchange for 90% memory savings. Redis TTL ensures automatic cleanup of expired frequency cap data.
Q4: How would you handle a situation where a DSP is consistently slow and causing timeouts?
Answer: I would implement an adaptive timeout and circuit breaker pattern. First, track per-DSP latency metrics (p50, p95, p99) using a sliding window of the last 1000 requests. If a DSP's p95 latency exceeds the timeout threshold for 5 consecutive minutes, reduce the timeout allocated to that DSP from 60ms to 30ms. If it continues to time out, open the circuit (stop sending bid requests) for 60 seconds, then send a probe request. If the probe succeeds, gradually restore traffic over 5 minutes. If it fails, keep the circuit open for 5 minutes. This prevents a single slow DSP from consuming the latency budget and degrading the entire ad serving experience. Additionally, I would log slow DSP patterns to identify systemic issues (e.g., DSP overloads during certain hours) and adjust the callout strategy accordingly — perhaps excluding that DSP during peak hours entirely.
Q5: How do you ensure attribution accuracy across devices?
Answer: Cross-device attribution is one of the hardest problems in ad tech because users interact with ads on multiple devices (phone, laptop, tablet, smart TV) and we need to connect these interactions to a single person. The approaches include deterministic matching (logged-in users across devices via email/login ID — most accurate but limited coverage), probabilistic matching (using device fingerprinting, IP addresses, household data, and machine learning to predict device ownership — broader coverage but lower accuracy), and people-based measurement (using data clean rooms where both the advertiser and publisher contribute hashed customer data for matching without exposing raw data). Google's Privacy Sandbox aims to provide cross-device attribution through Topics API and Attribution Reporting API without revealing individual user data. In practice, deterministic matching via logged-in users (Google, Facebook, Amazon) provides the most accurate cross-device attribution, which is why walled gardens have a significant advantage in measurement.
Q6: Explain the difference between server-side and client-side header bidding. When would you use each?
Answer: Client-side header bidding runs in the browser using Prebid.js. The browser sends bid requests to multiple SSPs/exchanges in parallel, collects bids, and passes the winning bid to the ad server. The advantage is access to browser-level signals (cookies, browser fingerprint, user agent) which provide better targeting. The disadvantage is that it adds 300-2000ms to page load time because the browser must execute JavaScript and wait for network round-trips. Server-side header bidding (Prebid Server, Google Open Bidding) moves this process server-to-server. The publisher's server sends bid requests on behalf of the user, reducing latency to under 100ms. However, server-side loses access to browser-level user data, potentially reducing CPMs by 10-30%. The recommended approach is a hybrid: use server-side header bidding as the primary path (for latency), with client-side as a fallback for premium inventory where the CPM uplift justifies the latency cost. Google's Open Bidding is particularly effective because it runs within Google's infrastructure, maintaining access to Google's cookie graph while being server-side fast.
Q7: How would you handle GDPR compliance in the ad serving pipeline?
Answer: GDPR compliance requires changes at every layer of the ad serving pipeline. At the entry point, the ad request must include the user's consent string (TCF v2.2 format) from the CMP. The ad server parses this consent string and determines which processing purposes are consented. For users who have not consented to personalized advertising (purposes 1-4), the system falls back to contextual-only targeting, stripping all personal data from bid requests. The bid request to DSPs includes the GDPR flag and consent string, allowing DSPs to determine whether they can lawfully process the bid request. Impression and click logs are pseudonymized (user IDs are hashed) and stored with restricted access. A data retention policy automatically deletes personal data after the specified period. Users can request data deletion via a Subject Access Request (SAR) API, which triggers deletion of all personal data across all systems. Audit logs track all data processing activities for regulatory reporting.
Q8: How do you detect and prevent ad fraud at the infrastructure level?
Answer: Ad fraud detection operates at multiple layers: (1) Pre-bid filtering: Before serving an impression, check the bid request for fraud signals — known bot IP ranges, suspicious user agents, data center IP addresses, known fraud device fingerprints. Block these at the edge using a rules engine. (2) Real-time behavioral analysis: Use machine learning models to analyze click patterns, mouse movements, scroll behavior, and dwell time in real-time. A click that arrives within 50ms of page load with no mouse movement is likely a bot. (3) Post-bid verification: After serving an impression, send the event to a third-party verification service (IAS, DoubleVerify) for independent fraud scoring. Impressions flagged as fraudulent are excluded from billing. (4) Pattern detection: Run batch analytics to detect systematic fraud patterns — click farms show unusual geographic concentration, bot networks share IP blocks and timing patterns, domain spoofing is detected via ads.txt mismatches. (5) Supply chain verification: Use ads.txt, sellers.json, andads.cert to verify that the publisher is authorized to sell the inventory and the bid request hasn't been tampered with.
Q9: How would you design the reporting pipeline to support both real-time dashboards and historical analytics?
Answer: The reporting pipeline uses a lambda architecture with both a speed layer and a batch layer. The speed layer uses Redis counters (HyperLogLog for unique users, sorted sets for time series) to provide sub-second aggregations for real-time dashboards. These counters are updated directly from the Kafka event stream by a Flink processor. The batch layer writes detailed event data to ClickHouse, partitioned by date and campaign, with materialized views pre-aggregating common query patterns (hourly, daily, by publisher, by country). ClickHouse provides exact aggregations with 5-30 second query latency for historical queries. For very old data (>90 days), events are archived to S3 and can be queried via Athena or restored to ClickHouse on demand. The reporting API first checks Redis for real-time data (last 24 hours), then falls back to ClickHouse for historical data. Dashboard widgets poll the API every 30 seconds for live updates, while detailed reports are generated as batch queries with caching (5-minute TTL).
Q10: What are the key trade-offs in designing an ad serving platform?
Answer: The fundamental trade-offs include: (1) Latency vs. accuracy: Shorter DSP timeouts improve latency but reduce the number of bids and potentially the winning bid price. (2) Personalization vs. privacy: More user data enables better targeting and higher CPMs but increases privacy risk and regulatory burden. (3) Revenue optimization vs. user experience: Aggressive ad frequency increases short-term revenue but drives users to install ad blockers. (4) Consistency vs. availability: Budget enforcement needs strong consistency (prevent overspend), but reporting can tolerate eventual consistency (slightly stale data is acceptable). (5) Cost vs. performance: Premium infrastructure (dedicated Redis clusters, cross-connect links to DSPs) reduces latency but increases cost. (6) Simplicity vs. flexibility: A simpler architecture is easier to maintain but may not support the long tail of targeting options and deal types that advertisers demand. The key is to understand which trade-offs matter most for your specific market position and optimize accordingly.