Table of Contents

1. Introduction — What Is a Web Crawler?
2. Functional & Non-Functional Requirements
3. Capacity Estimation & Back-of-Envelope
4. Data Model & Storage Schema
5. API Design
6. High-Level Architecture
7. URL Frontier & Priority Queue
8. Web Fetching Pipeline
9. URL Filtering & Deduplication
10. Politeness & robots.txt
11. HTML Parsing & Content Extraction
12. Distributed Architecture
13. Content Storage & Indexing
14. Re-crawling & Freshness
15. DNS Resolution at Scale
16. Error Handling & Resilience
17. Monitoring & Observability
18. Security & Anti-Abuse
19. Case Studies — Production Crawlers
20. Cost Estimation
21. Edge Cases
22. Interview Q&A
23. Conclusion

1. Introduction — What Is a Web Crawler?

A web crawler (also called a spider, bot, or web robot) is an automated program that systematically browses the World Wide Web, downloading pages and extracting links for further crawling. Google's web crawler processes over 100 billion pages per day, consuming 1 petabyte of data daily. Bing crawls approximately 10 billion pages. The Common Crawl dataset, which is publicly available, contains 3.1 billion web pages and uses 250 TB of storage.

Web crawlers are the foundation of search engines, but they serve many other purposes: monitoring website changes, competitive intelligence, price comparison, academic research, content aggregation, and training machine learning models. Building a web crawler at scale requires solving several distributed systems challenges: managing billions of URLs in a priority queue, politeness constraints (not overwhelming servers), duplicate detection, distributed coordination, and fault tolerance across thousands of machines.

2. Functional & Non-Functional Requirements

Functional Requirements

Non-Functional Requirements

3. Capacity Estimation & Back-of-Envelope

Daily Crawl Volume

Storage Estimates

Bandwidth Estimates

4. Data Model & Storage Schema

Entity Relationship

Cassandra Schema (URL Frontier — Write-Optimized)

CQL
CREATE TABLE url_frontier (
    crawl_group text,        -- Priority group: "high", "medium", "low"
    domain text,             -- For politeness grouping
    url text,
    priority int,            -- Within-group priority
    status text,             -- 'pending', 'crawling', 'completed', 'failed'
    discovered_at timestamp,
    last_crawled_at timestamp,
    next_crawl_at timestamp, -- When this URL should be crawled next
    crawl_count int,
    content_hash text,       -- For deduplication
    PRIMARY KEY (crawl_group, domain, url)
) WITH CLUSTERING ORDER BY (domain ASC, url ASC);

CREATE TABLE crawled_content (
    url_hash text,           -- SHA-256 of URL
    url text,
    domain text,
    crawled_at timestamp,
    http_status int,
    content_type text,
    content blob,            -- Raw HTML
    extracted_text text,     -- Cleaned text content
    links list<text>,        -- Extracted outbound links
    content_hash text,       -- For content dedup
    metadata map<text, text>,
    PRIMARY KEY (url_hash, crawled_at)
) WITH CLUSTERING ORDER BY (crawlled_at DESC)
  AND default_time_to_live = 7776000;  -- 90 days TTL

CREATE TABLE domain_state (
    domain text PRIMARY KEY,
    robots_txt blob,
    robots_disallow_paths list<text>,
    crawl_delay int,         -- seconds between requests
    last_crawl_at timestamp,
    pages_crawled_today int,
    daily_limit int,
    avg_response_time_ms int,
    error_rate float
);
            

Redis Structures

C#
// URL deduplication bloom filter (Redis module)
// FP rate: 0.1% with 10 bits per URL for 10 billion URLs
redis.call('BF.RESERVE', 'url_bloom', 0.001, 10000000000);
redis.call('BF.ADD', 'url_bloom', normalizedUrl);

// Domain crawl rate limiting
// Key: domain_rate:{domain}
// Sorted set with timestamps for sliding window
redis.call('ZADD', 'domain_rate:example.com', timestampMs, requestId);
redis.call('ZREMRANGEBYSCORE', 'domain_rate:example.com', 0, timestampMs - 1000);

// Crawl queue priorities (Redis Sorted Sets)
// Score = priority + time_decay
redis.call('ZADD', 'crawl_queue:high', score, url);
redis.call('ZPOPMIN', 'crawl_queue:high');
            

5. API Design

External API

HTTP
// Submit URLs for crawling
POST /api/v1/crawl/submit
{
    "urls": [
        "https://example.com/page1",
        "https://example.com/page2"
    ],
    "priority": "high",
    "callback_url": "https://your-app.com/webhook/crawl-complete",
    "metadata": {
        "campaign": "product_launch",
        "tag": "competitor_analysis"
    }
}

// Check crawl status
GET /api/v1/crawl/{job_id}/status

// Get crawled content
GET /api/v1/crawl/{job_id}/content?page=1&format=json

// List all crawl jobs
GET /api/v1/crawl/jobs?status=completed&limit=20

// Delete/cancel crawl job
DELETE /api/v1/crawl/{job_id}

// Get crawl statistics
GET /api/v1/stats/domains?top=100
            

Internal APIs

HTTP
// Worker fetches next URL from frontier
GET /internal/v1/frontier/next?worker_id=crawler-42&count=100

// Worker reports crawl result
POST /internal/v1/crawl/result
{
    "url": "https://example.com/page1",
    "worker_id": "crawler-42",
    "status": "success",
    "http_status": 200,
    "content_length": 45230,
    "content_hash": "sha256:abc123...",
    "links_found": [
        "https://example.com/page2",
        "https://other.com/page1"
    ],
    "crawl_time_ms": 1250
}

// DNS resolver service
POST /internal/v1/dns/resolve
{
    "domains": ["example.com", "other.com"]
}

// robots.txt fetcher
GET /internal/v1/robots/{domain}
            

6. High-Level Architecture

Component Responsibilities

7. URL Frontier & Priority Queue

Frontier Architecture

Priority Calculation

C#
public class UrlPriorityCalculator
{
    public int CalculatePriority(UrlEntry url, DomainState domain)
    {
        int score = 50; // Base score

        // PageRank factor (if known)
        if (url.PageRank.HasValue)
            score += (int)(url.PageRank.Value * 30); // 0-30 points

        // Freshness factor � how recently was it updated?
        if (url.LastCrawledAt.HasValue)
        {
            var age = DateTime.UtcNow - url.LastCrawledAt.Value;
            if (age < TimeSpan.FromHours(1))
                score += 20; // Very fresh, crawl again soon
            else if (age < TimeSpan.FromDays(1))
                score += 10; // Moderately fresh
            else if (age > TimeSpan.FromDays(30))
                score -= 10; // Stale, lower priority
        }

        // Domain authority
        if (domain.IsHighAuthority)
            score += 10;

        // Change frequency � pages that change often get higher priority
        if (url.ChangeFrequency == "daily")
            score += 15;
        else if (url.ChangeFrequency == "weekly")
            score += 5;

        // Link depth � shallower pages are higher priority
        score -= url.DepthFromSeed * 2;

        // Error penalty � URLs that recently failed get lower priority
        if (url.ConsecutiveFailures > 3)
            score -= 20;

        return Math.Clamp(score, 0, 100);
    }
}
            

Frontier Storage Design

8. Web Fetching Pipeline

Fetcher Implementation

C#
public class WebFetcher : BackgroundService
{
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly IFrontierService _frontier;
    private readonly IRobotsChecker _robotsChecker;
    private readonly IDnsCache _dnsCache;
    private readonly IContentStore _contentStore;
    private readonly ILinkExtractor _linkExtractor;
    private readonly IDeduplicator _deduplicator;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var urlEntry = await _frontier.GetNextUrlAsync(
                workerId: _workerId,
                batchSize: 10);

            foreach (var entry in urlEntry)
            {
                try
                {
                    await CrawlUrlAsync(entry, stoppingToken);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Failed to crawl {Url}", entry.Url);
                    await _frontier.ReportFailureAsync(entry);
                }
            }
        }
    }

    private async Task CrawlUrlAsync(UrlEntry entry, CancellationToken ct)
    {
        var domain = entry.Domain;

        // 1. Check robots.txt
        var robotsResult = await _robotsChecker.CheckAsync(domain, entry.Url);
        if (!robotsResult.Allowed)
        {
            _logger.LogInformation("Blocked by robots.txt: {Url}", entry.Url);
            await _frontier.SkipAsync(entry, "robots.txt disallowed");
            return;
        }

        // 2. Respect crawl delay
        if (robotsResult.CrawlDelay > 0)
        {
            await Task.Delay(TimeSpan.FromSeconds(robotsResult.CrawlDelay), ct);
        }

        // 3. Rate limit per domain
        if (!await _rateLimiter.AllowAsync(domain))
        {
            await _frontier.RequeueAsync(entry);
            return;
        }

        // 4. DNS resolution
        var ip = await _dnsCache.ResolveAsync(domain);

        // 5. Fetch the page
        var client = _httpClientFactory.CreateClient();
        client.DefaultRequestHeaders.Add("User-Agent", _config.CrawlerUserAgent);
        client.Timeout = TimeSpan.FromSeconds(30);

        var response = await client.GetAsync(entry.Url, ct);

        if (!response.IsSuccessStatusCode)
        {
            await _frontier.ReportFailureAsync(entry, response.StatusCode);
            return;
        }

        var html = await response.Content.ReadAsStringAsync(ct);
        var contentHash = ComputeHash(html);

        // 6. Content deduplication
        if (await _deduplicator.IsDuplicateContentAsync(contentHash))
        {
            _logger.LogInformation("Duplicate content: {Url}", entry.Url);
            await _frontier.SkipAsync(entry, "duplicate content");
            return;
        }

        // 7. Store content
        await _contentStore.StoreAsync(new CrawledPage
        {
            Url = entry.Url,
            Domain = domain,
            Content = html,
            ContentHash = contentHash,
            CrawledAt = DateTime.UtcNow,
            HttpStatusCode = (int)response.StatusCode,
            ContentLength = html.Length
        });

        // 8. Extract links
        var links = _linkExtractor.ExtractLinks(html, entry.Url);
        var newUrls = new List<string>();

        foreach (var link in links)
        {
            var normalized = NormalizeUrl(link, entry.Url);
            if (normalized != null && !await _deduplicator.IsDuplicateUrlAsync(normalized))
            {
                newUrls.Add(normalized);
            }
        }

        // 9. Submit new URLs to frontier
        if (newUrls.Any())
        {
            await _frontier.SubmitUrlsAsync(newUrls, discoveredFrom: entry.Url);
        }

        // 10. Report success
        await _frontier.ReportSuccessAsync(entry, linksFound: links.Count);
    }
}
            

9. URL Filtering & Deduplication

URL Normalization

Two-Level Deduplication

C#
public class TwoLevelDeduplicator
{
    private readonly IBloomFilter _bloomFilter;       // Level 1: Fast probabilistic check
    private readonly IContentHashStore _contentStore;  // Level 2: Exact check

    // Level 1: URL deduplication using bloom filter
    public async Task<bool> IsDuplicateUrlAsync(string normalizedUrl)
    {
        // Bloom filter: O(1) check, 0.1% false positive rate
        if (!_bloomFilter.MightContain(normalizedUrl))
        {
            return false; // Definitely not seen before
        }

        // False positive � confirm with exact check
        return await _contentStore.UrlExistsAsync(normalizedUrl);
    }

    // Level 2: Content deduplication using SimHash
    public async Task<bool> IsDuplicateContentAsync(string contentHash)
    {
        var simHash = ComputeSimHash(contentHash);

        // Find similar hashes within Hamming distance threshold
        var similarHashes = await _contentStore.FindSimilarHashesAsync(simHash, threshold: 3);

        if (similarHashes.Count == 0)
            return false;

        // For each similar hash, do full comparison
        foreach (var existingHash in similarHashes)
        {
            if (ComputeHammingDistance(simHash, existingHash) <= 3)
            {
                return true; // Content is similar enough to be a duplicate
            }
        }

        return false;
    }

    private ulong ComputeSimHash(string content)
    {
        var tokens = Tokenize(content);
        var vector = new int[64]; // 64-bit SimHash

        foreach (var token in tokens)
        {
            var hash = HashFunction(token);
            for (int i = 0; i < 64; i++)
            {
                vector[i] += ((hash >> i) & 1) == 1 ? 1 : -1;
            }
        }

        ulong simHash = 0;
        for (int i = 0; i < 64; i++)
        {
            if (vector[i] > 0)
                simHash |= (1UL << i);
        }

        return simHash;
    }
}
            

Deduplication Performance

10. Politeness & robots.txt

robots.txt Parsing

C#
public class RobotsTxtChecker
{
    private readonly IDistributedCache _cache;

    public async Task<RobotsResult> CheckAsync(string domain, string url)
    {
        var robots = await GetOrFetchRobotsTxtAsync(domain);

        if (robots == null)
            return new RobotsResult { Allowed = true }; // No robots.txt = everything allowed

        // Check if URL matches any Disallow pattern
        var path = new Uri(url).AbsolutePath;
        foreach (var disallowPath in robots.DisallowPaths)
        {
            if (path.StartsWith(disallowPath, StringComparison.OrdinalIgnoreCase))
            {
                return new RobotsResult
                {
                    Allowed = false,
                    Reason = $"Disallowed by robots.txt: {disallowPath}"
                };
            }
        }

        // Check crawl-delay
        var crawlDelay = robots.CrawlDelay ?? 1; // Default 1 second

        return new RobotsResult
        {
            Allowed = true,
            CrawlDelay = crawlDelay,
            SitemapUrl = robots.SitemapUrl
        };
    }

    private async Task<RobotsTxt> GetOrFetchRobotsTxtAsync(string domain)
    {
        var cacheKey = $"robots:{domain}";
        var cached = await _cache.GetAsync(cacheKey);

        if (cached != null)
            return JsonSerializer.Deserialize<RobotsTxt>(Encoding.UTF8.GetString(cached));

        // Fetch robots.txt
        try
        {
            var url = $"https://{domain}/robots.txt";
            var response = await _httpClient.GetAsync(url, TimeSpan.FromSeconds(5));

            if (response.IsSuccessStatusCode)
            {
                var content = await response.Content.ReadAsStringAsync();
                var parsed = ParseRobotsTxt(content);

                // Cache for 24 hours
                await _cache.SetAsync(cacheKey,
                    Encoding.UTF8.GetBytes(JsonSerializer.Serialize(parsed)),
                    new DistributedCacheEntryOptions
                    {
                        AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(24)
                    });

                return parsed;
            }
        }
        catch { }

        return null; // Fetch failed, assume allowed
    }
}
            

Politeness Rules

11. HTML Parsing & Content Extraction

Parsing Pipeline

C#
public class HtmlContentExtractor
{
    public ExtractedContent Extract(string html, string baseUrl)
    {
        var doc = new HtmlDocument();
        doc.LoadHtml(html);

        var content = new ExtractedContent
        {
            Title = ExtractTitle(doc),
            MetaDescription = ExtractMetaDescription(doc),
            TextContent = ExtractTextContent(doc),
            Links = ExtractLinks(doc, baseUrl),
            Images = ExtractImages(doc, baseUrl),
            Metadata = ExtractMetadata(doc),
            Language = DetectLanguage(doc),
            CanonicalUrl = ExtractCanonicalUrl(doc)
        };

        return content;
    }

    private List<string> ExtractLinks(HtmlDocument doc, string baseUrl)
    {
        var links = new List<string>();

        foreach (var anchor in doc.DocumentNode.SelectNodes("//a[@href]") ?? Enumerable.Empty<HtmlNode>())
        {
            var href = anchor.GetAttributeValue("href", "");
            if (string.IsNullOrEmpty(href)) continue;

            // Resolve relative URLs
            var absoluteUrl = new Uri(new Uri(baseUrl), href).ToString();

            // Filter out non-HTTP links (mailto:, javascript:, tel:)
            if (absoluteUrl.StartsWith("http://") || absoluteUrl.StartsWith("https://"))
            {
                links.Add(absoluteUrl);
            }
        }

        return links.Distinct().ToList();
    }

    private string ExtractTextContent(HtmlDocument doc)
    {
        // Remove script, style, nav, footer elements
        var nodesToRemove = doc.DocumentNode.SelectNodes(
            "//script|//style|//nav|//footer|//header|//aside|//noscript");

        if (nodesToRemove != null)
        {
            foreach (var node in nodesToRemove)
                node.Remove();
        }

        // Extract text from body
        var body = doc.DocumentNode.SelectSingleNode("//body");
        return body?.InnerText?.Trim() ?? "";
    }
}
            

Content Extraction Features

12. Distributed Architecture

Work Distribution Strategy

Domain-Based Sharding Implementation

C#
public class DomainShardRouter
{
    private readonly int _totalShards;
    private readonly int _shardId;

    public bool IsMyDomain(string domain)
    {
        var hash = HashFunction(domain);
        return (hash % _totalShards) == _shardId;
    }

    // Each worker processes URLs for its assigned domains
    public async Task ProcessUrlsAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            // Fetch URLs only for domains assigned to this shard
            var urls = await _frontier.GetUrlsForMyShardsAsync(
                _shardId, _totalShards, batchSize: 100);

            foreach (var url in urls)
            {
                if (!IsMyDomain(url.Domain))
                {
                    // Forward to correct shard
                    var targetShard = HashFunction(url.Domain) % _totalShards;
                    await _frontier.ForwardToShardAsync(url, targetShard);
                    continue;
                }

                await CrawlUrlAsync(url);
            }
        }
    }
}
            

13. Content Storage & Indexing

Storage Architecture

HDFS Storage Layout

Bash
# Daily crawl partition structure
/ crawl-data/
    / 2025/
        / 01/
            / 15/
                / html/
                    / part-00000.gz    # Compressed HTML pages
                    / part-00001.gz
                    / ...
                / metadata/
                    / part-00000.json  # URL metadata (status, hash, links)
                    / ...
                / extracted-text/
                    / part-00000.txt   # Cleaned text content
                    / ...
                / links/
                    / part-00000.csv   # Source URL ? Target URL mappings
                    / ...

# Storage per day (1 billion pages, 500KB each)
# Raw HTML: 500 TB/day ? compressed to ~100 TB/day (5:1 gzip ratio)
# Metadata: 50 TB/day ? compressed to ~5 TB/day
# Total: ~105 TB/day, ~38 PB/year
            

Storage Options Comparison

14. Re-crawling & Freshness

Freshness Strategy

C#
public class RecrawlScheduler
{
    public async Task<TimeSpan> CalculateRecrawlIntervalAsync(UrlEntry url, DomainState domain)
    {
        var baseInterval = url.ChangeFrequency switch
        {
            "always" => TimeSpan.FromMinutes(5),
            "hourly" => TimeSpan.FromHours(1),
            "daily" => TimeSpan.FromDays(1),
            "weekly" => TimeSpan.FromDays(7),
            "monthly" => TimeSpan.FromDays(30),
            _ => TimeSpan.FromDays(7) // Default: weekly
        };

        // Adjust based on observed change rate
        if (url.ChangeHistory != null && url.ChangeHistory.Count >= 3)
        {
            var recentChanges = url.ChangeHistory
                .Where(c => c > DateTime.UtcNow.AddDays(-30))
                .Count();

            if (recentChanges >= 10)
                baseInterval = TimeSpan.FromHours(1); // Changes frequently
            else if (recentChanges == 0)
                baseInterval = baseInterval.Add(TimeSpan.FromDays(7)); // Rarely changes
        }

        // Adjust based on page importance
        if (url.PageRank.HasValue && url.PageRank.Value > 0.8)
            baseInterval = TimeSpan.FromHours(Math.Max(1, baseInterval.TotalHours / 2));

        // Respect domain crawl limits
        if (domain.PagesCrawledToday >= domain.DailyLimit)
            baseInterval = baseInterval.Add(TimeSpan.FromHours(1));

        return baseInterval;
    }

    public async Task ScheduleRecrawlAsync(UrlEntry url)
    {
        var interval = await CalculateRecrawlIntervalAsync(url, url.DomainState);
        url.NextCrawlAt = DateTime.UtcNow.Add(interval);
        await _frontier.UpdateNextCrawlAsync(url);
    }
}
            

Freshness Metrics

15. DNS Resolution at Scale

DNS Caching Architecture

C#
public class DnsResolutionService
{
    private readonly IDistributedCache _redisCache;
    private readonly ConcurrentDictionary<string, DnsEntry> _localCache = new();

    public async Task<string> ResolveAsync(string domain)
    {
        // Level 1: Local in-memory cache (per worker)
        if (_localCache.TryGetValue(domain, out var localEntry) &&
            localEntry.ExpiresAt > DateTime.UtcNow)
        {
            return localEntry.IpAddress;
        }

        // Level 2: Redis distributed cache
        var cached = await _redisCache.GetStringAsync($"dns:{domain}");
        if (cached != null)
        {
            var entry = JsonSerializer.Deserialize<DnsEntry>(cached);
            _localCache[domain] = entry; // Populate local cache
            return entry.IpAddress;
        }

        // Level 3: Actual DNS resolution
        var addresses = await Dns.GetHostAddressesAsync(domain);
        var ip = addresses.FirstOrDefault()?.ToString();

        if (ip != null)
        {
            var entry = new DnsEntry
            {
                IpAddress = ip,
                ExpiresAt = DateTime.UtcNow.AddMinutes(5) // Short TTL for local
            };

            _localCache[domain] = entry;

            await _redisCache.SetStringAsync($"dns:{domain}",
                JsonSerializer.Serialize(entry),
                new DistributedCacheEntryOptions
                {
                    AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(1)
                });
        }

        return ip;
    }
}
            

DNS Caching Benefits

16. Error Handling & Resilience

Error Classification

Circuit Breaker per Domain

C#
public class DomainCircuitBreaker
{
    private readonly ConcurrentDictionary<string, CircuitState> _circuits = new();

    public async Task<bool> ShouldCrawlAsync(string domain)
    {
        var state = _circuits.GetOrAdd(domain, _ => new CircuitState());

        if (state.Status == CircuitStatus.Open)
        {
            if (DateTime.UtcNow - state.LastFailure > state.RecoveryTimeout)
            {
                state.Status = CircuitStatus.HalfOpen;
                return true; // Allow one test request
            }
            return false; // Circuit open, skip this domain
        }

        return true;
    }

    public void RecordResult(string domain, bool success)
    {
        var state = _circuits.GetOrAdd(domain, _ => new CircuitState());

        if (success)
        {
            state.FailureCount = 0;
            state.Status = CircuitStatus.Closed;
        }
        else
        {
            state.FailureCount++;
            state.LastFailure = DateTime.UtcNow;

            if (state.FailureCount >= 5)
            {
                state.Status = CircuitStatus.Open;
                state.RecoveryTimeout = TimeSpan.FromMinutes(
                    Math.Min(60, state.FailureCount * 5)); // 5-60 min backoff
            }
        }
    }
}
            

17. Monitoring & Observability

Key Metrics Dashboard

Alerting Rules

18. Security & Anti-Abuse

Security Measures

Responsible Crawling Practices

• Identify yourself: Use a descriptive User-Agent string with contact info
• Respect robots.txt: Always check and honor the directives
• Rate limit: No more than 1 request per second per domain
• Handle errors gracefully: Back off on 429/503, don't retry aggressively
• Provide opt-out: Honor removal requests promptly
• Don't circumvent protections: If a site blocks you, respect it
• Minimize impact: Cache robots.txt, DNS results, and avoid redundant requests

19. Case Studies � Production Crawlers

Google Crawler (Googlebot)

Common Crawl

Other Notable Crawlers

20. Cost Estimation

Monthly Cost (1B pages/day)

21. Edge Cases

22. Interview Q&A

23. Conclusion

Designing a web crawler at scale requires mastering several distributed systems challenges: managing billions of URLs in a priority queue, implementing polite crawling with per-domain rate limits, detecting duplicates efficiently across 10+ billion URLs, and handling the diverse landscape of web content (HTML, JavaScript, PDFs, images, redirects, errors).

The key insight for interviews is that a web crawler is fundamentally a work scheduling problem. You have a massive queue of URLs to process, constrained by politeness rules (rate limits, robots.txt), and you need to maximize the quality and freshness of the crawled content. The architecture should be designed around this core constraint: a distributed URL frontier feeding thousands of fetcher workers, with deduplication at the front and content storage at the back.

Remember that web crawlers have real-world impact. Over-crawling can overload servers, violate terms of service, and even lead to legal action. Always design for politeness, identify your crawler clearly, and respect the wishes of website operators. The best crawlers are invisible � they efficiently discover and re-crawl content without causing any disruption to the websites they visit.