1. Introduction: Why GraphQL Federation

Modern software architectures have shifted decisively toward microservices. Organizations that once served their entire application through a single monolithic backend now distribute responsibilities across dozens or even hundreds of independently deployable services. This decomposition brings tremendous benefits — independent deployability, technology heterogeneity, fault isolation, and team autonomy — but it also introduces a significant challenge: how do clients consume data that is spread across many services without coupling themselves to the internal topology of those services?

REST APIs have long been the default answer, but they impose constraints that become painful at scale. Clients must make multiple round-trip requests to assemble a single view of data. Over-fetching and under-fetching are endemic. API versioning becomes a maintenance nightmare as different consumers evolve at different paces. The backend-for-frontend (BFF) pattern emerged as a partial solution, but it tends to create new monoliths — a layer that must know about every downstream service and must be updated every time a service changes its contract.

GraphQL was introduced by Facebook in 2012 and open-sourced in 2015 as a response to exactly these problems. Instead of exposing rigid, resource-oriented endpoints, GraphQL exposes a single endpoint through which clients can precisely describe the shape of the data they need. The server resolves each field independently, allowing the client to request exactly the data it requires in a single network round trip. This solves over-fetching, under-fetching, and the chatty client problem in one stroke.

However, a single GraphQL server becomes a monolith in its own right as an organization grows. When every team must contribute resolvers to a single schema, coordination overhead explodes. Schema conflicts arise. Deployments become coupled. The benefits of microservices are lost. This is the problem that GraphQL Federation solves.

GraphQL Federation, introduced by Apollo in 2019 and matured significantly with Federation v2 in 2023, allows an organization to split a single, unified GraphQL schema across multiple independently deployable services called subgraphs. Each subgraph owns a portion of the overall schema, defines its own types and resolvers, and can be developed, tested, and deployed independently. A gateway — sometimes called a router — sits in front of these subgraphs and presents a single, coherent GraphQL API to clients. Clients remain entirely unaware of the fact that their queries are being resolved across multiple services.

The gateway performs several critical functions. It composes the individual subgraph schemas into a single, unified supergraph schema at startup (or on-demand via a schema registry). It plans the execution of incoming queries, determining which subgraphs must be consulted and in what order. It resolves entities — shared types that span multiple subgraphs — by fetching data from one subgraph and passing it to another for resolution of additional fields. It handles authentication, authorization, rate limiting, caching, and observability at the edge of the graph.

Federation v2 brought several important improvements over v1. It introduced a new @shareable directive that allows multiple subgraphs to resolve the same field, eliminating a common source of friction. It improved type merging semantics so that types can be extended across subgraphs with fewer restrictions. It added the @override directive for migrating ownership of fields between subgraphs without breaking clients. And it tightened up the composition algorithm to catch more errors at compose-time rather than at query-time.

The implications for system design are profound. A well-designed federation gateway becomes the single source of truth for an organization's API surface. It decouples client evolution from backend evolution. It enables domain-driven design at the API layer, where each bounded context maps to a subgraph. And it provides a platform for cross-cutting concerns — observability, security, traffic management — to be implemented once and applied uniformly.

This guide is aimed at senior and staff-level engineers who are responsible for designing, building, or operating a GraphQL Federation gateway. We will cover the full lifecycle: understanding the core GraphQL primitives, designing subgraph boundaries, composing schemas, planning query execution, resolving entities, preventing N+1 queries, handling subscriptions, implementing caching and auth, registering and versioning schemas, monitoring performance, testing in CI/CD, and migrating from a monolithic GraphQL server to a federated architecture. Along the way, we will provide concrete C# code examples, Mermaid diagrams, and comparison tables to ground every concept in practical reality.

Whether you are building a new federation gateway from scratch or evolving an existing one, this guide will give you the architectural vocabulary and implementation patterns you need to succeed. Let us begin with a review of GraphQL fundamentals before diving into the federation-specific material.

2. GraphQL Fundamentals Recap

Before diving into federation, it is essential to have a solid grounding in the core GraphQL primitives. GraphQL is a query language for APIs and a runtime for executing those queries against your data. Unlike REST, where the server determines the shape of the response, in GraphQL the client specifies exactly what data it needs, and the server returns data in that exact shape.

Schemas and Types

Every GraphQL API is defined by a schema. The schema is a strongly-typed contract that describes all the data a client can request and all the operations it can perform. The schema is written in the Schema Definition Language (SDL), which is independent of any programming language. A schema defines object types (the data entities), scalar types (the leaf values like Int, String, Boolean, Float, and ID, plus custom scalars), interface types, union types, input types, and enum types.

The schema also defines the entry points for queries (read operations), mutations (write operations), and subscriptions (real-time operations). Each entry point is a field on a root type — Query, Mutation, or Subscription. The type of each field determines what data can be returned, and the resolvers attached to each field determine how that data is fetched.

Resolvers

Resolvers are functions that fetch the data for each field in a schema. Every field in a GraphQL schema has a resolver. If no resolver is explicitly defined, GraphQL uses a default resolver that looks for a property with the same name on the parent object. Resolvers receive four arguments: the parent object (the result of the previous resolver), the arguments passed to the field, the context (shared across all resolvers in a single request), and the query information (AST, fragments, etc.).

Resolver design is critical to performance. A naive resolver that makes a database call for every field will result in N+1 query problems. A well-designed resolver uses batching (DataLoader) and caching to minimize the number of data fetches. In a federated architecture, resolvers in subgraphs must also conform to specific conventions — particularly around entity resolution — to enable the gateway to coordinate data fetching across services.

Subscriptions

GraphQL subscriptions enable real-time data delivery over WebSockets. While queries and mutations are request-response operations, subscriptions establish a persistent connection. The server pushes data to the client whenever a specific event occurs. Subscriptions are defined in the schema just like queries and mutations, but they map to an event source rather than a data source.

In a federated architecture, subscriptions present unique challenges because the gateway must maintain WebSocket connections to clients while routing subscription events from the appropriate subgraphs. Federation v2 includes mechanisms for subscription routing, but they require careful design to avoid excessive WebSocket connections and to ensure that events are delivered in the correct order.

Directives

GraphQL provides built-in directives like @deprecated and @specifiedBy (for custom scalars). Federation extends this with a rich set of directives: @key, @shareable, @external, @requires, @provides, @override, @inaccessible, and @tag. Each of these directives communicates something specific to the composition algorithm about how types and fields should be treated when merging subgraph schemas into a supergraph schema.

Understanding these primitives is non-negotiable before attempting to design a federation gateway. The federation layer does not replace these concepts — it builds on top of them. Every mistake in a federation architecture can usually be traced back to a misunderstanding of basic GraphQL semantics.

Here is a comparison of the core GraphQL operations:

With this foundation in place, we can now explore how Apollo Federation v2 builds on these primitives to enable a distributed, multi-team GraphQL architecture.

3. Apollo Federation v2 Architecture

Apollo Federation v2 is the current standard for building distributed GraphQL architectures. It introduces a composition model in which multiple independently developed and deployed subgraph schemas are merged into a single supergraph schema. The gateway (or router) executes client queries against this supergraph schema, decomposing each query into fetches directed at the appropriate subgraphs.

The Subgraph Contract

Each subgraph is an independent GraphQL service that owns a portion of the overall schema. Subgraphs use special federation directives to communicate ownership and sharing semantics to the composition algorithm. The key directives in Federation v2 are:

• @key(fields: "id") — Declares that a type is an entity and specifies the fields that form its primary key. Entities can be resolved across subgraphs using their key.
• @shareable — Indicates that a field can be resolved by multiple subgraphs. In v1, only one subgraph could resolve a given field; v2 relaxes this constraint.
• @external — Marks a field as defined in another subgraph. Used when a subgraph needs to reference a field from another subgraph in a @requires directive.
• @requires(fields: "...") — Specifies that a field depends on data that must be fetched from another subgraph first.
• @provides(fields: "...") — Indicates that a subgraph can provide certain fields of an entity when resolving it by key, even though those fields are not part of the entity's key.
• @override(from: "subgraph-name") — Migrates a field from one subgraph to another. During the transition period, the field is resolved by the new subgraph.
• @inaccessible — Hides a field or type from the composed supergraph schema. Useful for internal fields that should not be exposed to clients.
• @tag(name: "internal") — Tags a field for selective exposure. Combined with composition configuration, this allows different subgraphs of the schema to be exposed to different audiences.

The Composition Process

Composition is the process of merging subgraph schemas into a single supergraph schema. This can be done at build time (static composition) or at runtime (dynamic composition via a schema registry). The composition algorithm validates that the subgraph schemas are compatible — no conflicting type definitions, no missing entity keys, no circular dependencies — and produces a supergraph schema that the gateway uses for query planning.

If composition fails, the gateway cannot start (or cannot update its schema). This is by design: composition failures are caught early, before they can cause runtime errors. The schema registry (typically Apollo GraphOS or a self-hosted alternative) stores the composed supergraph schema and provides it to gateways on startup or on change notification.

The Gateway / Router

Apollo Router is the reference implementation of a federation gateway. It is written in Rust for performance and is designed to handle high-throughput, low-latency query execution. The gateway receives client queries, consults the supergraph schema to determine which subgraphs must be involved, builds a query plan (a DAG of fetches), and executes the plan. It then assembles the results from multiple subgraphs into a single response and returns it to the client.

The gateway is stateless and horizontally scalable. It does not store any data — it is purely a coordination layer. This makes it easy to deploy, scale, and operate. However, the gateway does maintain in-memory caches (for query plans, entity representations, and parsed queries) to avoid redundant computation.

The federation model enables true domain-driven API design. Each subgraph corresponds to a bounded context. The User service owns the User type and its core fields. The Order service owns the Order type. The Product service owns the Product type. Cross-cutting concerns — like adding a user's name to an order — are handled through entity resolution: the Order subgraph declares that it needs the User entity (keyed by ID), and the gateway fetches the User's name from the User subgraph when the client requests it.

This architecture scales organizationally as well as technically. Teams can evolve their subgraphs independently, deploy on their own cadences, and even use different programming languages and frameworks. The composition algorithm enforces correctness at the API boundary, so integration issues are caught at build time rather than in production.

4. System Architecture Overview

The following Mermaid diagram illustrates the high-level architecture of a GraphQL Federation Gateway system. The client sends a query to the gateway, which decomposes it into sub-queries directed at the appropriate subgraphs. Each subgraph resolves its portion of the data using its own data sources. The gateway assembles the results and returns them to the client.

As shown in the diagram, the gateway is the central coordination point. It does not own any data; its role is purely to receive queries, plan execution, fetch data from subgraphs, and assemble responses. The schema registry acts as the source of truth for the composed supergraph schema, ensuring that all gateway instances have a consistent view of the API surface.

Data Flow for a Typical Query

Consider a client query that requests a user's name, their recent orders, and the products in those orders. The gateway receives this query and consults the supergraph schema to determine that the User type is owned by the User subgraph, the Order type by the Order subgraph, and the Product type by the Product subgraph. The query planner generates a plan:

1. Fetch the user by ID from the User subgraph, returning the user's name and order IDs.
2. Fetch the orders by ID from the Order subgraph, returning the order details and product IDs.
3. Fetch the products by ID from the Product subgraph, returning the product names and prices.

Steps 2 and 3 can run in parallel because they depend on different data (order IDs vs. product IDs), but step 2 depends on step 1 (because we need the order IDs from the user). The query planner represents this as a directed acyclic graph (DAG) and executes it with maximum parallelism.

This sequence illustrates the power of federation: the client makes a single query, and the gateway transparently orchestrates three round-trips to three different services, each owned by a different team. The client has no knowledge of this orchestration.

Deployment Topology

In production, the gateway is typically deployed as a set of stateless containers behind a load balancer. Each gateway instance loads the supergraph schema from the schema registry on startup. When a subgraph publishes a new schema version, the schema registry notifies all gateway instances, which reload their schema and resume serving traffic with the updated API surface. This rolling update process ensures zero-downtime schema deployments.

The architecture is designed for organizational scale. With 20+ teams contributing to a single GraphQL API, the federation model provides clear ownership boundaries, independent deployability, and schema-level contracts that prevent teams from stepping on each other. The gateway ensures that clients always see a unified API, regardless of how many subgraphs back it.

5. Subgraph Design and Ownership

Designing subgraph boundaries is the most important architectural decision in a federated GraphQL system. Poorly chosen boundaries lead to excessive cross-subgraph dependencies, chatty entity resolution, and tight coupling between teams. Well-chosen boundaries align with domain-driven design principles and enable teams to move independently.

Principles of Subgraph Boundary Design

The primary principle is that each subgraph should own a coherent domain. A subgraph should correspond to a bounded context in DDD terms — a cohesive set of types, resolvers, and business logic that can be understood and evolved independently. A subgraph that owns the User type should also own all user-related fields, authentication logic, and user profile management. It should not own order-related logic, even though orders reference users.

The second principle is to minimize cross-subgraph dependencies. Every time one subgraph needs to reference a type owned by another subgraph, it incurs a dependency. These dependencies are not free — they create query plan complexity, increase latency (because additional network hops are required), and make schema changes harder to reason about. The goal is to design boundaries that minimize the number of entity resolutions required for typical client queries.

The third principle is to align team ownership with subgraph ownership. Each subgraph should be owned by a single team that has full authority over its schema, resolvers, data sources, and deployment pipeline. This alignment ensures that schema changes are made by the people who understand the domain best and who bear the consequences of those changes.

Entity Design in C#

Below is a C# example of an entity definition in a federated subgraph using Hot Chocolate (the leading .NET GraphQL server with federation support):

C#
using HotChocolate.Types;

[GraphQLKey("id")]
[GraphQLExtendObjectType("Order")]
public class OrderEntity
{
    [GraphQLKey("id")]
    public int Id { get; set; }

    // This field is resolved by the Order subgraph
    public string Status { get; set; } = string.Empty;

    public decimal TotalAmount { get; set; }

    // This field requires the userId from the User subgraph
    [GraphQLRequires("userId")]
    public async Task<UserEntity?> GetUser(
        int userId,
        [Service] IUserRepository userRepository)
    {
        return await userRepository.GetByIdAsync(userId);
    }

    public DateTime CreatedAt { get; set; }
}

In this example, the Order subgraph defines an OrderEntity that is keyed by ID. It owns the Status, TotalAmount, and CreatedAt fields. It declares a dependency on the User subgraph through the [GraphQLRequires] attribute, indicating that resolving the associated user requires the userId field to be available. The gateway will ensure that the userId is fetched (from the Order subgraph itself) before attempting to resolve the user from the User subgraph.

Anti-Patterns in Subgraph Design

The most common anti-pattern is the "God Subgraph" — a single subgraph that owns almost every type and field. This typically happens when an organization migrates from a monolithic GraphQL server and simply wraps the existing schema as a single subgraph. It defeats the entire purpose of federation because there is no domain decomposition, no team independence, and no reduction in coordination overhead.

The second anti-pattern is the "Entity soup" — when too many types are declared as entities and too many fields require cross-subgraph resolution. This leads to extremely complex query plans, high latency, and fragile schemas. The rule of thumb is that a typical client query should require no more than 2-3 entity resolutions. If a query requires 5 or more, the subgraph boundaries should be reconsidered.

The third anti-pattern is "shared ownership without sharing" — when two subgraphs define the same type or field without using the @shareable directive, leading to composition errors. If multiple subgraphs genuinely need to resolve the same field, use @shareable. If only one subgraph should resolve it, ensure that only one subgraph defines it.

Getting subgraph boundaries right is an iterative process. Start with a coarse decomposition based on your current domain model, monitor query performance and team friction, and adjust boundaries as you learn. Federation v2's @override directive makes it straightforward to migrate fields between subgraphs without breaking clients.

6. Schema Composition and Conflict Resolution

Schema composition is the process of merging multiple subgraph schemas into a single, unified supergraph schema. This is the step where errors in subgraph design are caught — before they reach production. The composition algorithm validates type compatibility, entity key consistency, directive usage, and field sharing semantics.

The Composition Algorithm

The composition algorithm operates in several phases. First, it collects all subgraph schemas and parses them into AST representations. Then, it validates each subgraph schema independently for correct directive usage and type definitions. Next, it merges types across subgraphs — for each type, it collects all fields defined across subgraphs and checks for conflicts. Two fields conflict if they have the same name but different types, or if they are not marked as @shareable but are defined in multiple subgraphs.

For entity types, the algorithm verifies that each entity has exactly one @key definition (or multiple key definitions if the entity is resolved by different keys in different subgraphs). It also verifies that entity fields referenced in @requires directives are actually available from other subgraphs.

Finally, the algorithm produces the supergraph schema — a single schema that includes all types and fields from all subgraphs, with federation-specific metadata (like which subgraph owns which field) embedded as extensions. The gateway uses this supergraph schema for query planning.

Conflict Resolution Strategies

When composition errors occur, the team must decide how to resolve them. The most common conflicts and their resolutions are:

1. Type Mismatch: Two subgraphs define the same field with different types. Resolution: Align the types, or use @inaccessible to hide one of the definitions.
2. Duplicate Non-Shareable Field: Two subgraphs define the same field without @shareable. Resolution: Assign ownership to one subgraph, or mark as @shareable if both subgraphs can resolve it.
3. Missing Entity Key: A subgraph references an entity but does not define its key. Resolution: Add the @key directive to the entity definition.
4. Circular Entity Dependencies: Subgraph A requires an entity from Subgraph B, which requires an entity from Subgraph A. Resolution: Redesign the entity boundaries to break the cycle.
5. Directive Misuse: A directive like @override references a subgraph that does not define the field. Resolution: Ensure the source subgraph defines the field.

Composition in Practice with Hot Chocolate

Below is a C# example showing how to configure federation composition in a Hot Chocolate subgraph:

C#
using HotChocolate;
using HotChocolate.Federation;

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddGraphQLServer()
            .AddQueryType<Query>()
            .AddType<OrderEntity>()
            .AddFederation(v =>
            {
                v.ComposeAs("order-service", "1.0.0");
            });
    }
}

public class Query
{
    [Query]
    public OrderEntity? GetOrder(int id, [Service] IOrderRepository repo)
    {
        return repo.GetById(id);
    }
}

public class FederationConfig
{
    public static void Configure(IObjectTypeDescriptor<OrderEntity> descriptor)
    {
        descriptor
            .Key(t => t.Id)
            .ResolveReference((ctx, key) =>
            {
                var repo = ctx.Service<IOrderRepository>();
                return repo.GetById(key.Id);
            });
    }
}

The Hot Chocolate framework handles federation directives through its attribute-based type system. The [GraphQLKey] attribute maps to the @key directive, [GraphQLShareable] maps to @shareable, and so on. The composition is performed by the Apollo Rover CLI or Apollo GraphOS when the subgraph schema is published.

Composition should be treated as a CI step. Every subgraph schema change should be validated against the full set of subgraph schemas before deployment. The Apollo Rover CLI or a custom composition script can be integrated into the CI pipeline to catch errors early. Schema composition failures should block deployment, not propagate to production.

7. Query Planning and Execution

Query planning is the brain of the federation gateway. When a client sends a query, the gateway must determine which subgraphs need to be consulted, in what order, and how to assemble the results. This is a non-trivial problem because queries can be deeply nested, can involve entity resolution across multiple subgraphs, and can have complex dependency chains.

How the Query Planner Works

The query planner takes two inputs: the client's query (as an AST) and the supergraph schema (which includes metadata about which subgraph owns which field). It produces a query plan — a directed acyclic graph (DAG) of fetch operations. Each node in the DAG represents a fetch to a specific subgraph, and edges represent data dependencies (the output of one fetch is needed as input to another).

The planner works by walking the query AST and, for each field, determining which subgraph resolves it. If consecutive fields in a query can be resolved by the same subgraph, they are grouped into a single fetch. When the planner encounters an entity field that crosses subgraph boundaries, it generates a dependent fetch — first fetch the entity's key from the current subgraph, then use that key to fetch additional fields from the target subgraph.

Fetch Strategies

The query planner supports several fetch strategies:

• Parallel Fetch: When multiple fetches are independent (no data dependencies between them), they can be executed in parallel. This is the default for top-level fields that belong to different subgraphs.
• Sequential Fetch: When one fetch depends on the output of another, they must be executed sequentially. This is typical for entity resolution chains.
• Batch Fetch: When multiple entities of the same type need to be resolved, the planner can batch them into a single fetch. This reduces the number of network round-trips and is critical for preventing N+1 problems.
• Deferred Fetch: Federation v2 introduces the @defer directive, which allows certain parts of a query to be resolved after the initial response is sent. This improves perceived latency for large, complex queries.

C# Query Planner Implementation

Below is a simplified C# implementation of a query planner for a federation gateway:

C#
public class QueryPlan
{
    public List<FetchNode> Nodes { get; set; } = new();
    public List<FetchEdge> Edges { get; set; } = new();
}

public class FetchNode
{
    public string Id { get; set; } = Guid.NewGuid().ToString("N");
    public string SubgraphName { get; set; } = string.Empty;
    public string Query { get; set; } = string.Empty;
    public FetchStrategy Strategy { get; set; } = FetchStrategy.Parallel;
}

public class FetchEdge
{
    public string FromNodeId { get; set; } = string.Empty;
    public string ToNodeId { get; set; } = string.Empty;
    public string DependencyPath { get; set; } = string.Empty;
}

public enum FetchStrategy
{
    Parallel,
    Sequential,
    Batch
}

public class QueryPlanner
{
    private readonly SupergraphSchema _schema;

    public QueryPlanner(SupergraphSchema schema)
    {
        _schema = schema;
    }

    public QueryPlan Plan(DocumentNode query)
    {
        var plan = new QueryPlan();
        var fieldGroups = GroupFieldsBySubgraph(query);

        FetchNode? previousNode = null;
        foreach (var group in fieldGroups)
        {
            var node = new FetchNode
            {
                SubgraphName = group.SubgraphName,
                Query = BuildSubquery(group.Fields),
                Strategy = group.RequiresExternalData
                    ? FetchStrategy.Sequential
                    : FetchStrategy.Parallel
            };
            plan.Nodes.Add(node);

            if (previousNode != null && group.RequiresExternalData)
            {
                plan.Edges.Add(new FetchEdge
                {
                    FromNodeId = previousNode.Id,
                    ToNodeId = node.Id,
                    DependencyPath = group.DependencyPath
                });
            }
            previousNode = node;
        }
        return plan;
    }

    private List<FieldGroup> GroupFieldsBySubgraph(DocumentNode query)
    {
        // Simplified: walk AST, resolve field owners, group by subgraph
        var groups = new List<FieldGroup>();
        // ... implementation details ...
        return groups;
    }

    private string BuildSubquery(List<FieldNode> fields)
    {
        // Build a valid GraphQL subquery from grouped fields
        // ... implementation details ...
        return string.Empty;
    }
}

This simplified planner demonstrates the core concepts: group fields by subgraph, determine dependencies, build a DAG, and execute with appropriate strategies. The actual Apollo Router planner is significantly more complex, handling edge cases like fragment spreads, inline fragments, conditional directives, and deferred fields.

Query plan caching is essential for performance. Since most queries are repeated (from mobile apps, SPAs, or cached clients), the gateway can cache the parsed query plan and reuse it for subsequent identical queries. Apollo Router caches query plans in memory with an LRU eviction policy, achieving sub-millisecond plan retrieval for cached queries.

8. Entity Resolution Across Subgraphs

Entity resolution is the mechanism by which the gateway fetches an entity from one subgraph and uses its key to fetch additional fields from another subgraph. This is the fundamental building block of federation — it allows types to span multiple subgraphs while presenting a unified view to clients.

How Entity Resolution Works

When a client query requests fields from an entity that span multiple subgraphs, the gateway performs entity resolution in multiple steps. First, it fetches the entity's key fields from the subgraph that owns the entity definition. Then, it passes the key values to the second subgraph, which resolves the additional fields. This process can chain across multiple subgraphs if the entity's fields are spread across three or more services.

The gateway communicates entity representations between subgraphs using a standardized format called the entity representation. An entity representation is a JSON object containing the entity's type name and key fields. For example, if the User entity is keyed by ID, an entity representation might look like { "__typename": "User", "id": 123 }. The second subgraph receives this representation and uses it to fetch the requested fields.

C# Entity Resolution Service

Below is a C# implementation of an entity resolution service that handles cross-subgraph entity fetching:

C#
public class EntityRepresentation
{
    public string TypeName { get; set; } = string.Empty;
    public Dictionary<string, object?> KeyFields { get; set; } = new();
}

public interface IEntityResolver
{
    Task<object?> ResolveEntityAsync(
        EntityRepresentation representation,
        List<string> requestedFields,
        CancellationToken cancellationToken);
}

public class FederationEntityResolver : IEntityResolver
{
    private readonly ISubgraphClientFactory _clientFactory;
    private readonly IEntityCache _cache;
    private readonly ILogger<FederationEntityResolver> _logger;

    public FederationEntityResolver(
        ISubgraphClientFactory clientFactory,
        IEntityCache cache,
        ILogger<FederationEntityResolver> logger)
    {
        _clientFactory = clientFactory;
        _cache = cache;
        _logger = logger;
    }

    public async Task<object?> ResolveEntityAsync(
        EntityRepresentation representation,
        List<string> requestedFields,
        CancellationToken cancellationToken)
    {
        var cacheKey = BuildCacheKey(representation, requestedFields);
        var cached = await _cache.GetAsync<object>(cacheKey, cancellationToken);
        if (cached != null) return cached;

        var ownerSubgraph = _clientFactory.GetSubgraphForEntity(
            representation.TypeName);

        var subquery = BuildEntitySubquery(
            representation.TypeName,
            representation.KeyFields,
            requestedFields);

        var result = await ownerSubgraph.ExecuteAsync(
            subquery, cancellationToken);

        if (result != null)
        {
            await _cache.SetAsync(cacheKey, result,
                TimeSpan.FromMinutes(5), cancellationToken);
        }

        _logger.LogInformation(
            "Resolved entity {Type} with key {Key} from {Subgraph}",
            representation.TypeName,
            string.Join(",", representation.KeyFields.Values),
            ownerSubgraph.Name);

        return result;
    }

    public async Task<List<object?>> ResolveEntitiesBatchAsync(
        string typeName,
        List<EntityRepresentation> representations,
        List<string> requestedFields,
        CancellationToken cancellationToken)
    {
        var results = new List<object?>(representations.Count);
        var batches = representations
            .GroupBy(r => _clientFactory.GetSubgraphForEntity(typeName).Name)
            .ToList();

        foreach (var batch in batches)
        {
            var subquery = BuildBatchEntitySubquery(
                typeName, batch.Select(r => r.KeyFields).ToList(),
                requestedFields);

            var subgraph = _clientFactory.GetSubgraphForEntity(typeName);
            var batchResults = await subgraph.ExecuteAsync(
                subquery, cancellationToken);

            results.AddRange(batchResults);
        }

        return results;
    }

    private string BuildCacheKey(
        EntityRepresentation rep, List<string> fields)
    {
        var keyParts = string.Join(":",
            rep.KeyFields.OrderBy(k => k.Key)
                .Select(k => $"{k.Key}={k.Value}"));
        return $"entity:{rep.TypeName}:{keyParts}:{string.Join(",", fields)}";
    }

    private string BuildEntitySubquery(
        string typeName,
        Dictionary<string, object?> keyFields,
        List<string> requestedFields)
    {
        var keyArgs = string.Join(", ",
            keyFields.Select(k => $"{k.Key}: {FormatValue(k.Value)}"));
        var fieldSelections = string.Join("\n  ", requestedFields);

        return $@"query {{
            _entities(representations: [{{ __typename: ""{typeName}"", {keyArgs} }}]) {{
                ... on {typeName} {{
                    {fieldSelections}
                }}
            }}
        }}";
    }

    private string BuildBatchEntitySubquery(
        string typeName,
        List<Dictionary<string, object?>> keyFieldsList,
        List<string> requestedFields)
    {
        var representations = string.Join(",\n      ",
            keyFieldsList.Select(kf =>
            {
                var args = string.Join(", ",
                    kf.Select(k => $"{k.Key}: {FormatValue(k.Value)}"));
                return $"{{ __typename: \"{typeName}\", {args} }}";
            }));
        var fieldSelections = string.Join("\n      ", requestedFields);

        return $@"query {{
            _entities(representations: [
      {representations}
    ]) {{
        ... on {typeName} {{
            {fieldSelections}}
        }}
    }}
}}";
    }

    private static string FormatValue(object? value) => value switch
    {
        string s => $"\"{s}\"",
        null => "null",
        _ => value.ToString() ?? "null"
    };
}

This implementation demonstrates several important patterns. First, it uses a cache to avoid redundant entity resolution for the same entity representation. Second, it batches entity resolution requests to the same subgraph, reducing network round-trips. Third, it constructs the _entities query that the gateway sends to subgraphs — this is a special root query that every federated subgraph must implement.

Entity resolution is the price of federation. The trade-off is organizational independence and domain isolation. The key to managing this trade-off is to design entity boundaries carefully, use batching and caching aggressively, and monitor entity resolution latency in production.

9. DataLoader and N+1 Prevention

The N+1 query problem is one of the most common performance issues in GraphQL. It occurs when resolving a list of entities causes one database query for the list and then one additional query for each entity in the list. In a federated architecture, the N+1 problem is amplified because each additional query may be a network hop to a different subgraph.

Understanding the N+1 Problem

Consider a query that fetches 50 orders and, for each order, fetches the associated user. Without batching, this results in 1 query to fetch the orders and 50 queries to fetch the users — a total of 51 queries. With DataLoader, this can be reduced to 2 queries: one to fetch the orders and one batched query to fetch all 50 users at once.

DataLoader in C# with Hot Chocolate

Hot Chocolate has built-in DataLoader support that integrates with its federation directives. Here is an example implementation:

C#
using GreenDonut;

public class UserBatchLoader : BatchLoader<int, UserDto>
{
    private readonly IDbConnectionFactory _connectionFactory;

    public UserBatchLoader(IDbConnectionFactory connectionFactory)
    {
        _connectionFactory = connectionFactory;
    }

    protected override async Task<IReadOnlyList<Result<UserDto>>> LoadBatchAsync(
        IReadOnlyList<int> keys,
        CancellationToken cancellationToken)
    {
        await using var connection = await _connectionFactory.OpenAsync();
        await using var command = connection.CreateCommand();
        command.CommandText = @"
            SELECT id, name, email, created_at
            FROM users
            WHERE id = ANY(@ids)";

        var parameter = command.CreateParameter();
        parameter.ParameterName = "@ids";
        parameter.Value = keys.ToArray();
        command.Parameters.Add(parameter);

        await using var reader = await command.ExecuteReaderAsync(cancellationToken);
        var userMap = new Dictionary<int, UserDto>();

        while (await reader.ReadAsync(cancellationToken))
        {
            var user = new UserDto
            {
                Id = reader.GetInt32(0),
                Name = reader.GetString(1),
                Email = reader.GetString(2),
                CreatedAt = reader.GetDateTime(3)
            };
            userMap[user.Id] = user;
        }

        return keys.Select(key =>
            userMap.TryGetValue(key, out var user)
                ? Result<UserDto>.Resolve(user)
                : Result<UserDto>.Resolve(null!)
        ).ToList();
    }
}

public class OrderResolver
{
    public async Task<UserDto?> GetUser(
        OrderEntity order,
        UserBatchLoader loader,
        CancellationToken cancellationToken)
    {
        return await loader.LoadAsync(order.UserId, cancellationToken);
    }
}

// For federation entity resolution
public class UserEntityResolver : IEntityResolver
{
    private readonly UserBatchLoader _batchLoader;

    public UserEntityResolver(UserBatchLoader batchLoader)
    {
        _batchLoader = batchLoader;
    }

    public async Task<UserDto?> ResolveByRepresentationAsync(
        EntityRepresentation representation,
        CancellationToken cancellationToken)
    {
        var userId = (int)representation.KeyFields["id"];
        return await _batchLoader.LoadAsync(userId, cancellationToken);
    }
}

The DataLoader collects all individual LoadAsync calls that occur during a single request, deduplicates them, and executes them as a single batch. The key insight is that the DataLoader is scoped to a single request — it collects loads during one request's execution and clears after the response is sent.

Server-Side DataLoader vs. Gateway-Level Batching

In a federated architecture, there are two levels at which batching can occur. The first is within a subgraph: when a subgraph resolves a list of entities, it can use DataLoader to batch database queries. The second is at the gateway level: when the gateway needs to resolve entities from a subgraph, it can batch multiple entity resolution requests into a single _entities query.

The most effective strategy is to batch at both levels. The gateway batches entity resolution requests into a single _entities query per subgraph. The subgraph's DataLoader then batches the resulting database queries. This two-level batching approach can reduce a query that would otherwise require hundreds of database calls down to a handful.

Common DataLoader Pitfalls

The most common mistake is using DataLoader with request-scoped state that leaks between requests. DataLoader instances must be created fresh for each request (typically via dependency injection with scoped lifetime). Sharing a DataLoader across requests causes batch contamination and stale data.

The second mistake is overusing DataLoader for fields that do not benefit from batching. If a field is always resolved individually (not as part of a list), DataLoader adds overhead without benefit. Use DataLoader specifically for batch-resolving lists of related entities.

The third mistake is ignoring error handling in batch loaders. When one item in a batch fails to load, the DataLoader must still return a result for every key in the batch. The convention is to return a null or default value for failed items, and to log the error for investigation. Failing to handle partial batch failures will cause the entire query to fail.

Proper DataLoader usage is essential for production-grade federation gateways. Without it, even simple queries will cause performance degradation as the number of entities grows. With it, the gateway can efficiently resolve complex queries involving hundreds of entities across multiple subgraphs with minimal latency overhead.

10. Subscription Federation and Real-Time Updates

GraphQL subscriptions enable real-time data delivery over persistent connections, typically WebSockets. In a federated architecture, subscriptions present unique challenges because the gateway must maintain WebSocket connections to clients while routing subscription events from the appropriate subgraphs. Federation v2 provides mechanisms for subscription routing, but they require careful design.

Subscription Routing Models

There are two primary models for routing subscriptions in a federation:

1. Direct Connection: The client establishes a WebSocket connection directly to the subgraph that owns the subscription. The gateway is not involved in the subscription data flow. This model is simpler but breaks the abstraction that the gateway provides — clients must know which subgraph to connect to.
2. Gateway Proxied: The client establishes a WebSocket connection to the gateway. The gateway forwards subscription events from the appropriate subgraph to the client. This model maintains the gateway abstraction but requires the gateway to manage additional WebSocket connections and forward events efficiently.

Federation v2 primarily supports the direct connection model, where each subgraph handles its own WebSocket connections. However, many organizations prefer the gateway-proxied model for operational simplicity — clients only need to know the gateway URL, and the gateway handles all routing.

C# Subscription Gateway Implementation

Below is a C# implementation of a gateway-proxied subscription handler that routes events from federated subgraphs to connected clients:

C#
using System.Collections.Concurrent;
using System.Net.WebSockets;
using System.Text.Json;

public class SubscriptionRouter : IHostedService, IDisposable
{
    private readonly ConcurrentDictionary<string, ClientSubscription> _subscriptions = new();
    private readonly ISubgraphEventSource _eventSource;
    private readonly ILogger<SubscriptionRouter> _logger;
    private CancellationTokenSource _cts = new();

    public SubscriptionRouter(
        ISubgraphEventSource eventSource,
        ILogger<SubscriptionRouter> logger)
    {
        _eventSource = eventSource;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        await foreach (var subgraphEvent in
            _eventSource.SubscribeAsync(_cts.Token))
        {
            var matchingSubscriptions = _subscriptions.Values
                .Where(s => s.SubscriptionType == subgraphEvent.TypeName
                    && s.Fields.Intersect(subgraphEvent.Fields).Any())
                .ToList();

            foreach (var subscription in matchingSubscriptions)
            {
                try
                {
                    await SendToClient(subscription, subgraphEvent);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex,
                        "Failed to send event to client {ClientId}",
                        subscription.ClientId);
                    _subscriptions.TryRemove(
                        subscription.ClientId, out _);
                }
            }
        }
    }

    public async Task RegisterSubscriptionAsync(
        string clientId,
        WebSocket socket,
        string subscriptionQuery,
        Dictionary<string, string?> variables,
        CancellationToken cancellationToken)
    {
        var parsedQuery = ParseSubscriptionQuery(subscriptionQuery);
        var subscription = new ClientSubscription
        {
            ClientId = clientId,
            Socket = socket,
            SubscriptionType = parsedQuery.TypeName,
            Fields = parsedQuery.Fields,
            Variables = variables,
            Query = subscriptionQuery
        };

        _subscriptions[clientId] = subscription;
        _logger.LogInformation(
            "Client {ClientId} subscribed to {Type}",
            clientId, parsedQuery.TypeName);

        // Keep the connection alive and handle unsubscribe
        var buffer = new byte[1024 * 4];
        while (socket.State == WebSocketState.Open)
        {
            var result = await socket.ReceiveAsync(
                new ArraySegment<byte>(buffer), cancellationToken);
            if (result.MessageType == WebSocketMessageType.Close)
            {
                _subscriptions.TryRemove(clientId, out _);
                await socket.CloseAsync(
                    WebSocketCloseStatus.NormalClosure,
                    "Client disconnected",
                    cancellationToken);
            }
        }
    }

    private async Task SendToClient(
        ClientSubscription subscription,
        SubgraphEvent subgraphEvent)
    {
        if (subscription.Socket.State != WebSocketState.Open) return;

        var payload = new
        {
            data = FilterPayload(subgraphEvent.Data, subscription.Fields)
        };
        var json = JsonSerializer.Serialize(payload);
        var bytes = System.Text.Encoding.UTF8.GetBytes(json);

        await subscription.Socket.SendAsync(
            new ArraySegment<byte>(bytes),
            WebSocketMessageType.Text,
            true,
            CancellationToken.None);
    }

    private object FilterPayload(
        object data, HashSet<string> requestedFields)
    {
        // Filter the event payload to only include requested fields
        // ... implementation details ...
        return data;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _cts.Cancel();
        return Task.CompletedTask;
    }

    public void Dispose()
    {
        _cts.Dispose();
    }
}

public class ClientSubscription
{
    public string ClientId { get; set; } = string.Empty;
    public WebSocket Socket { get; set; } = null!;
    public string SubscriptionType { get; set; } = string.Empty;
    public HashSet<string> Fields { get; set; } = new();
    public Dictionary<string, string?> Variables { get; set; } = new();
    public string Query { get; set; } = string.Empty;
}

public interface ISubgraphEventSource
{
    IAsyncEnumerable<SubgraphEvent> SubscribeAsync(
        CancellationToken cancellationToken);
}

public record SubgraphEvent(
    string TypeName,
    HashSet<string> Fields,
    object Data);

Subscription federation is one of the more complex aspects of a federated architecture. The key design decisions — direct vs. proxied, event sourcing, backpressure management — should be made early and revisited as the system scales. For most organizations, starting with a direct connection model (clients connect directly to subgraphs) and moving to a gateway-proxied model only when operational requirements demand it is the pragmatic approach.

11. Caching Strategies

Caching is critical for the performance and scalability of a GraphQL Federation gateway. Unlike REST APIs, which can leverage HTTP caching natively, GraphQL queries are typically sent as POST requests with a body, which bypasses most HTTP caching infrastructure. Federation gateways must implement caching at multiple levels to compensate.

Response Caching

Response caching stores the complete response for a specific query + variables combination. When the same query is received again, the cached response is returned without executing the query plan or fetching data from subgraphs. This is the highest-impact caching layer because it eliminates all downstream work.

Response caching can be implemented at the gateway level (in-memory LRU cache or distributed cache like Redis) or at the CDN level (using persisted queries with GET requests). Apollo Router supports both approaches: in-memory response caching for hot queries and CDN-based caching using persisted queries.

Entity Caching

Entity caching stores the resolved data for individual entities, keyed by their type and key fields. When the gateway needs to resolve an entity, it first checks the entity cache. If the entity is cached and still fresh, it returns the cached data without making a network call to the subgraph. This is particularly effective for entity resolution, where the same entity may be resolved across multiple queries.

C# Multi-Level Cache Implementation

Below is a C# implementation of a multi-level caching strategy for a federation gateway:

C#
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.Memory;

public class MultiLevelCache : IQueryCache
{
    private readonly IMemoryCache _l1Cache;
    private readonly IDistributedCache _l2Cache;
    private readonly TimeSpan _l1Ttl = TimeSpan.FromMinutes(5);
    private readonly TimeSpan _l2Ttl = TimeSpan.FromMinutes(30);
    private readonly ILogger<MultiLevelCache> _logger;

    public MultiLevelCache(
        IMemoryCache l1Cache,
        IDistributedCache l2Cache,
        ILogger<MultiLevelCache> logger)
    {
        _l1Cache = l1Cache;
        _l2Cache = l2Cache;
        _logger = logger;
    }

    public async Task<CachedResponse?> GetResponseAsync(
        string queryHash,
        string variablesHash,
        CancellationToken cancellationToken)
    {
        // L1: In-memory cache (fastest, per-instance)
        var l1Key = $"{queryHash}:{variablesHash}";
        if (_l1Cache.TryGetValue<CachedResponse>(l1Key, out var l1Result))
        {
            _logger.LogDebug("L1 cache hit for {Key}", l1Key);
            return l1Result;
        }

        // L2: Distributed cache (shared across instances)
        var l2Key = $"gql:resp:{queryHash}:{variablesHash}";
        var l2Bytes = await _l2Cache.GetAsync(l2Key, cancellationToken);
        if (l2Bytes != null)
        {
            var l2Result = JsonSerializer.Deserialize<CachedResponse>(l2Bytes);
            if (l2Result != null)
            {
                _logger.LogDebug("L2 cache hit for {Key}", l2Key);
                // Promote to L1
                _l1Cache.Set(l1Key, l2Result, _l1Ttl);
                return l2Result;
            }
        }

        _logger.LogDebug("Cache miss for {Key}", l1Key);
        return null;
    }

    public async Task SetResponseAsync(
        string queryHash,
        string variablesHash,
        CachedResponse response,
        CacheScope scope,
        CancellationToken cancellationToken)
    {
        var l1Key = $"{queryHash}:{variablesHash}";
        var l2Key = $"gql:resp:{queryHash}:{variablesHash}";

        // L1: Always set
        _l1Cache.Set(l1Key, response, _l1Ttl);

        // L2: Set based on scope
        if (scope == CacheScope.Shared)
        {
            var bytes = JsonSerializer.SerializeToUtf8Bytes(response);
            await _l2Cache.SetAsync(l2Key, bytes,
                new DistributedCacheEntryOptions
                {
                    AbsoluteExpirationRelativeToNow = _l2Ttl
                }, cancellationToken);
        }
    }

    public async Task<EntityData?> GetEntityAsync(
        string typeName,
        string keyHash,
        CancellationToken cancellationToken)
    {
        var l1Key = $"entity:{typeName}:{keyHash}";
        if (_l1Cache.TryGetValue<EntityData>(l1Key, out var entity))
        {
            return entity;
        }

        var l2Key = $"gql:entity:{typeName}:{keyHash}";
        var l2Bytes = await _l2Cache.GetAsync(l2Key, cancellationToken);
        if (l2Bytes != null)
        {
            var result = JsonSerializer.Deserialize<EntityData>(l2Bytes);
            if (result != null)
            {
                _l1Cache.Set(l1Key, result, _l1Ttl);
                return result;
            }
        }

        return null;
    }

    public async Task SetEntityAsync(
        string typeName,
        string keyHash,
        EntityData entity,
        CancellationToken cancellationToken)
    {
        var l1Key = $"entity:{typeName}:{keyHash}";
        var l2Key = $"gql:entity:{typeName}:{keyHash}";

        _l1Cache.Set(l1Key, entity, _l1Ttl);

        var bytes = JsonSerializer.SerializeToUtf8Bytes(entity);
        await _l2Cache.SetAsync(l2Key, bytes,
            new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow = _l2Ttl
            }, cancellationToken);
    }
}

public enum CacheScope { Instance, Shared }

public record CachedResponse(
    object Data,
    Dictionary<string, string> Extensions,
    DateTime CachedAt);

public record EntityData(
    string TypeName,
    Dictionary<string, object?> KeyFields,
    Dictionary<string, object?> Fields,
    DateTime CachedAt);

The most effective caching strategy combines all four levels. Hot queries are served from L1 (in-memory) with sub-millisecond latency. Warm queries are served from L2 (Redis) with a few milliseconds of latency. Public, immutable queries are served from L3 (CDN) at the edge. And slow-changing data is cached at L4 (subgraph level). This layered approach ensures that most queries are served from cache while maintaining data freshness for the queries that need it.

12. Authentication and Authorization in Federation

Authentication and authorization in a federated GraphQL architecture require coordination across the gateway and subgraphs. The gateway is typically the single entry point for clients, making it the natural place to validate authentication tokens. Subgraphs then handle authorization — determining whether the authenticated user has permission to access specific fields or entities.

Authentication at the Gateway

The gateway intercepts every incoming request and extracts the authentication token (typically a JWT or OAuth token) from the Authorization header. It validates the token's signature, checks expiration, and extracts claims. The claims are then propagated to subgraphs as context headers, so subgraphs can make authorization decisions without re-validating the token.

Authorization Directives

Federation supports custom authorization directives that can be applied at the field, type, or schema level. Hot Chocolate provides a rich authorization framework that integrates with ASP.NET Core's authorization policies:

C#
using HotChocolate.AspNetCore.Authorization;

public class Query
{
    [Query]
    [Authorize(Roles = new[] { "admin" })]
    public List<UserDto> GetAllUsers(
        [Service] IUserRepository repository)
    {
        return repository.GetAll();
    }

    [Query]
    public OrderDto? GetOrder(int id, [Service] IOrderRepository repo)
    {
        return repo.GetById(id);
    }
}

public class OrderDto
{
    public int Id { get; set; }
    public string Status { get; set; } = string.Empty;

    [Authorize(Policy = "OrderOwner")]
    public decimal TotalAmount { get; set; }

    public DateTime CreatedAt { get; set; }
}

public class OrderOwnerAuthorizationHandler
    : AuthorizationHandler<OrderOwnerRequirement, OrderDto>
{
    private readonly IHttpContextAccessor _httpContextAccessor;

    public OrderOwnerAuthorizationHandler(
        IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }

    protected override Task HandleRequirementAsync(
        AuthorizationHandlerContext context,
        OrderOwnerRequirement requirement,
        OrderDto resource)
    {
        var userId = _httpContextAccessor.HttpContext?.User?
            .FindFirst("sub")?.Value;

        // In federation, the userId comes from the gateway's
        // propagated claims
        if (userId != null)
        {
            context.Succeed(requirement);
        }

        return Task.CompletedTask;
    }
}

public class OrderOwnerRequirement : IAuthorizationRequirement { }

Cross-Subgraph Authorization

In a federated architecture, authorization decisions may span multiple subgraphs. For example, the Order subgraph might own the TotalAmount field but delegate the authorization check to the User subgraph (which knows whether the current user is the owner of the order). This cross-subgraph authorization pattern requires careful design to avoid circular dependencies and excessive network hops.

The recommended approach is to use the @requires directive combined with authorization headers. The gateway propagates the authenticated user's claims to every subgraph. Each subgraph uses these claims to make independent authorization decisions for the fields it owns. If a subgraph needs to make an authorization decision based on data from another subgraph, it can use the @provides directive to receive that data at entity resolution time.

Authorization in federation must be designed with the principle of least privilege in mind. Each subgraph should only have access to the data it needs to make its authorization decisions. The gateway should propagate only the claims that subgraphs require — not the full JWT payload. This limits the blast radius of a compromised subgraph and ensures that authorization decisions are made at the appropriate level of abstraction.

13. Schema Registry and Versioning

The schema registry is the central repository that stores all subgraph schemas and the composed supergraph schema. It is the source of truth for the API surface and the mechanism by which schema changes are validated, published, and propagated to gateways. A well-designed schema registry ensures that schema changes are safe, auditable, and reversible.

Core Responsibilities

The schema registry has four core responsibilities:

1. Schema Storage: Store every version of every subgraph schema, along with metadata (who published it, when, and why).
2. Composition: Compose subgraph schemas into a supergraph schema, validating for conflicts and errors.
3. Change Detection: Detect when subgraph schemas change and trigger composition and gateway reload.
4. Gateway Notification: Notify all gateway instances of schema changes so they can reload their supergraph schema.

Schema Versioning Strategy

Schema versioning in GraphQL federation follows a continuous evolution model rather than the discrete versioning model (v1, v2, v3) common in REST APIs. The rationale is that GraphQL schemas are designed to be backward-compatible by nature — fields can be added without breaking existing queries, and fields can be deprecated (with @deprecated) without removing them immediately.

The schema registry stores every published version of each subgraph schema. When a subgraph publishes a new schema version, the registry composes it with the current versions of all other subgraphs. If composition succeeds, the new supergraph schema is stored and gateways are notified. If composition fails, the registry rejects the change and provides detailed error messages to the developer.

C#
public class SchemaRegistryService
{
    private readonly ISchemaRepository _repository;
    private readonly ISchemaComposer _composer;
    private readonly IGatewayNotifier _notifier;
    private readonly ILogger<SchemaRegistryService> _logger;

    public SchemaRegistryService(
        ISchemaRepository repository,
        ISchemaComposer composer,
        IGatewayNotifier notifier,
        ILogger<SchemaRegistryService> logger)
    {
        _repository = repository;
        _composer = composer;
        _notifier = notifier;
        _logger = logger;
    }

    public async Task<SchemaPublishResult> PublishSchemaAsync(
        string subgraphName,
        string schemaSdl,
        string publishedBy,
        CancellationToken cancellationToken)
    {
        _logger.LogInformation(
            "Publishing schema for subgraph {Name} by {User}",
            subgraphName, publishedBy);

        // Store the subgraph schema version
        var version = await _repository.SaveSubgraphSchemaAsync(
            subgraphName, schemaSdl, publishedBy, cancellationToken);

        // Get all current subgraph schemas
        var allSchemas = await _repository
            .GetAllSubgraphSchemasAsync(cancellationToken);

        // Attempt composition
        var compositionResult = await _composer.ComposeAsync(
            allSchemas, cancellationToken);

        if (!compositionResult.IsSuccess)
        {
            _logger.LogWarning(
                "Composition failed for {Name}: {Errors}",
                subgraphName,
                string.Join("; ", compositionResult.Errors));

            return new SchemaPublishResult
            {
                Success = false,
                Errors = compositionResult.Errors,
                SupergraphSchema = null
            };
        }

        // Store the composed supergraph schema
        await _repository.SaveSupergraphSchemaAsync(
            compositionResult.SupergraphSchema,
            compositionResult.CompositionMetadata,
            cancellationToken);

        // Notify all gateway instances
        await _notifier.NotifySchemaChangeAsync(
            compositionResult.SupergraphSchema,
            cancellationToken);

        _logger.LogInformation(
            "Schema published and composition successful for {Name}",
            subgraphName);

        return new SchemaPublishResult
        {
            Success = true,
            SupergraphSchema = compositionResult.SupergraphSchema,
            CompositionMetadata = compositionResult.CompositionMetadata
        };
    }

    public async Task<SchemaCheckResult> CheckSchemaAsync(
        string subgraphName,
        string schemaSdl,
        CancellationToken cancellationToken)
    {
        var allSchemas = await _repository
            .GetAllSubgraphSchemasAsync(cancellationToken);
        allSchemas[subgraphName] = schemaSdl;

        var result = await _composer.ComposeAsync(
            allSchemas, cancellationToken);

        return new SchemaCheckResult
        {
            IsValid = result.IsSuccess,
            Errors = result.Errors,
            Warnings = result.Warnings
        };
    }
}

The schema registry is a critical piece of infrastructure for any federation deployment. It enforces schema quality, prevents breaking changes, and provides an audit trail for all schema modifications. For small teams, Apollo GraphOS provides a managed solution. For larger organizations with compliance requirements or air-gapped environments, a self-hosted registry provides full control over schema management.

14. Performance Monitoring and Query Complexity Analysis

Monitoring a federated GraphQL gateway requires visibility into multiple layers: client query patterns, gateway performance (query planning, execution, caching), subgraph performance (resolver latency, database queries), and end-to-end latency. Without comprehensive observability, performance issues are difficult to diagnose and optimize.

Key Metrics to Monitor

The following metrics are essential for monitoring a federation gateway:

• Query Latency (p50, p95, p99): End-to-end latency from the client's perspective.
• Query Plan Duration: Time spent building the query plan (should be <1ms for cached plans).
• Subgraph Fetch Latency: Time spent fetching data from each subgraph.
• Entity Resolution Count: Number of entity resolution hops per query.
• Cache Hit Rate: Percentage of queries served from cache (response and entity).
• Error Rate: Percentage of queries that result in errors.
• Query Complexity Score: Computed complexity of incoming queries.
• Throughput: Queries per second.

Query Complexity Analysis

Query complexity analysis is a defense mechanism against expensive queries. GraphQL allows clients to request arbitrarily nested and broad queries. Without complexity limits, a single malicious or accidental query can consume all available resources. The query planner assigns a complexity score to each query based on field costs, list multipliers, and depth. Queries exceeding a threshold are rejected before execution.

C#
public class QueryComplexityAnalyzer
{
    private readonly int _maxComplexity;
    private readonly int _maxDepth;
    private readonly int _maxAliases;
    private readonly ILogger<QueryComplexityAnalyzer> _logger;

    public QueryComplexityAnalyzer(
        int maxComplexity = 1000,
        int maxDepth = 15,
        int maxAliases = 50,
        ILogger<QueryComplexityAnalyzer> logger)
    {
        _maxComplexity = maxComplexity;
        _maxDepth = maxDepth;
        _maxAliases = maxAliases;
        _logger = logger;
    }

    public ComplexityResult Analyze(DocumentNode query)
    {
        var result = new ComplexityResult();
        var context = new AnalysisContext();

        foreach (var definition in query.Definitions.OfType<OperationDefinitionNode>())
        {
            AnalyzeSelectionSet(definition.SelectionSet, context, 1);
        }

        result.TotalComplexity = context.CurrentComplexity;
        result.MaxDepth = context.MaxDepthFound;
        result.AliasCount = context.AliasCount;

        if (result.TotalComplexity > _maxComplexity)
        {
            result.IsValid = false;
            result.Errors.Add(
                $"Query complexity {result.TotalComplexity} exceeds maximum {_maxComplexity}");
        }

        if (result.MaxDepth > _maxDepth)
        {
            result.IsValid = false;
            result.Errors.Add(
                $"Query depth {result.MaxDepth} exceeds maximum {_maxDepth}");
        }

        if (result.AliasCount > _maxAliases)
        {
            result.IsValid = false;
            result.Errors.Add(
                $"Alias count {result.AliasCount} exceeds maximum {_maxAliases}");
        }

        _logger.LogInformation(
            "Query complexity: {Complexity}, depth: {Depth}, aliases: {Aliases}, valid: {IsValid}",
            result.TotalComplexity, result.MaxDepth, result.AliasCount, result.IsValid);

        return result;
    }

    private void AnalyzeSelectionSet(
        SelectionSetNode selectionSet,
        AnalysisContext context,
        int depthMultiplier)
    {
        context.Depth++;
        if (context.Depth > context.MaxDepthFound)
            context.MaxDepthFound = context.Depth;

        foreach (var selection in selectionSet.Selections)
        {
            switch (selection)
            {
                case FieldNode field:
                    var fieldCost = GetFieldCost(field.Name.Value);
                    context.CurrentComplexity += fieldCost * depthMultiplier;
                    context.AliasCount += field.Alias != null ? 1 : 0;

                    if (field.SelectionSet != null)
                    {
                        var childMultiplier = GetListMultiplier(field.Name.Value);
                        AnalyzeSelectionSet(
                            field.SelectionSet, context,
                            depthMultiplier * childMultiplier);
                    }
                    break;

                case InlineFragmentNode inline:
                    AnalyzeSelectionSet(
                        inline.SelectionSet, context, depthMultiplier);
                    break;

                case FragmentSpreadNode spread:
                    // Resolve fragment and analyze its selection set
                    break;
            }
        }

        context.Depth--;
    }

    private int GetFieldCost(string fieldName)
    {
        // Fields that require expensive operations have higher cost
        return fieldName switch
        {
            "users" => 20,
            "orders" => 15,
            "products" => 10,
            _ => 1
        };
    }

    private int GetListMultiplier(string fieldName)
    {
        // Fields that return lists multiply the cost of their children
        return fieldName switch
        {
            "users" => 50,
            "orders" => 20,
            "products" => 30,
            "items" => 10,
            _ => 1
        };
    }
}

public class AnalysisContext
{
    public int CurrentComplexity { get; set; }
    public int Depth { get; set; }
    public int MaxDepthFound { get; set; }
    public int AliasCount { get; set; }
}

public class ComplexityResult
{
    public bool IsValid { get; set; } = true;
    public int TotalComplexity { get; set; }
    public int MaxDepth { get; set; }
    public int AliasCount { get; set; }
    public List<string> Errors { get; set; } = new();
    public List<string> Warnings { get; set; } = new();
}

Monitoring should be implemented from day one, not retrofitted after production issues arise. The federation gateway is the ideal place to collect metrics because it sees every client request. Use OpenTelemetry for distributed tracing across gateway and subgraphs, Prometheus for metrics collection, and Grafana for dashboards. Set up alerts for latency spikes, error rate increases, and cache hit rate drops.

15. Testing and CI/CD for Federated Schemas

Testing a federated GraphQL architecture requires a multi-layered approach. Individual subgraphs can be tested independently using standard unit and integration tests. The federation layer itself — composition, query planning, entity resolution — requires additional testing to ensure that the composed schema works correctly and that queries are resolved as expected across subgraph boundaries.

Testing Layers

CI/CD Pipeline

The CI/CD pipeline for a federated GraphQL system should include composition validation as a mandatory step. Every subgraph schema change must be validated against the full set of subgraph schemas before deployment. The pipeline should also run subgraph-specific tests, integration tests, and optionally E2E federation tests.

C# Integration Test Example

Below is a C# example of an integration test that validates entity resolution across subgraphs in a federation setup:

C#
using Xunit;
using GraphQL.Client.Http;
using GraphQL.Client.Serializer.SystemTextJson;

public class FederationIntegrationTests : IAsyncLifetime
{
    private GraphQLHttpClient _gatewayClient = null!;
    private TestServer _userSubgraph = null!;
    private TestServer _orderSubgraph = null!;

    public async Task InitializeAsync()
    {
        // Start test subgraphs
        _userSubgraph = await StartUserSubgraphAsync();
        _orderSubgraph = await StartOrderSubgraphAsync();

        // Start gateway pointing to test subgraphs
        var gateway = await StartGatewayAsync(
            _userSubgraph, _orderSubgraph);
        _gatewayClient = new GraphQLHttpClient(
            new Uri("http://localhost:5000/graphql"),
            new SystemTextJsonSerializer());
    }

    [Fact]
    public async Task Query_CrossSubgraphEntityResolution_ReturnsCompleteData()
    {
        var query = @"
        query {
            user(id: 1) {
                name
                email
                orders {
                    id
                    status
                    totalAmount
                }
            }
        }";

        var response = await _gatewayClient.SendQueryAsync<UserWithOrders>(
            new GraphQLRequest(query));

        Assert.Null(response.Errors);
        Assert.Equal("Alice", response.Data.User.Name);
        Assert.Equal(3, response.Data.User.Orders.Count);
        Assert.All(response.Data.User.Orders, order =>
        {
            Assert.False(string.IsNullOrEmpty(order.Status));
            Assert.True(order.TotalAmount > 0);
        });
    }

    [Fact]
    public async Task Query_NestedEntityResolution_ResolvesCorrectly()
    {
        var query = @"
        query {
            orders(userId: 1) {
                id
                user {
                    name
                    email
                }
                items {
                    product {
                        name
                        price
                    }
                    quantity
                }
            }
        }";

        var response = await _gatewayClient.SendQueryAsync<OrdersWithDetails>(
            new GraphQLRequest(query));

        Assert.Null(response.Errors);
        Assert.NotEmpty(response.Data.Orders);

        foreach (var order in response.Data.Orders)
        {
            Assert.NotNull(order.User);
            Assert.NotEmpty(order.Items);
            foreach (var item in order.Items)
            {
                Assert.NotNull(item.Product);
                Assert.True(item.Product.Price > 0);
            }
        }
    }

    [Fact]
    public async Task Query_EntityResolution_FallsBackGracefullyOnSubgraphError()
    {
        // Stop the user subgraph to simulate failure
        await _userSubgraph.StopAsync();

        var query = @"
        query {
            user(id: 1) {
                name
                orders {
                    id
                    status
                }
            }
        }";

        var response = await _gatewayClient.SendQueryAsync<UserWithOrders>(
            new GraphQLRequest(query));

        // Gateway should return partial data with error for failed subgraph
        Assert.NotNull(response.Errors);
        Assert.Contains(response.Errors, e =>
            e.Message.Contains("user"));
    }

    public async Task DisposeAsync()
    {
        _gatewayClient.Dispose();
        if (_userSubgraph != null) await _userSubgraph.DisposeAsync();
        if (_orderSubgraph != null) await _orderSubgraph.DisposeAsync();
    }
}

// DTOs for test assertions
public class UserWithOrders
{
    public UserData User { get; set; } = null!;
}

public class UserData
{
    public string Name { get; set; } = string.Empty;
    public string Email { get; set; } = string.Empty;
    public List<OrderData> Orders { get; set; } = new();
}

public class OrderData
{
    public int Id { get; set; }
    public string Status { get; set; } = string.Empty;
    public decimal TotalAmount { get; set; }
    public UserData? User { get; set; }
    public List<OrderItemData> Items { get; set; } = new();
}

public class OrderItemData
{
    public ProductData Product { get; set; } = null!;
    public int Quantity { get; set; }
}

public class ProductData
{
    public string Name { get; set; } = string.Empty;
    public decimal Price { get; set; }
}

public class OrdersWithDetails
{
    public List<OrderData> Orders { get; set; } = new();
}

The CI/CD pipeline should treat composition validation as a gate — if composition fails, the deployment is blocked. This ensures that broken schemas never reach production. Additionally, schema compatibility checks should be run to detect breaking changes (removing required fields, changing field types, etc.) before they are deployed.

Test data management is an important consideration for E2E federation tests. Each test should set up its own data state to ensure reproducibility. Use test databases (or in-memory alternatives) that are reset between test runs. Avoid sharing test data across tests — each test should be independent and idempotent.

16. Migration from Monolith GraphQL to Federation

Migrating from a monolithic GraphQL server to a federated architecture is a common journey for growing organizations. The monolith served well in the early days — a single schema, a single server, a single team. But as the organization grows, the monolith becomes a bottleneck: schema conflicts, coupled deployments, and coordination overhead across teams. Federation offers a path out, but the migration must be carefully planned and executed.

Migration Strategy: The Strangler Fig Pattern

The safest migration strategy is the strangler fig pattern: gradually extract domains from the monolith into subgraphs while the monolith continues to serve traffic. The gateway sits in front of both the monolith and the new subgraphs, routing queries to the appropriate backend. Over time, more and more of the monolith's functionality is extracted into subgraphs until the monolith can be retired entirely.

Step-by-Step Migration

The migration proceeds in four phases:

1. Phase 1 — Introduce the Gateway: Deploy a federation gateway in front of the monolith. The monolith becomes a subgraph (the "legacy" subgraph). Clients are migrated to point at the gateway instead of the monolith directly. At this point, there is no change in functionality — the gateway simply proxies to the monolith.
2. Phase 2 — Extract the First Subgraph: Choose a domain to extract — typically the most independently owned domain with the least cross-cutting dependencies. Create a new subgraph for this domain, implement its resolvers, and register it with the schema registry. The gateway now routes queries for this domain to the new subgraph and routes everything else to the monolith. Use the @override directive to migrate fields from the monolith to the new subgraph without breaking clients.
3. Phase 3 — Extract Remaining Subgraphs: Repeat Phase 2 for each remaining domain. Each extraction should be validated with composition checks, integration tests, and canary deployments. As more domains are extracted, the monolith handles fewer and fewer queries.
4. Phase 4 — Retire the Monolith: Once all domains have been extracted, the monolith has no remaining functionality and can be retired. Remove it from the schema registry and decommission its infrastructure.

C#
// Migration configuration for strangler fig pattern
public class MigrationConfiguration
{
    public List<DomainMigration> Migrations { get; set; } = new();
}

public class DomainMigration
{
    public string DomainName { get; set; } = string.Empty;
    public MigrationPhase Phase { get; set; }
    public List<string> FieldsToMigrate { get; set; } = new();
    public string? TargetSubgraph { get; set; }
    public DateTime? PlannedDate { get; set; }
    public DateTime? CompletedDate { get; set; }
}

public enum MigrationPhase
{
    Planned,
    InProgress,
    Validating,
    Completed,
    RolledBack
}

public class MigrationExecutor
{
    private readonly ISchemaRegistryClient _registry;
    private readonly IRouterConfigClient _router;
    private readonly ILogger<MigrationExecutor> _logger;

    public MigrationExecutor(
        ISchemaRegistryClient registry,
        IRouterConfigClient router,
        ILogger<MigrationExecutor> logger)
    {
        _registry = registry;
        _router = router;
        _logger = logger;
    }

    public async Task ExecuteMigrationAsync(
        DomainMigration migration,
        CancellationToken cancellationToken)
    {
        _logger.LogInformation(
            "Starting migration for domain {Domain} to {Subgraph}",
            migration.DomainName, migration.TargetSubgraph);

        // Step 1: Add @override directives to monolith schema
        await _registry.AddOverrideDirectivesAsync(
            "monolith",
            migration.FieldsToMigrate,
            migration.TargetSubgraph!,
            cancellationToken);

        // Step 2: Publish updated monolith schema
        await _registry.PublishSchemaAsync(
            "monolith",
            await _registry.GetCurrentSchemaAsync("monolith", cancellationToken),
            cancellationToken);

        // Step 3: Publish new subgraph schema
        await _registry.PublishSchemaAsync(
            migration.TargetSubgraph!,
            await _registry.GetNewSubgraphSchemaAsync(
                migration.TargetSubgraph!, cancellationToken),
            cancellationToken);

        // Step 4: Verify composition succeeds
        var compositionResult = await _registry.CheckCompositionAsync(
            cancellationToken);
        if (!compositionResult.IsValid)
        {
            _logger.LogError(
                "Composition failed: {Errors}",
                string.Join(", ", compositionResult.Errors));
            throw new InvalidOperationException("Composition failed");
        }

        // Step 5: Enable traffic splitting (canary)
        await _router.SetTrafficSplitAsync(
            migration.DomainName,
            migration.TargetSubgraph!,
            percentage: 10, // Start with 10% traffic
            cancellationToken);

        _logger.LogInformation(
            "Migration for {Domain} started with 10% canary traffic",
            migration.DomainName);
    }

    public async Task PromoteMigrationAsync(
        DomainMigration migration,
        CancellationToken cancellationToken)
    {
        // Increase traffic to 100%
        await _router.SetTrafficSplitAsync(
            migration.DomainName,
            migration.TargetSubgraph!,
            percentage: 100,
            cancellationToken);

        // Remove override directives from monolith
        await _registry.RemoveOverrideDirectivesAsync(
            "monolith",
            migration.FieldsToMigrate,
            cancellationToken);

        migration.CompletedDate = DateTime.UtcNow;
        migration.Phase = MigrationPhase.Completed;

        _logger.LogInformation(
            "Migration for {Domain} completed",
            migration.DomainName);
    }
}

The strangler fig pattern minimizes risk by ensuring that at every step, the system is functional and clients are unaffected. The gateway provides a clean abstraction layer that decouples the migration process from client code. The key to a successful migration is starting with the least complex domain, validating the process, and then applying the same pattern to increasingly complex domains.

17. Interview Q&A

Q1: What problem does GraphQL Federation solve?

A: GraphQL Federation solves the problem of scaling a single GraphQL schema across multiple teams and services. Without federation, a single GraphQL server becomes a monolith: every team must contribute to the same schema, deployments are coupled, and coordination overhead grows quadratically with the number of teams. Federation allows each team to own a portion of the schema (a subgraph), deploy independently, and use their preferred technology stack. A gateway composes the subgraph schemas into a unified API and handles query planning, entity resolution, and cross-cutting concerns. This gives clients a single, coherent API while giving teams the independence of microservices.

Q2: How does entity resolution work across subgraphs?

A: Entity resolution is the process by which the gateway fetches an entity from one subgraph and uses its key to fetch additional fields from another subgraph. When a client queries fields that span multiple subgraphs, the gateway first fetches the entity's key fields from the owning subgraph. It then constructs an _entities query with the entity representation (type name + key fields) and sends it to the target subgraph. The target subgraph resolves the requested fields using its own resolvers and data sources. This process can chain across multiple subgraphs. The gateway batches entity resolution requests when possible to minimize network round-trips.

Q3: What is the difference between @shareable and @external?

A: The @shareable directive indicates that a field can be resolved by multiple subgraphs. When two or more subgraphs define the same field on the same type, @shareable tells the composition algorithm that this is intentional and not a conflict. The @external directive marks a field as defined in another subgraph. It is used when a subgraph needs to reference a field from another subgraph (e.g., in a @requires directive) but does not resolve that field itself. The key difference is that @shareable means "I resolve this field," while @external means "someone else resolves this field, but I need to reference it."

Q4: How do you prevent N+1 queries in a federated architecture?

A: N+1 prevention in federation requires batching at two levels. At the subgraph level, use DataLoader to batch database queries when resolving lists of related entities. DataLoader collects individual load requests that occur during a single request execution, deduplicates them, and executes them as a single batch query. At the gateway level, batch entity resolution requests into a single _entities query per subgraph. Instead of making one entity resolution request per entity, the gateway collects all entities of the same type that need resolution from the same subgraph and sends them in a single batch. The combination of subgraph-level DataLoader and gateway-level batching reduces what could be hundreds of database queries to a handful.

Q5: How do you handle schema composition errors in CI/CD?

A: Schema composition should be a mandatory CI step. Every subgraph schema change should be validated against the full set of subgraph schemas before deployment. The CI pipeline should run the composition tool (Apollo Rover or equivalent) and fail the build if composition fails. Additionally, the pipeline should run schema compatibility checks to detect breaking changes (removing fields, changing types, removing required arguments). Schema checks should be run on pull requests so that composition errors are caught before code is merged. The schema registry provides a pre-publish check API that can be called from CI to validate a proposed schema change without actually publishing it.

Q6: How do you handle cross-subgraph authorization?

A: The recommended approach is a hybrid model where the gateway validates the authentication token (JWT) and extracts user claims, then propagates these claims to subgraphs as context headers. Each subgraph uses these claims to make independent authorization decisions for the fields it owns. For fields that require authorization based on data from another subgraph, the @provides directive can supply the necessary data at entity resolution time. This approach keeps authorization close to the data, avoids circular dependencies, and ensures that each subgraph has the context it needs to make correct authorization decisions.

Q7: What are the trade-offs of using a federation gateway?

A: The primary trade-offs are: (1) Latency — the gateway adds an extra network hop between clients and subgraphs, though this is typically minimal (<5ms) with a well-optimized gateway. (2) Complexity — the gateway introduces a new infrastructure component that must be deployed, monitored, and maintained. (3) Entity resolution overhead — queries that span multiple subgraphs require entity resolution, which adds latency and complexity compared to a monolith that can resolve everything locally. (4) Schema governance overhead — composition and schema registry processes add ceremony to schema changes. The benefits — team independence, independent deployability, domain-driven API design, and centralized observability — generally outweigh these trade-offs for organizations with multiple teams contributing to a single API.

Q8: How do you migrate from a monolith GraphQL server to federation?

A: Use the strangler fig pattern. First, deploy a gateway in front of the monolith (making the monolith a subgraph). Then, incrementally extract domains into new subgraphs. For each extraction, use the @override directive to migrate fields from the monolith to the new subgraph without breaking clients. Use canary deployments to gradually shift traffic to the new subgraph. Once a domain is fully migrated and validated, remove the override directives from the monolith. Repeat for each domain until the monolith has no remaining functionality and can be retired. Throughout the migration, the gateway ensures that clients continue to see a unified API regardless of which backend serves each field.

Q9: How do you handle caching in a federation gateway?

A: Caching in federation requires a multi-layer approach. At the gateway level, use response caching (in-memory LRU + distributed Redis) to cache complete query responses for repeated queries. At the entity level, cache entity representations keyed by type and key fields to avoid redundant entity resolution. At the CDN level, use persisted queries with GET requests to enable HTTP caching for public, immutable queries. At the subgraph level, each subgraph can implement its own caching strategy (database query caching, response caching) based on the data's freshness requirements. The key insight is that different data has different freshness requirements — user profiles change rarely and can be cached aggressively, while order statuses change frequently and should be cached with short TTLs or not at all.

Q10: What metrics should you monitor in a federation gateway?

A: The essential metrics are: (1) Query latency percentiles (p50, p95, p99) — end-to-end latency from the client's perspective. (2) Query plan duration — time to build the query plan (should be <1ms for cached plans). (3) Subgraph fetch latency — time spent fetching data from each subgraph (helps identify slow subgraphs). (4) Entity resolution count — number of cross-subgraph entity resolutions per query (indicates schema boundary quality). (5) Cache hit rate — percentage of queries served from cache. (6) Error rate — percentage of queries resulting in errors. (7) Query complexity violations — number of queries rejected for exceeding complexity limits. (8) Throughput — queries per second. Use OpenTelemetry for distributed tracing, Prometheus for metrics, and Grafana for dashboards. Set up alerts for latency spikes, error rate increases, and cache hit rate drops.