Fallback Strategy, Observability & Operational Hardening in High-Throughput Systems

In Part 1, we explored architectural abstraction and failure-rate-based circuit isolation using Polly.

But circuit breaking alone does not make a system resilient.

Opening the circuit simply stops calling Redis.

The real question is:

What happens next?

In high-throughput healthcare platforms — where ontology lookups, terminology resolution, and patient context queries are frequent and latency-sensitive — a cache failure must not translate into a service outage.

That requires:

• Transparent read-through fallback
• Intelligent cache repopulation
• Read replica support
• Full observability
• Careful operational tuning

This article dives into those mechanics.

1. Read-Through Caching with Cosmos DB Fallback

The goal is simple:

If Redis fails or misses, the system should:

1. Query the primary data store (Cosmos DB)
2. Return the correct data
3. Repopulate the cache
4. Remain transparent to the caller

The calling microservice should not need to know whether:

• Redis returned the data
• The circuit was open
• Cosmos DB served the request

That abstraction lives inside RedisCacheClient.

Core Read-Through Implementation

public async Task<T> GetData<T>(

    string key,

    Func<Task<T>> dataFunction,

    int  expireTimeInHour = 24,

    bool isReset          = false,

    bool readReplica      = false)

{

    if (!isReset)

    {

        var data = await GetDataAsync<T>(key, readReplica);

        if (!CheckNullOrEmpty(data))

        {

            logger.LogInformation("Cache HIT for key: {Key}", key);

            return data;

        }

    }

    logger.LogInformation("Cache MISS for key: {Key} -- querying Cosmos DB", key);

    var result = await dataFunction();

    await Task.Run(() => SetData(key, result, expireTimeInHour));

    return result;

}

Key behaviors:

• If cache hit → return immediately.
• If cache miss or circuit open → execute fallback function.
• Automatically repopulate Redis.
• Return consistent type T.

This is a clean read-through pattern.

2. Controlled Cache Reset (isReset Flag)

There are scenarios where underlying data changes.

Instead of manually clearing keys across services, the API allows:

NamedIndividual_TQ namedIndividualTQ = await redisCacheClient.GetData(

    key:          cacheKey,

    dataFunction: () => cosmosRepository.GetNamedIndividualAsync(id),

    isReset:      true);

When isReset = true:

• Skip Redis lookup
• Force fresh Cosmos DB read
• Refresh cache entry

This ensures:

• Consistency during updates
• Minimal code duplication
• Controlled cache invalidation

In healthcare systems, data correctness can outweigh performance.

This flag provides explicit control.

3. Replica Routing for Read Scaling

Azure Cache for Redis supports read replicas.

For read-heavy workloads (e.g., ontology lookups), routing to replicas prevents primary saturation.

Implementation:

private T ExecuteGetData<T>(string key, bool readReplica = false)

{

    var db   = GetDataBase();

    var flag = readReplica ? CommandFlags.PreferReplica : CommandFlags.None;

    var value = db.StringGet(key, flag);

    return value.HasValue

        ? helper.Deserialize<T>(value.ToString())

        : default(T);

}

With:

await redisCacheClient.GetDataAsync<NamedIndividual_TQ>(key, useReplica: true);

This allows horizontal read scaling without changing business logic.

Architecturally, this prevents a read-heavy service from overwhelming the primary Redis node.

4. What Happens During a Redis Outage?

Let’s simulate.

1. Redis becomes unstable.
2. Circuit breaker opens.
3. GetDataAsync returns default(T).
4. Read-through logic calls dataFunction().
5. Cosmos DB serves the request.
6. Cache repopulates once Redis stabilizes.

System behavior:

• Latency increases.
• No crash.
• No retry storm.
• No cascading failure.

This is controlled degradation.

However, this introduces a new risk:

Fallback amplification.

5. Fallback Amplification & Thundering Herd Risk

If Redis fails during high traffic:

• All requests shift to Cosmos DB.
• RU consumption spikes.
• Latency increases.
• Throttling may occur.

Additionally:

If a popular key expires under high concurrency:

• Thousands of requests may simultaneously call dataFunction().

This is the thundering herd problem.

While this design protects Redis isolation, it assumes:

Cosmos DB can absorb fallback load.

In moderate SaaS scale, this is acceptable.

At hyperscale, you would add:

• Request coalescing
• Distributed locking
• Concurrency limits
• Rate limiting

Understanding this boundary is part of architectural maturity.

6. Application Insights Telemetry

Resilience without observability is incomplete.

Every Redis operation is tracked as a dependency:

private T Execute<T>(Func<T> redisOperation, string operationName)

{

    var operation = telemetry.StartOperation<DependencyTelemetry>(operationName);

    operation.Telemetry.Type   = "Azure Redis";

    operation.Telemetry.Target = connectionString;

    try

    {

        var result = redisOperation();

        operation.Telemetry.Success = true;

        return result;

    }

    catch (Exception ex)

    {

        operation.Telemetry.Success = false;

        operation.Telemetry.Properties["Error"] = ex.Message;

        throw;

    }

    finally

    {

        telemetry.StopOperation(operation);

    }

}

Additionally:

• Circuit OPEN events → Warning logs
• HALF-OPEN events → Informational logs
• RESET events → Recovery logs

This enables:

• Monitoring fallback frequency
• Detecting Redis instability
• Measuring degradation windows
• Aligning with SLOs

In distributed healthcare platforms, auditability and visibility are operational requirements.

7. Configuration-Driven Operational Tuning

All resilience parameters are externalized via appsettings.json.

Example:

{

  "Redis": {

    "UseCircuitBreaker": true,

    "CircuitBreaker": {

      "TimeoutMs": 3000,

      "FailureThreshold": 0.5,

      "SamplingDurationSeconds": 30,

      "MinimumThroughput": 100,

      "DurationOfBreakSeconds": 30

    }

  }

}

This allows:

• Runtime threshold adjustments
• Faster tuning during incidents
• No redeployment required

In production systems, configuration agility reduces MTTR.

8. Handling StackExchange.Redis 2.10 Changes

Version 2.10 removed nullable TimeSpan? support for expiry in StringSet.

Old approach (now invalid):

db.StringSet(key, value, expiry: TimeSpan.FromHours(24));

Updated implementation:

db.StringSet(key, serialisedValue);

if (expiry.HasValue)

    db.KeyExpire(key, expiry.Value);

Async variant:

await db.StringSetAsync(key, serialisedValue);

if (expiry.HasValue)

    await db.KeyExpireAsync(key, expiry.Value);

This preserves backward compatibility and avoids runtime errors after package upgrades.

Operational detail matters.

9. Redis List Operations

Beyond key-value strings, list operations are supported:

• ListRightPush
• ListRange
• ListRemove

All follow the same:

• Telemetry tracking
• Circuit breaker enforcement
• Timeout policies

Consistency across API surface ensures no hidden unprotected paths.

10. Real-World Failure Behavior

Under normal load:

• High cache hit ratio
• Low Cosmos traffic
• Minimal latency

During Redis degradation:

• Circuit opens
• Cosmos traffic increases
• Latency increases modestly
• No service crash

Under extreme traffic spike + Redis outage:

• Cosmos RU spike
• Possible throttling
• Increased latency
• Still no cascading failure

The system favors:

Predictable degradation over unpredictable collapse.

11. Business Context: Why This Matters

In healthcare platforms:

• Ontology resolution cannot block workflows.
• Clinical context queries must not crash APIs.
• Terminology services cannot trigger cascading failure.

This design ensures:

Redis instability = performance event Not = availability event

That distinction protects operational continuity.

Key Takeaways

This implementation demonstrates:

• Transparent read-through fallback
• Clean cache invalidation strategy
• Read replica routing
• Observability-first design
• Configuration-driven resilience
• Explicit awareness of fallback amplification risk

Circuit isolation prevents failure from spreading.

Fallback ensures continuity.

Telemetry ensures visibility.

Together, they form a resilient, production-grade caching strategy suitable for high-throughput, healthcare-grade distributed systems.