YURKOL Ltd - Custom Software and Cloud Architectural Solutions

Storage Selection Principles

When architecting cloud-native systems, storage selection should be driven by data access patterns, consistency requirements, and scaling characteristics. The well-established CAP theorem frames these decisions. In essence it goes that in the presence of network partitions, a system must choose between consistency and availability.

GCP's storage portfolio offers varying positions on this spectrum, with some services prioritizing consistency (Spanner), others availability (Firestore in certain modes), and many providing configurable trade-offs. These theoretical underpinnings have concrete implications for production systems.

Comparative Analysis

Analysis by Storage Type

Cloud SQL

Cloud SQL provides the familiar relational model with predictable performance characteristics. The service handles automated backups, replication, and failover, but retains fundamental scaling limitations of traditional RDBMS. Useful for systems with moderate transaction volumes and well-understood growth patterns. While read replicas can distribute query load, write scaling remains vertically constrained.

-- Cloud SQL supports standard SQL with extensions
CREATE TABLE clinical_observations (
    patient_id VARCHAR(64) NOT NULL,
    recorded_at TIMESTAMP NOT NULL,
    observation_code VARCHAR(20) NOT NULL,
    value NUMERIC,
    unit VARCHAR(10),
    PRIMARY KEY (patient_id, observation_code, recorded_at)
);

-- Typical query patterns work predictably
SELECT * FROM clinical_observations
WHERE patient_id = '12345'
AND recorded_at BETWEEN '2024-01-01' AND '2024-02-01'
ORDER BY recorded_at DESC;
                

Spanner

Spanner combines relational structure with horizontal scalability and strong consistency guarantees. It leverages Google's TrueTime API and Paxos algorithm to maintain external consistency across a distributed database. This architectural complexity translates to higher latency baselines and increased cost, justified primarily for systems where consistency violations would have significant business impact.

-- Spanner schemas use interleaving for related data
CREATE TABLE patients (
    patient_id STRING(64) NOT NULL,
    name STRING(MAX),
) PRIMARY KEY (patient_id);

CREATE TABLE observations (
    patient_id STRING(64) NOT NULL,
    timestamp TIMESTAMP NOT NULL,
    observation_type STRING(20) NOT NULL,
    value NUMERIC,
    unit STRING(10),
) PRIMARY KEY (patient_id, timestamp, observation_type),
  INTERLEAVE IN PARENT patients ON DELETE CASCADE;
                

Firestore

Firestore implements a document-oriented model with hierarchical collections and real-time capabilities. Its automatic scaling and offline-first design patterns accelerate development for client-heavy applications. However, complex reporting and analytics become challenging as data volumes grow, requiring careful planning of data structure and denormalization strategies.

// Firestore data models use hierarchical collections
// Avoid deeply nested structures for query efficiency

// Simple document retrieval (efficient)
patientRef := client.Collection("patients").Doc("patient123")
doc, err := patientRef.Get(ctx)
if err != nil {
    log.Fatalf("Failed to get patient: %v", err)
}

// Querying across collections (potential performance pitfalls)
startDate := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)
endDate := time.Date(2024, 1, 31, 23, 59, 59, 0, time.UTC)

observations := client.Collection("patients").Doc("patient123").
    Collection("observations").
    Where("timestamp", ">=", startDate).
    Where("timestamp", "<=", endDate).
    OrderBy("timestamp", firestore.Desc)

iter := observations.Documents(ctx)
defer iter.Stop()

// Processing results requires iterating through each document
for {
    doc, err := iter.Next()
    if err == iterator.Done {
        break
    }
    if err != nil {
        log.Fatalf("Failed to iterate: %v", err)
    }

    var observation Observation
    if err := doc.DataTo(&observation); err != nil {
        log.Printf("Warning: could not parse document: %v", err)
        continue
    }

    // Process each observation
}

// UPDATE operations are cumbersome with Go struct types
// Partial updates require map[string]interface{} instead of typed structs
// This creates disconnect between read and write models
_, err = patientRef.Update(ctx, []firestore.Update{
    {Path: "lastVisit", Value: time.Now()},
    {Path: "status", Value: "active"},
    {Path: "vitalSigns.bloodPressure", Value: "120/80"},
})
// Structured updates force you to choose between type safety and flexibility

BigQuery

BigQuery implements a column-oriented storage architecture optimized for analytical workloads. Its serverless nature allows scaling to petabytes without infrastructure management, but introduces unpredictable query costs based on data scanned. Best applied to analytical workloads where query patterns are well-understood and materialized views or partitioning strategies can be effectively employed.

-- BigQuery partition and cluster design significantly impacts performance
CREATE TABLE clinical_data.observations
PARTITION BY DATE(timestamp)
CLUSTER BY patient_id, observation_type
AS (
  SELECT * FROM source_data.raw_observations
);

-- Query cost is primarily determined by data scanned
SELECT
  patient_id,
  observation_type,
  AVG(value) as avg_value,
  COUNT(*) as observation_count
FROM clinical_data.observations
WHERE DATE(timestamp) BETWEEN '2024-01-01' AND '2024-01-31'
GROUP BY patient_id, observation_type;
                

Bigtable

Bigtable provides a sparse, distributed, persistent multi-dimensional map optimized for high-throughput time-series data. Its performance scales linearly with node count, but requires explicit capacity planning and row key design. This architectural complexity is justified for systems processing millions of data points per second where access patterns are well-defined and predictable.

// Row key design is critical for Bigtable performance
// For time-series data with device readings:
// rowKey = deviceId#reversedTimestamp

// Example row key construction
String deviceId = "device123";
long timestamp = System.currentTimeMillis();
String reversedTimestamp = Long.toString(Long.MAX_VALUE - timestamp);
String rowKey = deviceId + "#" + reversedTimestamp;

// Data model is sparse columnar
put.addColumn(
    Bytes.toBytes("metrics"),
    Bytes.toBytes("temperature"),
    Bytes.toBytes(98.6)
);
                

Cloud Storage

Cloud Storage (GCS) implements an immutable object store with strong consistency for individual objects. While primarily designed for unstructured data, it has proven effective as an intermediate layer in structured data workflows. The combination of notification triggers, access controls, and storage classes makes it particularly suitable for staged data processing and compliance-driven archives.

// GCS works well as a pipeline stage for ETL workflows
func ProcessNewDataFile(ctx context.Context, e GCSEvent) error {
    // Object is already created with strong consistency
    // Object name: e.Name, bucket: e.Bucket

    // Read the new object
    client, err := storage.NewClient(ctx)
    if err != nil {
        return fmt.Errorf("storage.NewClient: %v", err)
    }
    defer client.Close()

    rc, err := client.Bucket(e.Bucket).Object(e.Name).NewReader(ctx)
    if err != nil {
        return fmt.Errorf("Object(%q).NewReader: %v", e.Name, err)
    }
    defer rc.Close()

    // Process content and potentially write to another storage system
    // ...

    return nil
}
                

Memorystore

Memorystore (Redis) provides in-memory data structure storage with optional persistence. Its sub-millisecond latency and support for complex data structures make it ideal for caching, session management, and real-time operations. The service reduces operational overhead but requires careful capacity planning and eviction policy selection to avoid unexpected performance degradation under memory pressure.

// Redis works well for real-time session data and leaderboards
func GetUserSession(ctx context.Context, sessionID string) (*UserSession, error) {
    // Check cache first
    sessionData, err := redisClient.Get(ctx, "session:"+sessionID).Result()
    if err == nil {
        // Cache hit
        var session UserSession
        if err := json.Unmarshal([]byte(sessionData), &session); err != nil {
            return nil, err
        }
        return &session, nil
    }

    if err != redis.Nil {
        // Unexpected Redis error
        log.Printf("Redis error: %v", err)
    }

    // Cache miss, load from primary storage
    session, err := loadSessionFromDatabase(ctx, sessionID)
    if err != nil {
        return nil, err
    }

    // Update cache with expiration
    sessionJSON, _ := json.Marshal(session)
    redisClient.Set(ctx, "session:"+sessionID, sessionJSON, 30*time.Minute)

    return session, nil
}
                

Polyglot Persistence Patterns

Most sophisticated systems implement polyglot persistence—using multiple storage technologies suited to different aspects of the application. This approach optimizes for specific access patterns but introduces complexity in maintaining data consistency across boundaries. Effective implementations typically include:

• Transactional operations in a strongly consistent store (Cloud SQL/Spanner)
• Hot path reads served from in-memory caching layer (Memorystore)
• Analytics and reporting against a separate analytical store (BigQuery)
• Event sourcing or change data capture to maintain consistency boundaries
• Time-series telemetry in a specialized store (Bigtable)

The key architectural consideration is establishing clear boundaries between these systems, with well-defined consistency mechanisms for cross-database operations. This often involves implementing the Saga pattern for distributed transactions or leveraging eventual consistency with compensating actions.

Practical Selection Framework

Considerations:

• Query complexity and flexibility requirements
• Write throughput expectations and scaling patterns
• Consistency requirements for the business domain
• Latency sensitivity for primary access patterns
• Operational overhead and team expertise
• Cost predictability and scaling economics

This technical foundation should drive architecture decisions rather than trending technologies or marketing claims. The most elegant solution is often the simplest one that meets the actual requirements, not the most sophisticated option available.