Hidden Memory Growth in Go Slices

Some hidden mechanic in the 'append' function in Golang can lead to seemingly unexpected behavior. A slice may grow from 1MB to 32MB, and this can be missed by a programmer.

Growth Pattern

Let's look at a simple example that demonstrates this behavior:

primes := make([]int, 0, 1024*1024) // capacity = 1MB

for i := 0; i < 2*1024*1024; i++ {
    primes = append(primes, i)
    // Internal growth logic varies:
    // < 1024: doubles capacity
    // > 1024: complex growth pattern
}

The internal growth strategy in Go's runtime changes based on slice size. For small slices (less than 1024 elements), capacity doubles on each growth. For larger slices, the growth follows a more complex pattern balancing memory usage and performance.

Memory Analysis

Such memory allocation patterns can be easily observed using pprof:

import "runtime/pprof"

f, _ := os.Create("mem.prof")
pprof.WriteHeapProfile(f)
f.Close()

After running this code, use go tool pprof mem.prof to analyze the allocation patterns.

Small Buffer Optimization

For slices with fewer than 256 elements, small buffer optimization can keep data on stack:

type Buffer struct {
    buf    [256]byte    // small buffer on stack
    slice  []byte       // points to buf or heap
}

This approach avoids heap allocations for small slices, improving performance for common cases.