Varnish memory usage

[Michael Lenaghan]

Thanks for the reply, Ken.

>>   966403669          .            .   SMA outstanding bytes
>
> There's your 1GB cache buffer, which is what you're setting.

OK, I see that "SMA bytes allocated" - "SMA bytes free" = "SMA
outstanding bytes". Perhaps, in the process of allocating and freeing,
memory has become fragmented--leaving unusable holes...?

> Linux (including 10.04) sets a default stack size of 8MB:
>
> -s: stack size (kbytes)        8192
>
> With 500 threads, you could in theory see up to 4GB of VIRT.

Varnish sets the stack size to 65K on this system by default:

thread_pool_stack          65536 [bytes]
                           Default is -1
                           Worker thread stack size.

That works out to 32 MB for 500 threads. And I force the creation of
all threads up front in order to avoid surprise memory allocations
under load.

> Run "pmap -x PID" on the child process (or grandchildren), and you will see where the memory goes.
> It should reflect the above -- you'll see 500+ 8MB stacks, plus a lot of other 1MB allocations, which
> is the block size used by jemalloc IIRC.

Using pmap I can confirm the 65 KB thread stacks. After pushing
Varnish a little further--it's at 1.6 GB virtual memory now--I also
see one line like this:

52f00000 1432576       -       -       - rw---    [ anon ]

That's a single 1.4 GB allocation?

The next biggest allocations (in address order) are:

ac3f6000   10240       -       -       - rw---    [ anon ]
acdf7000   10240       -       -       - rw---    [ anon ]
ad7f8000   10240       -       -       - rw---    [ anon ]
ae200000   12288       -       -       - rw---    [ anon ]
afaf0000   10304       -       -       - rw---    [ anon ]
b05ed000   10240       -       -       - rw---    [ anon ]
b0fee000   10240       -       -       - rw---    [ anon ]
b19ef000   10240       -       -       - rw---    [ anon ]
b23ef000   81988       -       -       - rw-s-  _.vsl

Those ten allocations account for nearly all of the memory used by
Varnish. (The first line alone is just under 90%.)

> If you really only want to use 1G, specify something like "malloc,800M".

Yes, I can try that. And in production I think I can set things up so
that I guarantee Varnish never fills its cache. But I wanted to see
how Varnish performed at the limit. And I'd like to get as close to
the limit as I safely can.