Thread pools

[Poul-Henning Kamp]

In message <294d5daa0908171755y44f5c132o587f3c8188493a4 at mail.gmail.com>, Mark M
oseley writes:
>I've seen various things in the wiki and threads on this list talking
>about thread pools. In general, the advice is typically conservative,
>i.e. don't use more than the default 2 thread pools unless you have
>to. I've also seen the occasional comment suggesting one run as many
>thread pools as there are cores/cpus.

I think the point here is "don't make 1000 or even 100 pools".

One pool per core should be all you need to practically eliminate
thread contention, but to truly realize this, we would have to pin
pools on cores and other nasty and often backfiring "optimizations".

Having a few too many pools probably does not hurt too much, but
may increase the thread create/kill ratio a bit.

>Also, the wiki mentions that it's mainly appropriate when you run into
>locks tying things up. Is that mainly a case of high LRU turnover or
>are there other scenarios where locking is an issue? What are the
>symptoms of locking becoming an issue with the current configuration
>and what fields in varnishstat should I be looking at?

POSIX unfortunately does not offer any standard tools for analysing
lock behaviour, so we have had to make some pretty crude ones
ourselves.

The main sign of lock issues is that the number of context switches
increase drastically, you OS can probably give you a view of that
number.

If you want to go deeper, we have a flag in diag_bitmaps that enables
shmlogging of lock contentions (or even all lock operations), together
with varnishtop suitably filtered, that gives a good idea which locks
we have trouble with.

>but I worry about wasting any CPU% on those 8 core 1950s [...]

Varnish is all about wasting CPU%, normally we barely touch the
CPUs, man systems running with 80-90% idle CPUs.

Have you played a bit with varnihshist ?   I suspect that may be
the most sensitive, if crude, indicator of overall performance
we can offer.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.