Tuning varnish for high load

[Poul-Henning Kamp]

In message <86db848d0802291223n2400e8c5ya4ef0e648f4d124b at mail.gmail.com>, "Mich
ael S. Fischer" writes:

>Funny you should ask, I've been spending a lot of time with Varnish in
>the lab.  Here are a few observations I've made:

>- When the cache hit ratio is very high (i.e. 100%), we discovered
>that Varnish's default configuration of thread_pool_max is too high.

That's unexpected for me, I wouldn't expect many threads to be
created in this case.

You don't say which version of varnish you are running, but there
have been some serious changes in -trunk recently, so if you 
can check if this still happens with -trunk, that would help me
understand the problem better.

Do notice, that if you have a high hit ratio under high traffic,
adding more thread pools is recommended to lower the mutex
congestion.

In -trunk you can set the diag_bitmap paramter to 0x10 and run
	varnishtop -i debug -I MTX
to get a view of contests mutexes, output from that would be
welcome.

>When there are too many worker threads, Varnish spends an inordinate
>amount of time in system call space.

Yes, having more workerthreads than you need is not recommended,
it just gives the scheduler too much work.

>I find this interesting, because one would think that Varnish would be
>making the system spend much more time in the scheduler in the second
>scenario because it is doing significantly less work (no lookups, just
>handing off connections to the appropriate backend).

Connections are not "handed off", the data transferred between the
two connections by vanrish.

>Finally: Varnish performance is absolutely atrocious on a 8-CPU RHEL
>4.5 system -- so bad that I have to turn down thread_pool_max to 4 or
>restrict it to run only on 4 CPUs via taskset(1).  I've heard that
>MySQL has similar problems, so I suspect that this is a Linux kernel
>issue.

I'm not up to date on the various Linux kernels performance.  You
can find some benchmarking material from Kris Kennaway, who's keeping
an eye on FreeBSD's scalability and often compares to various
Linux kernels here:
	http://people.freebsd.org/~kris/scaling/


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.