[Varnish] #84: High number of "dropped work requests"

[Varnish]

#84: High number of "dropped work requests"
----------------------+-----------------------------------------------------
 Reporter:  Thias     |       Owner:  phk
     Type:  defect    |      Status:  new
 Priority:  high      |   Milestone:     
Component:  varnishd  |     Version:  1.0
 Severity:  normal    |    Keywords:     
----------------------+-----------------------------------------------------
 I've just installed varnish on an x86 RHEL4 server (Linux 2.6.9-11.ELsmp).
 Here is the content of my vcl file :
 {{{
 backend default {
         set backend.host = "192.168.39.234";
         set backend.port = "80";
 }

 sub vcl_recv {
         # pass mode can't handle POST (yet)
         if (req.request == "POST") {
                 pipe;
         }

         # force lookup even when cookies are present
         if (req.request == "GET" && req.http.cookie) {
                 lookup;
         }
 }

 sub vcl_fetch {
         # force minimum ttl of 180 seconds
         if (obj.ttl < 180s) {
                 set obj.ttl = 180s;
         }
 }
 }}}

 Which is pretty much the default configuration with the backend changed.
 And here is the command line to my varnish instance :

 {{{
 /usr/sbin/varnishd \
     -a 0.0.0.0:80 \
     -h classic \
     -f /etc/varnish/vcl.conf \
     -T 127.0.0.1:6082 \
     -t 1200 \
     -w 1,INF,10 \
     -s file,/var/lib/varnish/varnish_storage.bin,2g
 }}}

 Again, this is using the /etc/sysconfig/varnish file and only the
 host/port and storage size changed. Well, testing a connection to varnish
 works, and I get to see the page I expect from the backend server. So far
 so good. But then I start sending traffic, a few Mbps, and I start getting
 "zero sized replies" from varnish. I see plenty of dropped requests...
 most of them, actually!

 {{{
 # varnishstat -1
        26833  Client connections accepted
        14279  Client requests received
         8459  Cache hits
           25  Cache hits for pass
         5795  Cache misses
         5820  Backend connections initiated
         5732  Backend connections recycles
            0  Backend connections unused
          414  N struct srcaddr
            4  N active struct srcaddr
           63  N struct sess_mem
           13  N struct sess
         5796  N struct object
         5796  N struct objecthead
         5795  N struct smf
            0  N small free smf
            2  N large free smf
            1  N struct vbe_conn
            1  N worker threads
            1  N worker threads created
            0  N worker threads not created
         4718  N worker threads limited
            0  N queued work requests
         4719  N overflowed work requests
        25406  N dropped work requests
            0  N expired objects
            0  N objects on deathrow
            0  HTTP header overflows
         1514  Objects sent with sendfile
         8520  Objects sent with write
         5456  Total Sessions
        14279  Total Requests
            0  Total pipe
           27  Total pass
         5793  Total fetch
      3193858  Total header bytes
     78784328  Total body bytes
          140  Session Closed
           23  Session Pipeline
            5  Session Read Ahead
        14146  Session herd
       703381  SHM records
        74684  SHM writes
           15  SHM MTX contention
 }}}

 26833 requests accepted of which 25406 dropped!!! I've tried raising the
 max number of open files from 1024 to 500000, and also set
 net.ipv4.ip_local_port_range to "1200 64000", as those a pretty standard
 limitations with lots of connections, but it didn't help.

 What could be the cause? How can I try and find more explicit error
 messages? varnishlog shows lost of these :
 {{{
    13 StatSess         0 0 0 0 0 0 0 0
    13 SessionClose   dropped
    13 StatSess         0 0 0 0 0 0 0 0
    13 SessionClose   dropped
    13 StatSess         0 0 0 0 0 0 0 0
    13 SessionClose   dropped
    13 StatSess         0 0 0 0 0 0 0 0
    13 SessionClose   dropped
 }}}

 This is with varnish 1.0.2.

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/84>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator