many workers threads failed with EAGAIN

[David Birdsong]

varnishlog has a lot of these:
 0 Debug        - "Create worker thread failed 11 Resource temporarily
unavailable"

sure enough, overflowed and dropped work requests are steadily on the rise
Hitrate ratio:       10      100     1000
Hitrate avg:     0.8584   0.8506   0.8581

     1123681       423.11       380.14 Client connections accepted
      706252       189.05       238.92 Client requests received
      587166       159.04       198.64 Cache hits
        6928         3.00         2.34 Cache hits for pass
      111419        27.01        37.69 Cache misses
      119397        30.01        40.39 Backend conn. success
           2         0.00         0.00 Backend conn. not attempted
        5744          .            .   N struct sess_mem
        3663          .            .   N struct sess
       95290          .            .   N struct object
       95912          .            .   N struct objecthead
           6          .            .   N struct vbe_conn
        1007          .            .   N worker threads
        1007         0.00         0.34 N worker threads created
       12425         4.00         4.20 N worker threads not created
        2002         1.00         0.68 N queued work requests
      651706       157.04       220.47 N overflowed work requests
      460396       262.07       155.75 N dropped work requests
           9          .            .   N backends
       16469          .            .   N LRU nuked objects
           0          .            .   N LRU moved objects
           5         0.00         0.00 HTTP header overflows
      580405       158.04       196.35 Objects sent with write
      674521       186.05       228.19 Total Sessions
      706252       189.05       238.92 Total Requests
         904         0.00         0.31 Total pipe
        6928         3.00         2.34 Total pass
      118116        30.01        39.96 Total fetch
   189339016     50665.57     64052.44 Total header bytes
 49792939291  18055006.14  16844702.06 Total body bytes
      143000        47.01        48.38 Session Closed
         779         0.00         0.26 Session Pipeline
          83         0.00         0.03 Session Read Ahead
      609288       155.04       206.12 Session Linger


My arguments to start varnishd:
ulimit -n 500000
# stack size
ulimit -s 256
/usr/local/sbin/varnishd \
  -T localhost:6082 \
  -f $VCL \
  -s malloc,10GB \
  -P /var/run/varnish.pid \
  -h classic,1000003 \
  -p lru_interval=3600 \
  -p thread_pool_min=500 \
  -p thread_pool_max=2000 \
  -p thread_pools=2 \
  -p obj_workspace=4096 \
  -u apache \
  -p listen_depth=4096