Child panics on OpenSolaris

[Paul Wright]

On 12 February 2010 12:37, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <282e72051002120417w6c8ce8bcha2c96a809e9badc1 at mail.gmail.com>, Paul
> Wright writes:
>
>>> If so: how long time elapses between the ACK and the RST ?
>>
>>0.031s
>
> Interesting.
>
> Can you try to run with this patch:
>
> http://phk.freebsd.dk/misc/poll.patch
>
> And see what values it puts in your varnishlog for these connections ?
>
> I wonder if Solaris has some kind of "I already told you it were
> closed once" logic...

Here's a snippet from varnishlog for one of these panics (also
attached to this email in case line wraps wreck formatting):

  143 ReqEnd       c 1290159921 1265995548.765989065
1265995548.766152620 0.394018888 0.000070572 0.000092983
  143 Debug        c "herding"
  143 Debug        c "Poll: 1 / 1"
  143 ReqStart     c 78.105.40.8 54467 1290159956
  143 RxRequest    c GET
  143 RxURL        c /i/template/2009/footer_bottom.gif
  143 RxProtocol   c HTTP/1.1
  143 RxHeader     c Accept: */*
  143 RxHeader     c Referer: http://www.firebox.com/?aff=1661
  143 RxHeader     c Accept-Language: en-GB
  143 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 8.0;
Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR
3.5.30729; .NET CLR 3.0.30729)
  143 RxHeader     c Accept-Encoding: gzip, deflate
  143 RxHeader     c Host: media1.firebox.com
  143 RxHeader     c Connection: Keep-Alive
  143 RxHeader     c Cookie: user_session=XXXX;
locale=company_id%3A0%2Ccurrency_id%3A0%2Clanguage_id%3A0%2Ccountry%3AUnited+Kingdom;
xp_list=52%3D1; LS_timeEntered=2010-02-12+17%3A25%3A45;
LS_siteID=9zbw6_pUJdo-CEmBfO.mTMjnl23RueBRWg; intl_aff=111
  143 VCL_call     c recv
  143 VCL_return   c lookup
  143 VCL_call     c hash
  143 VCL_return   c hash
  143 Hit          c 1290147201
  143 VCL_call     c hit
  143 VCL_return   c deliver
  143 Length       c 256
  143 VCL_call     c deliver
  143 VCL_return   c deliver
  143 TxProtocol   c HTTP/1.1
  143 TxStatus     c 200
  143 TxResponse   c OK
  143 TxHeader     c Server: Apache
  143 TxHeader     c Last-Modified: Mon, 11 May 2009 10:47:04 GMT
  143 TxHeader     c ETag: "ae80a9-100-469a0b2e6da00"
  143 TxHeader     c Cache-Control: max-age=2592000
  143 TxHeader     c Expires: Sun, 14 Mar 2010 17:22:56 GMT
  143 TxHeader     c Content-Type: image/gif
  143 TxHeader     c Content-Length: 256
  143 TxHeader     c Date: Fri, 12 Feb 2010 17:25:49 GMT
  143 TxHeader     c X-Varnish: 1290159956 1290147201
  143 TxHeader     c Age: 172
  143 TxHeader     c Via: 1.1 varnish
  143 TxHeader     c Connection: keep-alive
  143 ReqEnd       c 1290159956 1265995549.060598135
1265995549.060703278 0.294445515 0.000055552 0.000049591

Child (19545) died signal=6
Child (19545) Panic message: Assert error in TCP_nonblocking(), tcp.c line 172:
  Condition((ioctl(sock, ((int)((uint32_t)(0x80000000|(((sizeof
(int))&0xff)<<16)| ('f'<<8)|126))), &i)) == 0) not true.
errno = 9 (Bad file number)
thread = (cache-worker)
ident = -smalloc,-hcritbit,poll
Backtrace:
  4454cb: /opt/sbin/varnishd'pan_backtrace+0x1b [0x4454cb]
  4457d5: /opt/sbin/varnishd'pan_ic+0x1c5 [0x4457d5]
  fffffd7ff2fcdfec: /opt/lib/libvarnish.so.1.0.0'TCP_nonblocking+0x7c
[0xfffffd7ff2fcdfec]
  419091: /opt/sbin/varnishd'vca_return_session+0x1b1 [0x419091]
  42679d: /opt/sbin/varnishd'cnt_wait+0x2bd [0x42679d]
  42b98a: /opt/sbin/varnishd'CNT_Session+0x4ba [0x42b98a]
  44805b: /opt/sbin/varnishd'wrk_do_cnt_sess+0x19b [0x44805b]
  447654: /opt/sbin/varnishd'wrk_thread_real+0x854 [0x447654]
  447bb3: /opt/sbin/varnishd'wrk_thread+0x123 [0x447bb3]
  fffffd7ff653acf5: /lib/amd64/libc.so.1'_thrp_setup+0x8d [0xfffffd7ff653acf5]
sp = c84b38 {
  fd = 143, id = 143, xid = 0,
  client = 78.105.40.8:54467,
  step = STP_WAIT,
  handling = deliver,
  restarts = 0, esis = 0
  ws = c84ba8 {
    id = "sess",
    {s,f,r,e} = {c858b0,+18,+32786,+65536},
  },
  http[req] = {
    ws = c84ba8[sess]
      "",
      "/i/template/2009/footer_bottom.gif",
      "HTTP/1.1",
      "Accept: */*",
      "Referer: http://www.firebox.com/?aff=1661",
      "Accept-Language: en-GB",
      "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1;
Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR
3.0.30729)",
      "Accept-Encoding: gzip, deflate",
      "Connection: Keep-Alive",
      "host: media.firebox.com",
      "X-Forwarded-For: 78.105.40.8",
  },
},

Paul.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish_child.143.log
Type: application/octet-stream
Size: 3779 bytes
Desc: not available
Url : http://projects.linpro.no/pipermail/varnish-misc/attachments/20100212/4719c7e7/attachment-0001.obj