Child panics on OpenSolaris
Paul Wright
wrighty+varnishmisc at gmail.com
Wed Feb 17 11:30:29 CET 2010
On 16 February 2010 20:48, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
>
> OK,
>
> It took me a day to liberate a machine to install OpenSolaris on.
>
> Then it took most of a day to remember why I really like FreeBSD
> over Solaris.
>
> Then I got distracted by the failing non-blocking connect to backends
> and ran truss.
>
> Then I found the root-cause of the problem.
>
> We need CFLAGS to contain -mt on solaris, otherwise errno does not
> get macro-fied to be per-thread.
>
> That's where the: EBADF comes from, some entirely different
> filedescriptor a long time ago, in the master process...
>
> try:
> env CFLAGS=-mt sh configure
Thanks so much for looking into this. I spent yesterday attempting to
get some useful debug information out of dtrace with little success -
I either ended up with megabytes of debug dumps or empty files
depending on my D script. Not exactly successful.
I've compiled with the additional -mt flag, here's my current
compilation process:
make clean
./autogen.sh
CFLAGS="-m64 -mt" CC=/opt/SunStudioExpress/bin/cc ./configure --prefix=/opt
make
make install
And I'm starting up with:
newtask -p highfile /opt/sbin/varnishd -f /opt/etc/varnish/firebox.vcl -F \
-p cc_command='/opt/SunStudioExpress/bin/cc -Kpic -G -m64 -o %o %s' \
-T 127.0.0.1:9001 \
-s malloc,1G \
-p sess_timeout=5s \
-p max_restarts=12 \
-p waiter=poll \
-p connect_timeout=0s \
-p sess_workspace=65536
Varnish seems to be a *lot* happier and panics less regularly. Panics
now all seem to suggest the error is "Connection reset by peer"
instead of EBADF
child (14052) Started
Child (14052) said Closed fds: 3 4 5 9 10 12 13
Child (14052) said Child starts
Child (14052) died signal=6
Child (14052) Panic message: Assert error in TCP_nonblocking(), tcp.c line 172:
Condition((ioctl(sock, ((int)((uint32_t)(0x80000000|(((sizeof
(int))&0xff)<<16)| ('f'<<8)|126))), &i)) == 0) not true.
errno = 131 (Connection reset by peer)
thread = (cache-worker)
ident = -smalloc,-hcritbit,poll
Backtrace:
447adb: /opt/sbin/varnishd'pan_backtrace+0x1b [0x447adb]
447de5: /opt/sbin/varnishd'pan_ic+0x1c5 [0x447de5]
fffffd7ff181e6da: /opt/lib/libvarnish.so.1.0.0'TCP_nonblocking+0x8a
[0xfffffd7ff181e6da]
41924e: /opt/sbin/varnishd'vca_return_session+0x1de [0x41924e]
4276ea: /opt/sbin/varnishd'cnt_wait+0x2ea [0x4276ea]
42ccd6: /opt/sbin/varnishd'CNT_Session+0x526 [0x42ccd6]
44a7ef: /opt/sbin/varnishd'wrk_do_cnt_sess+0x1bf [0x44a7ef]
449d62: /opt/sbin/varnishd'wrk_thread_real+0x882 [0x449d62]
44a315: /opt/sbin/varnishd'wrk_thread+0x135 [0x44a315]
fffffd7ff653acf5: /lib/amd64/libc.so.1'_thrp_setup+0x8d [0xfffffd7ff653acf5]
sp = 1742b68 {
fd = 40, id = 40, xid = 0,
client = 94.226.122.36:3380,
step = STP_WAIT,
handling = deliver,
restarts = 0, esis = 0
ws = 1742bd8 {
id = "sess",
{s,f,r,e} = {17438e0,+19,+32787,+65536},
},
http[req] = {
ws = 1742bd8[sess]
"",
"/pic/p2607_small.jpg",
"HTTP/1.1",
"Accept: */*",
"Referer: http://www.firebox.com/?aff=1759/",
"Accept-Language: nl-be",
"User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1;
Trident/4.0)",
"Accept-Encoding: gzip, deflate",
"Connection: Keep-Alive",
"host: media.firebox.com",
"X-Forwarded-For: 94.226.122.36",
},
},
And I'm also seeing the same error occur in http_copyheader() - could
this be due to the Range: aspect of the client request?
Child (12008) died signal=6
Child (12008) Panic message: Assert error in http_copyheader(),
cache_http.c line 647:
Condition(n < to->shd) not true.
errno = 131 (Connection reset by peer)
thread = (cache-worker)
ident = -smalloc,-hcritbit,poll
Backtrace:
447adb: /opt/sbin/varnishd'pan_backtrace+0x1b [0x447adb]
447de5: /opt/sbin/varnishd'pan_ic+0x1c5 [0x447de5]
440791: /opt/sbin/varnishd'http_copyheader+0x1c1 [0x440791]
4423b1: /opt/sbin/varnishd'http_FilterFields+0xdc1 [0x4423b1]
429c4d: /opt/sbin/varnishd'cnt_fetch+0x11fd [0x429c4d]
42cf3a: /opt/sbin/varnishd'CNT_Session+0x78a [0x42cf3a]
44a7ef: /opt/sbin/varnishd'wrk_do_cnt_sess+0x1bf [0x44a7ef]
449d62: /opt/sbin/varnishd'wrk_thread_real+0x882 [0x449d62]
44a315: /opt/sbin/varnishd'wrk_thread+0x135 [0x44a315]
fffffd7ff653acf5: /lib/amd64/libc.so.1'_thrp_setup+0x8d [0xfffffd7ff653acf5]
sp = 397bf38 {
fd = 56, id = 56, xid = 1848010778,
client = 82.71.124.65:60324,
step = STP_FETCH,
handling = pass,
err_code = 206, err_reason = (null),
restarts = 0, esis = 0
ws = 397bfa8 {
id = "sess",
{s,f,r,e} = {397ccb0,+900,0,+65536},
},
http[req] = {
ws = 397bfa8[sess]
"GET",
"/xml/rss.top20.000.xml",
"HTTP/1.1",
"Host: www.firebox.com",
"User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5;
en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-gb,en;q=0.5",
"Accept-Encoding: gzip,deflate",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Keep-Alive: 115",
"Connection: keep-alive",
"X-Moz: livebookmarks",
"Cookie: user_session=XXXX;
locale=company_id%3A0%2Ccurrency_id%3A0%2Clanguage_id%3A0%2Ccountry%3AUnited+Kingdom;
xp_list=52%3D1;
__utma=64137912.1325032789.1266225177.1266397371.1266400910.11;
__utmc=64137912;
__utmz=64137912.1266225177.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none);
__utmb=64137912",
"Range: bytes=2494-",
"If-Range: "14b800b-678d-47fc841cd4140"",
"Cache-Control: max-age=0",
"X-Forwarded-For: 82.71.124.65",
},
worker = fffffd7ff7017d30 {
ws = fffffd7ff7017e78 {
id = "wrk",
{s,f,r,e} = {fffffd7ff7005c40,+323,0,+65536},
},
http[bereq] = {
ws = fffffd7ff7017e78[wrk]
"GET",
"/xml/rss.top20.000.xml",
"HTTP/1.1",
"Host: www.firebox.com",
"User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5;
en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6",
"Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-gb,en;q=0.5",
"Accept-Encoding: gzip,deflate",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"X-Moz: livebookmarks",
"Cookie: user_session=XXXX;
locale=company_id%3A0%2Ccurrency_id%3A0%2Clanguage_id%3A0%2Ccountry%3AUnited+Kingdom;
xp_list=52%3D1;
__utma=64137912.1325032789.1266225177.1266397371.1266400910.11;
__utmc=64137912;
__utmz=64137912.1266225177.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none);
__utmb=64137912",
"Range: bytes=2494-",
"If-Range: "14b800b-678d-47fc841cd4140"",
"X-Forwarded-For: 82.71.124.65",
"X-Varnish: 1848010778",
},
http[beresp] = {
ws = fffffd7ff7017e78[wrk]
"HTTP/1.1",
"206",
"Partial Content",
"Date: Wed, 17 Feb 2010 10:03:18 GMT",
"Server: Apache",
"Last-Modified: Wed, 17 Feb 2010 09:13:01 GMT",
"ETag: "14b800b-678d-47fc841cd4140"",
"Accept-Ranges: bytes",
"Content-Length: 24015",
"Content-Range: bytes 2494-26508/26509",
"Connection: close",
"Content-Type: application/xml",
},
},
vcl = {
srcname = {
"input",
"Default",
},
},
obj = 7202020 {
xid = 1848010778,
ws = 7202040 {
id = "obj",
{s,f,r,e} = {7202225,7202225,0,+220},
},
http[obj] = {
ws = 7202040[obj]
"HTTP/1.1",
"206",
"Partial Content",
"Date: Wed, 17 Feb 2010 10:03:18 GMT",
"Server: Apache",
"Last-Modified: Wed, 17 Feb 2010 09:13:01 GMT",
"ETag: "14b800b-678d-47fc841cd4140"",
},
len = 0,
store = {
},
},
},
Cheers,
Paul.
More information about the varnish-misc
mailing list