[Varnish] #771: caching large objects
Varnish
varnish-bugs at varnish-cache.org
Mon Mar 7 12:59:25 CET 2011
#771: caching large objects
----------------------+-----------------------------------------------------
Reporter: bcakipp | Owner: kristian
Type: defect | Status: new
Priority: normal | Milestone: Varnish 3.0 dev
Component: varnishd | Version: trunk
Severity: normal | Keywords:
----------------------+-----------------------------------------------------
Changes (by kristian):
* owner: phk => kristian
* milestone: => Varnish 3.0 dev
Old description:
> Hi,
>
> Working with Varnish 2.1.3 on CentOS5. I've noticed child threads dying
> with an assertion error when trying to fetch large objets. From what I
> can tell, objects with a Content-Length larger then 4GB will cause the
> child thread to die.
>
> For example (file is ~ 6GB).
>
> {{{
> $ curl http://glint1/aries/75123740001/75123740001_601962598001_ALO-
> IPA-083010.mov
> -o /dev/null -v -s
> * About to connect() to glint1 port 80
> * Trying 192.168.104.211... connected
> * Connected to glint1 (192.168.104.211) port 80
> > GET /aries/75123740001/75123740001_601962598001_ALO-IPA-083010.mov
> HTTP/1.1
> > User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
> OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> > Host: glint1
> > Accept: */*
> >
> * Empty reply from server
> * Connection #0 to host glint1 left intact
> * Closing connection #0
> }}}
>
> Syslog shows:
>
> {{{
> Sep 2 20:31:09 glint1 varnishd[21426]: Child (22486) died signal=6
> Sep 2 20:31:09 glint1 varnishd[21426]: Child (22486) Panic message:
> Assert error in fetch_straight(), cache_fetch.c line 64:
> Condition((uintmax_t)cl == cll) not true. thread = (cache-worker)
> ident = Linux,2.6.18-164.15.1.el5xen,x86_64,-sfile,-hcritbit,epoll
> Backtrace: 0x422726: /usr/sbin/varnishd [0x422726] 0x41a8f8:
> /usr/sbin/varnishd(FetchBody+0x4a8) [0x41a8f8] 0x412960:
> /usr/sbin/varnishd [0x412960] 0x413ed5:
> /usr/sbin/varnishd(CNT_Session+0x345) [0x413ed5] 0x424b78:
> /usr/sbin/varnishd [0x424b78] 0x423e5d: /usr/sbin/varnishd
> [0x423e5d] 0x3c9de06617: /lib64/libpthread.so.0 [0x3c9de06617]
> 0x3c9d6d3c2d: /lib64/libc.so.6(clone+0x6d) [0x3c9d6d3c2d] sp =
> 0x2abe71908008 { fd = 11, id = 11, xid = 674181670, client =
> 192.168.103.71:48903, step = STP_FETCH, handling = deliver,
> err_code = 200, err_reason = (null), restarts = 0, esis = 0 ws =
> 0x2abe71908078 { id = "sess", {s,f,r,e} =
> {0x2abe71908cd0,+240,(nil),+65536}, }, http[req] = { ws =
> 0x2abe71908078[sess] "
> }}}
>
> Looking at that line in cache_fetch.c:
>
> {{{
> uintmax_t cll;
> unsigned cl, sl;
> struct storage *st;
>
> cll = strtoumax(b, NULL, 0);
> if (cll == 0)
> return (0);
>
> cl = (unsigned)cll;
> assert((uintmax_t)cl == cll); /* Protect against bogusly large
> values */
> }}}
>
> My reading of that says that anything larger then an unsigned int (~4GB)
> will cause the client thread to die.
>
> I understand that it would be possible to just pipe these large files,
> but what I would really like to do is cache them in varnish. Is this
> possible?
New description:
Hi,
Working with Varnish 2.1.3 on CentOS5. I've noticed child threads dying
with an assertion error when trying to fetch large objets. From what I can
tell, objects with a Content-Length larger then 4GB will cause the child
thread to die.
For example (file is ~ 6GB).
{{{
$ curl http://glint1/aries/75123740001/75123740001_601962598001_ALO-
IPA-083010.mov
-o /dev/null -v -s
* About to connect() to glint1 port 80
* Trying 192.168.104.211... connected
* Connected to glint1 (192.168.104.211) port 80
> GET /aries/75123740001/75123740001_601962598001_ALO-IPA-083010.mov
HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: glint1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host glint1 left intact
* Closing connection #0
}}}
Syslog shows:
{{{
Sep 2 20:31:09 glint1 varnishd[21426]: Child (22486) died signal=6
Sep 2 20:31:09 glint1 varnishd[21426]: Child (22486) Panic message:
Assert error in fetch_straight(), cache_fetch.c line 64:
Condition((uintmax_t)cl == cll) not true. thread = (cache-worker)
ident = Linux,2.6.18-164.15.1.el5xen,x86_64,-sfile,-hcritbit,epoll
Backtrace: 0x422726: /usr/sbin/varnishd [0x422726] 0x41a8f8:
/usr/sbin/varnishd(FetchBody+0x4a8) [0x41a8f8] 0x412960:
/usr/sbin/varnishd [0x412960] 0x413ed5:
/usr/sbin/varnishd(CNT_Session+0x345) [0x413ed5] 0x424b78:
/usr/sbin/varnishd [0x424b78] 0x423e5d: /usr/sbin/varnishd
[0x423e5d] 0x3c9de06617: /lib64/libpthread.so.0 [0x3c9de06617]
0x3c9d6d3c2d: /lib64/libc.so.6(clone+0x6d) [0x3c9d6d3c2d] sp =
0x2abe71908008 { fd = 11, id = 11, xid = 674181670, client =
192.168.103.71:48903, step = STP_FETCH, handling = deliver,
err_code = 200, err_reason = (null), restarts = 0, esis = 0 ws =
0x2abe71908078 { id = "sess", {s,f,r,e} =
{0x2abe71908cd0,+240,(nil),+65536}, }, http[req] = { ws =
0x2abe71908078[sess] "
}}}
Looking at that line in cache_fetch.c:
{{{
uintmax_t cll;
unsigned cl, sl;
struct storage *st;
cll = strtoumax(b, NULL, 0);
if (cll == 0)
return (0);
cl = (unsigned)cll;
assert((uintmax_t)cl == cll); /* Protect against bogusly large
values */
}}}
My reading of that says that anything larger then an unsigned int (~4GB)
will cause the client thread to die.
I understand that it would be possible to just pipe these large files, but
what I would really like to do is cache them in varnish. Is this possible?
--
Comment:
Should be fixed, but I'm confirming this this week.
--
Ticket URL: <http://varnish-cache.org/trac/ticket/771#comment:2>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list