[Varnish] #1698: compressed objects over allocate when content-length is specified

[Varnish]

#1698: compressed objects over allocate when content-length is specified
---------------------+----------------------
 Reporter:  dward84  |       Type:  defect
   Status:  new      |   Priority:  normal
Milestone:           |  Component:  varnishd
  Version:  4.0.3    |   Severity:  normal
 Keywords:           |
---------------------+----------------------
 Varnish appears to be allocating the uncompressed object size when
 1. Declaring beresp.do_gzip (so that the server compresses content when it
 stores it in cache)
 2. The server returns content-length

 This results in excessive memory consumption.

 As an attempted work-around, I tried removing the content-length header in
 vcl_backend_response, which doesn't work.

 The following test shows that more than 40K is allocated for what
 compresses down to several hundred bytes.

 {{{

 varnishtest "bloated object size"

 server s1 {
     rxreq

     # add -nolen here to remove Content-Length. it will just allocate the
 fetch_chunksize, passing the test.
     txresp -bodylen 40000
 } -start

 varnish v1 -arg "-s malloc,1M" \
            -arg "-pgzip_level=8" \
            -arg "-phttp_gzip_support=true" \
            -arg "-pfetch_chunksize=4k" \
            -vcl+backend {

     sub vcl_backend_response {
         set beresp.do_gzip = true;
         set beresp.ttl = 1m;

         # This doesn't work... but should it?
         unset beresp.http.Content-Length;
     }
 } -start

 client c1 {
     txreq -url "/foo"
     rxresp
     expect resp.bodylen == 40000
 } -run

 varnish v1 -expect SMA.s0.g_bytes <= 40000

 }}}


 Is this expected behavior? If so, is there a workaround?

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1698>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator