From jb-varnish at wisemo.com  Wed Feb 15 10:23:16 2023
From: jb-varnish at wisemo.com (Jakob Bohm)
Date: Wed, 15 Feb 2023 11:23:16 +0100
Subject: CLI result = 300, how to troubleshoot?
Message-ID: <889e4976-dfa6-860a-fd68-0ce54c465176@wisemo.com>

Dear fellow users,

I am running varnish-cache 7.2.1 (compiled from source) in
preproduction.
After some seemingly minor settings changes, every time I
try to start varnishd, I get the following on the terminal:

    # /etc/init.d/varnish start Starting Varnish HTTP(S) proxy:
    varnishWarnings: Change will take effect when VCL script is reloaded
    Child launched OK CLI result = 300 ?failed!

Now the questions are what does this mean?, and how do I get
a more detailed error message?? Obviously, the failure to
start varnishd makes varnishlog useless, but maybe there is
another log file for such startup errors.

Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 S?borg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230215/d81d818a/attachment-0001.html>

From dridi at varni.sh  Wed Feb 15 10:56:51 2023
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 15 Feb 2023 10:56:51 +0000
Subject: CLI result = 300, how to troubleshoot?
In-Reply-To: <889e4976-dfa6-860a-fd68-0ce54c465176@wisemo.com>
References: <889e4976-dfa6-860a-fd68-0ce54c465176@wisemo.com>
Message-ID: <CABoVN9AOVNCWm=wB=zg=Lbnj0TcUKyFWw+Xu-42V1uaWYUdC4Q@mail.gmail.com>

On Wed, Feb 15, 2023 at 10:24 AM Jakob Bohm <jb-varnish at wisemo.com> wrote:
>
> Dear fellow users,
>
> I am running varnish-cache 7.2.1 (compiled from source) in
> preproduction.
> After some seemingly minor settings changes, every time I
> try to start varnishd, I get the following on the terminal:
>
> # /etc/init.d/varnish start
> Starting Varnish HTTP(S) proxy: varnishWarnings:
>
> Change will take effect when VCL script is reloaded
> Child launched OK
> CLI result = 300
>  failed!
>
> Now the questions are what does this mean?, and how do I get
> a more detailed error message?  Obviously, the failure to
> start varnishd makes varnishlog useless, but maybe there is
> another log file for such startup errors.

There may be something in your syslog, otherwise it is hard to tell
without knowing what the init script is up to.

Dridi

From phk at phk.freebsd.dk  Wed Feb 15 11:27:30 2023
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 15 Feb 2023 11:27:30 +0000
Subject: CLI result = 300, how to troubleshoot?
In-Reply-To: <CABoVN9AOVNCWm=wB=zg=Lbnj0TcUKyFWw+Xu-42V1uaWYUdC4Q@mail.gmail.com>
References: <889e4976-dfa6-860a-fd68-0ce54c465176@wisemo.com>
 <CABoVN9AOVNCWm=wB=zg=Lbnj0TcUKyFWw+Xu-42V1uaWYUdC4Q@mail.gmail.com>
Message-ID: <202302151127.31FBRU8K021633@critter.freebsd.dk>

--------
Dridi Boukelmoune writes:

> > Change will take effect when VCL script is reloaded
> > Child launched OK
> > CLI result = 300
> >  failed!
> >
> > Now the questions are what does this mean?, and how do I get
> > a more detailed error message?  Obviously, the failure to
> > start varnishd makes varnishlog useless, but maybe there is
> > another log file for such startup errors.
>
> There may be something in your syslog, otherwise it is hard to tell
> without knowing what the init script is up to.

CLI commands are logged to the shared memory log, so

	varnishlog -d -g raw

may be able to tell you something

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

From jb-varnish at wisemo.com  Wed Feb 15 14:48:55 2023
From: jb-varnish at wisemo.com (Jakob Bohm)
Date: Wed, 15 Feb 2023 15:48:55 +0100
Subject: CLI result = 300, how to troubleshoot?
In-Reply-To: <202302151127.31FBRU8K021633@critter.freebsd.dk>
References: <889e4976-dfa6-860a-fd68-0ce54c465176@wisemo.com>
 <CABoVN9AOVNCWm=wB=zg=Lbnj0TcUKyFWw+Xu-42V1uaWYUdC4Q@mail.gmail.com>
 <202302151127.31FBRU8K021633@critter.freebsd.dk>
Message-ID: <7349fe4c-4778-acea-24cb-8d40d2b679b0@wisemo.com>

On 2023-02-15 12:27, Poul-Henning Kamp wrote:
> --------
> Dridi Boukelmoune writes:
> 
>>> Change will take effect when VCL script is reloaded
>>> Child launched OK
>>> CLI result = 300
>>>   failed!
>>>
>>> Now the questions are what does this mean?, and how do I get
>>> a more detailed error message?  Obviously, the failure to
>>> start varnishd makes varnishlog useless, but maybe there is
>>> another log file for such startup errors.
>>
>> There may be something in your syslog, otherwise it is hard to tell
>> without knowing what the init script is up to.
> 
> CLI commands are logged to the shared memory log, so
> 
> 	varnishlog -d -g raw
> 
> may be able to tell you something
> 

Thanks for the tip.  Invoking varnishd (not varnishlog) with the -d 
option revealed the error: Varnish doesn't like binding to both a 
specific IPv4 address (such as 192.0.2.3) and all remaining IPv4
addresses (0.0.0.0) on the same command line.

Not working
   varnishd -a 192.0.2.3 -a 0.0.0.0 ...
Working
   varnishd -a 0.0.0.0 ...



Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 S?borg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

From karim.ayari at univ-lyon1.fr  Mon Feb 20 15:13:07 2023
From: karim.ayari at univ-lyon1.fr (Karim Ayari)
Date: Mon, 20 Feb 2023 16:13:07 +0100
Subject: varnish and mp4 files
Message-ID: <acbd6ad5-cac0-e5d2-864e-6256a1308ba7@univ-lyon1.fr>

Hi!

I am currently experiencing a memory load problem with video playback.

here is the infrastructure :

client --> haproxy --> varnish --> moodle workers (x5)

a teacher uploaded a 400MB video to Moodle, when we start playing the 
video with browser player, Varnish consumes all the memory until it runs 
out and oom killer to kill varnishd.i have no configuration for mp4 
files in my vcl file, so by default they are not hidden (?). I can't 
find a solution :(

I can give my vcl file if necessary.

(I am a beginner on varnish :))

thank you for your support.

Karim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/db4d0a39/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4295 bytes
Desc: Signature cryptographique S/MIME
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/db4d0a39/attachment.bin>

From guillaume.quintard at gmail.com  Mon Feb 20 17:55:19 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 20 Feb 2023 09:55:19 -0800
Subject: varnish and mp4 files
In-Reply-To: <acbd6ad5-cac0-e5d2-864e-6256a1308ba7@univ-lyon1.fr>
References: <acbd6ad5-cac0-e5d2-864e-6256a1308ba7@univ-lyon1.fr>
Message-ID: <CAOrPormuFTWqMyRn5=4YA56Ubx3Nu2a197aXE=HcZabbRfCmbA@mail.gmail.com>

Hello Karim,

You VCL would be useful to debug this (as well as the command line you are
running Varnish with), but it sounds like Varnish is using the Transient
storage (
https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
to store the file, and as the storage isn't bounded, it explodes.
We can fix this in a couple of ways, from storing the file in the regular
cache storage, to using pipe, to waiting a few days for
https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
to be released.

Question is: should that file be cached?

Cheers,

-- 
Guillaume Quintard


On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari at univ-lyon1.fr>
wrote:

> Hi!
>
> I am currently experiencing a memory load problem with video playback.
>
> here is the infrastructure :
>
> client --> haproxy --> varnish --> moodle workers (x5)
>
> a teacher uploaded a 400MB video to Moodle, when we start playing the
> video with browser player, Varnish consumes all the memory until it runs
> out and oom killer to kill varnishd. i have no configuration for mp4
> files in my vcl file, so by default they are not hidden (?). I can't find
> a solution :(
>
> I can give my vcl file if necessary.
>
> (I am a beginner on varnish :))
>
> thank you for your support.
>
> Karim
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/c7dd1d75/attachment-0001.html>

From karim.ayari at univ-lyon1.fr  Mon Feb 20 20:36:12 2023
From: karim.ayari at univ-lyon1.fr (Karim Ayari)
Date: Mon, 20 Feb 2023 21:36:12 +0100
Subject: varnish and mp4 files
In-Reply-To: <CAOrPormuFTWqMyRn5=4YA56Ubx3Nu2a197aXE=HcZabbRfCmbA@mail.gmail.com>
References: <acbd6ad5-cac0-e5d2-864e-6256a1308ba7@univ-lyon1.fr>
 <CAOrPormuFTWqMyRn5=4YA56Ubx3Nu2a197aXE=HcZabbRfCmbA@mail.gmail.com>
Message-ID: <c7a414f9-0912-632d-da3a-d7d9fb3b2f83@univ-lyon1.fr>

thank you both for your replies.I forgot the most important thing:the 
varnish server has 16 GB of ram and the cache is 1 GB.

the cache never seems full


the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T 
localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s 
malloc,1g****-p http_max_hdr=96***

I had tried using pipe it didn't work, but Rainer's lines works fine.

this solution should suffice because the videos will soon have to be 
hosted on our video platform.

/
/

my vcl (file found on github) :

/
/

/....
/

/sub vcl_recv {//
//
//??? # Keep client IP//
//??? if (req.restarts == 0) {//
//??????? if (req.http.x-forwarded-for) {//
//??????????? set req.http.X-Forwarded-For = req.http.X-Forwarded-For + 
", " + client.ip;//
//??????? } else {//
//?? ???? unset req.http.X-Forwarded-For;//
//??????????? set req.http.X-Forwarded-For = client.ip;//
//??????? }//
//??? }//
//
//??? if (req.http.X-Real-IP) {//
//??????? set req.http.X-Forwarded-For = req.http.X-Real-IP;//
//??? } else {//
//??????? set req.http.X-Forwarded-For = client.ip;//
//??? }//
////
//??? # Only deal with "normal" types//
//??? if (req.method != "GET" &&//
//????? req.method != "HEAD" &&//
//????? req.method != "PUT" &&//
//????? req.method != "POST" &&//
//????? req.method != "TRACE" &&//
//????? req.method != "OPTIONS" &&//
//????? req.method != "PATCH" &&//
//????? req.method != "DELETE") {//
//?? ???? /* Non-RFC2616 or CONNECT which is weird. *///
//?? ?/*Why send the packet upstream, while the visitor is using a 
non-valid HTTP method? *///
//?? ?return (synth(404, "Non-valid HTTP method!"));//
//????? }//
////
//??? # Varnish don't mess with healthchecks//
//??? if (req.url ~ "^/admin/tool/heartbeat" || req.url ~ 
"^/healthcheck.php") {//
//??????? return (pass);//
//??? }//
//??? # Pipe requests to backup.php straight to backend - prevents 
problem with progress bar long polling 503 problem//
//??? # This is here because backup.php is POSTing to itself - Filter 
before !GET&&!HEAD//
//??? if (req.url ~ "^/backup/backup.php")//
//??? {//
//??????? return (pipe);//
//??? }//
//
//??? # Varnish only deals with GET and HEAD by default. If request 
method is not GET or HEAD, pass request to backend//
//??? if (req.method != "GET" && req.method != "HEAD") {//
//????? return (pass);//
//??? }//
//
//??? if (req.http.Cookie) {//
//????? # Remove any Google Analytics based cookies//
//????? set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");//
//????? set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");//
//????? set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; 
)?", "");//
//????? set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; 
)?", "");//
//????? set req.http.Cookie = regsuball(req.http.Cookie, 
"__atuv.=[^;]+(; )?", "");//
//????? set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");//
//????? if (req.http.Cookie ~ "^\s*$") {//
//??????? unset req.http.Cookie;//
//????? }//
//??? }//
//
//?### Rules for Moodle ###//
//
//??? # Perform lookup for selected assets that we know are static but 
Moodle still needs a Cookie//
//??? if(? req.url ~ "^/theme/.+\.?" ||//
//?? ? req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||//
//???????? req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||//
//???????? req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" 
||//
//?? ? req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"//
//????? )//
//??? {//
//???????? # Set internal temporary header, based on which we will do 
things in vcl_backend_response//
//???????? set req.http.X-Long-TTL = "86400";//
//???????? return (hash);//
//??? }//
//??? # Requests containing "Cookie" or "Authorization" headers will not 
be cached//
//??? if (req.http.Authorization || req.http.Cookie) {//
//??????? return (pass);//
//??? }//
//??? # Almost everything in Moodle correctly serves Cache-Control 
headers, if//
//??? # needed, which varnish will honor, but there are some which 
don't. Rather//
//??? # than explicitly finding them all and listing them here we just 
fail safe//
//??? # and don't cache unknown urls that get this far.//
//??? return (pass);//
//
//}//
//
//sub vcl_backend_response {//
//??? # Set backend name//
//??? set beresp.http.X-Backend = beresp.backend.name;//
//
//??? if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && 
beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) && 
!beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL 
header//
//??????? set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset 
beresp.http.Pragma;//
//??????? set beresp.http.X-Orig-Cache-Control = 
beresp.http.Cache-Control;//
//??????? set beresp.http.Cache-Control = "public, 
max-age="+bereq.http.X-Long-TTL+", no-transform";//
//??????? set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);//
//??????? unset bereq.http.X-Long-TTL;//
//??? }//
//??? else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL && 
!beresp.http.WWW-Authenticate ) {//
//??????? set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset 
beresp.http.Pragma;//
//??????? set beresp.http.Cache-Control = "public, 
max-age="+bereq.http.X-Long-TTL+", no-transform";//
//??????? set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);//
//??????? unset bereq.http.X-Long-TTL;//
//??? }//
//??? else { # Don't touch headers if max-age > defined in X-Long-TTL 
header//
//??????? unset bereq.http.X-Long-TTL;//
//??? }//
//??? # Here we set X-Trace header, prepending it to X-Trace header 
received from backend. Useful for troubleshooting//
//??? if (beresp.http.x-trace && !beresp.was_304) {//
//??????? set beresp.http.X-Trace = regsub(server.identity, 
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name, 
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;//
//??? }//
//??? else {//
//??????? set beresp.http.X-Trace = regsub(server.identity, 
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name, 
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");//
//??? }//
//}//
//
//sub vcl_deliver {//
//
//# Revert back to original Cache-Control header before delivery to client//
//??? if (resp.http.X-Orig-Cache-Control)//
//??? {//
//??????? set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;//
//??????? unset resp.http.X-Orig-Cache-Control;//
//??? }//
//??? # Revert back to original Pragma header before delivery to client//
//??? if (resp.http.X-Orig-Pragma)//
//??? {//
//??????? set resp.http.Pragma = resp.http.X-Orig-Pragma;//
//??????? unset resp.http.X-Orig-Pragma;//
//??? }//
//
//? if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and 
the number of hits, disable when not needed//
//??? set resp.http.X-Cache = "HIT";//
//? } else {//
//??? set resp.http.X-Cache = "MISS";//
//? }//
//??? set resp.http.X-Cache-Hits = obj.hits;//
//
//# If desired "Via: 1.1 Varnish-v4" response header can be removed from 
response//
//??? unset resp.http.Via;//
//??? unset resp.http.Server;//
//
//??? return (deliver);//
//}//
//
//sub vcl_backend_error {//
//??? # More comprehensive varnish error page. Display time, instance 
hostname, host header, url for easier troubleshooting.//
//??? set beresp.http.Content-Type = "text/html; charset=utf-8";//
//??? set beresp.http.Retry-After = "5";//
//??? synthetic( {"//
//? <!DOCTYPE html>//
//? <html>//
//??? <head>//
//????? <title>"} + beresp.status + " " + beresp.reason + {"</title>//
//??? </head>//
//??? <body>//
//????? <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>//
//????? <p>"} + beresp.reason + {"</p>//
//????? <h3>Guru Meditation:</h3>//
//????? <p>Time: "} + now + {"</p>//
//????? <p>Node: "} + server.hostname + {"</p>//
//????? <p>Host: "} + bereq.http.host + {"</p>//
//????? <p>URL: "} + bereq.url + {"</p>//
//????? <p>XID: "} + bereq.xid + {"</p>//
//????? <hr>//
//????? <p>Varnish cache server//
//??? </body>//
//? </html>//
//? "} );//
//?? return (deliver);//
//}//
//
//sub vcl_synth {//
//??? #Redirect using '301 - Permanent Redirect', permanent redirect//
//??? if (resp.status == 851) { //
//??????? set resp.http.Location = req.http.x-redir;//
//??????? set resp.http.X-Varnish-Redirect = true;//
//??????? set resp.status = 301;//
//??????? return (deliver);//
//??? }//
//??? #Redirect using '302 - Found', temporary redirect//
//??? if (resp.status == 852) { //
//??????? set resp.http.Location = req.http.x-redir;//
//??????? set resp.http.X-Varnish-Redirect = true;//
//??????? set resp.status = 302;//
//??????? return (deliver);//
//??? }//
//??? #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests, 
dont change method on redirected requests//
//??? if (resp.status == 857) { //
//??????? set resp.http.Location = req.http.x-redir;//
//??????? set resp.http.X-Varnish-Redirect = true;//
//??????? set resp.status = 307;//
//??????? return (deliver);//
//??? }//
//??? #Respond with 403 - Forbidden//
//??? if (resp.status == 863) {//
//??????? set resp.http.X-Varnish-Error = true;//
//??????? set resp.status = 403;//
//??????? return (deliver);//
//??? }//
//}//
//
//sub vcl_purge {//
//? if (req.method != "PURGE") {//
//??? set req.http.X-Purge = "Yes";//
//??? return (restart);//
//? }//
//}/

...


Karim Ayari
Universit? Claude Bernard - Lyon 1
Service ICAP
Ing?nieur syst?mes

Le 20/02/2023 ? 18:55, Guillaume Quintard a ?crit?:
> Hello Karim,
>
> You VCL would be useful to debug this (as well as the command line you 
> are running Varnish with), but it sounds like Varnish is using the 
> Transient storage 
> (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage) 
> to store the file, and as the storage isn't bounded, it explodes.
> We can fix this in a couple of ways, from storing the file in the 
> regular cache storage, to using pipe, to waiting a few days for 
> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643 
> to be released.
>
> Question is: should that file be cached?
>
> Cheers,
>
> -- 
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari 
> <karim.ayari at univ-lyon1.fr> wrote:
>
>     Hi!
>
>     I am currently experiencing a memory load problem with video playback.
>
>     here is the infrastructure :
>
>     client --> haproxy --> varnish --> moodle workers (x5)
>
>     a teacher uploaded a 400MB video to Moodle, when we start playing
>     the video with browser player, Varnish consumes all the memory
>     until it runs out and oom killer to kill varnishd.i have no
>     configuration for mp4 files in my vcl file, so by default they are
>     not hidden (?). I can't find a solution :(
>
>     I can give my vcl file if necessary.
>
>     (I am a beginner on varnish :))
>
>     thank you for your support.
>
>     Karim
>
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org
>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4295 bytes
Desc: Signature cryptographique S/MIME
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0001.bin>

From guillaume.quintard at gmail.com  Mon Feb 20 20:50:19 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 20 Feb 2023 12:50:19 -0800
Subject: varnish and mp4 files
In-Reply-To: <c7a414f9-0912-632d-da3a-d7d9fb3b2f83@univ-lyon1.fr>
References: <acbd6ad5-cac0-e5d2-864e-6256a1308ba7@univ-lyon1.fr>
 <CAOrPormuFTWqMyRn5=4YA56Ubx3Nu2a197aXE=HcZabbRfCmbA@mail.gmail.com>
 <c7a414f9-0912-632d-da3a-d7d9fb3b2f83@univ-lyon1.fr>
Message-ID: <CAOrPor=zG5PkKr-_3sVnTjfFED_Xepmk5_BMBLEgosZG6r5bLw@mail.gmail.com>

Looks like Rainer replied directly to you and not to the list, would you
mind sharing/highlighting the fix for people having the same issue?

Cheers,

-- 
Guillaume Quintard


On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari at univ-lyon1.fr>
wrote:

> thank you both for your replies. I forgot the most important thing: the
> varnish server has 16 GB of ram and the cache is 1 GB.
>
> the cache never seems full
>
>
> the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
> localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
> malloc,1g* *-p http_max_hdr=96*
>
> I had tried using pipe it didn't work, but Rainer's lines works fine.
>
> this solution should suffice because the videos will soon have to be
> hosted on our video platform.
>
>
> my vcl (file found on github) :
>
>
>
> *.... *
>
> *sub vcl_recv {*
>
> *    # Keep client IP*
> *    if (req.restarts == 0) {*
> *        if (req.http.x-forwarded-for) {*
> *            set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ",
> " + client.ip;*
> *        } else {*
> *        unset req.http.X-Forwarded-For;*
> *            set req.http.X-Forwarded-For = client.ip;*
> *        }*
> *    }*
>
> *    if (req.http.X-Real-IP) {*
> *        set req.http.X-Forwarded-For = req.http.X-Real-IP;*
> *    } else {*
> *        set req.http.X-Forwarded-For = client.ip;*
> *    }*
>
> *    # Only deal with "normal" types*
> *    if (req.method != "GET" &&*
> *      req.method != "HEAD" &&*
> *      req.method != "PUT" &&*
> *      req.method != "POST" &&*
> *      req.method != "TRACE" &&*
> *      req.method != "OPTIONS" &&*
> *      req.method != "PATCH" &&*
> *      req.method != "DELETE") {*
> *        /* Non-RFC2616 or CONNECT which is weird. */*
> *    /*Why send the packet upstream, while the visitor is using a
> non-valid HTTP method? */*
> *    return (synth(404, "Non-valid HTTP method!"));*
> *      }*
>
> *    # Varnish don't mess with healthchecks*
> *    if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
> "^/healthcheck.php") {*
> *        return (pass);*
> *    }*
> *    # Pipe requests to backup.php straight to backend - prevents problem
> with progress bar long polling 503 problem*
> *    # This is here because backup.php is POSTing to itself - Filter
> before !GET&&!HEAD*
> *    if (req.url ~ "^/backup/backup.php")*
> *    {*
> *        return (pipe);*
> *    }*
>
> *    # Varnish only deals with GET and HEAD by default. If request method
> is not GET or HEAD, pass request to backend*
> *    if (req.method != "GET" && req.method != "HEAD") {*
> *      return (pass);*
> *    }*
>
> *    if (req.http.Cookie) {*
> *      # Remove any Google Analytics based cookies*
> *      set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");*
> *      set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");*
> *      set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
> )?", "");*
> *      set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
> )?", "");*
> *      set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(;
> )?", "");*
> *      set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");*
> *      if (req.http.Cookie ~ "^\s*$") {*
> *        unset req.http.Cookie;*
> *      }*
> *    }*
>
> * ### Rules for Moodle ###*
>
> *    # Perform lookup for selected assets that we know are static but
> Moodle still needs a Cookie*
> *    if(  req.url ~ "^/theme/.+\.?" ||*
> *     req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||*
> *         req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||*
> *         req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||*
> *     req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"*
> *      )*
> *    {*
> *         # Set internal temporary header, based on which we will do
> things in vcl_backend_response*
> *         set req.http.X-Long-TTL = "86400";*
> *         return (hash);*
> *    }*
> *    # Requests containing "Cookie" or "Authorization" headers will not be
> cached*
> *    if (req.http.Authorization || req.http.Cookie) {*
> *        return (pass);*
> *    }*
> *    # Almost everything in Moodle correctly serves Cache-Control headers,
> if*
> *    # needed, which varnish will honor, but there are some which don't.
> Rather*
> *    # than explicitly finding them all and listing them here we just fail
> safe*
> *    # and don't cache unknown urls that get this far.*
> *    return (pass);*
>
> *}*
>
> *sub vcl_backend_response {*
> *    # Set backend name*
> *    set beresp.http.X-Backend = beresp.backend.name
> <http://beresp.backend.name>;*
>
> *    if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl
> < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
> !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
> header*
> *        set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
> beresp.http.Pragma;*
> *        set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control;*
> *        set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";*
> *        set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
> *        unset bereq.http.X-Long-TTL;*
> *    }*
> *    else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
> !beresp.http.WWW-Authenticate ) {*
> *        set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
> beresp.http.Pragma;*
> *        set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";*
> *        set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
> *        unset bereq.http.X-Long-TTL;*
> *    }*
> *    else { # Don't touch headers if max-age > defined in X-Long-TTL
> header*
> *        unset bereq.http.X-Long-TTL;*
> *    }*
> *    # Here we set X-Trace header, prepending it to X-Trace header
> received from backend. Useful for troubleshooting*
> *    if (beresp.http.x-trace && !beresp.was_304) {*
> *        set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;*
> *    }*
> *    else {*
> *        set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");*
> *    }*
> *}*
>
> *sub vcl_deliver {*
>
> *# Revert back to original Cache-Control header before delivery to client*
> *    if (resp.http.X-Orig-Cache-Control)*
> *    {*
> *        set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;*
> *        unset resp.http.X-Orig-Cache-Control;*
> *    }*
> *    # Revert back to original Pragma header before delivery to client*
> *    if (resp.http.X-Orig-Pragma)*
> *    {*
> *        set resp.http.Pragma = resp.http.X-Orig-Pragma;*
> *        unset resp.http.X-Orig-Pragma;*
> *    }*
>
> *  if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
> the number of hits, disable when not needed*
> *    set resp.http.X-Cache = "HIT";*
> *  } else {*
> *    set resp.http.X-Cache = "MISS";*
> *  }*
> *    set resp.http.X-Cache-Hits = obj.hits;*
>
> *# If desired "Via: 1.1 Varnish-v4" response header can be removed from
> response*
> *    unset resp.http.Via;*
> *    unset resp.http.Server;*
>
> *    return (deliver);*
> *}*
>
> *sub vcl_backend_error {*
> *    # More comprehensive varnish error page. Display time, instance
> hostname, host header, url for easier troubleshooting.*
> *    set beresp.http.Content-Type = "text/html; charset=utf-8";*
> *    set beresp.http.Retry-After = "5";*
> *    synthetic( {"*
> *  <!DOCTYPE html>*
> *  <html>*
> *    <head>*
> *      <title>"} + beresp.status + " " + beresp.reason + {"</title>*
> *    </head>*
> *    <body>*
> *      <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>*
> *      <p>"} + beresp.reason + {"</p>*
> *      <h3>Guru Meditation:</h3>*
> *      <p>Time: "} + now + {"</p>*
> *      <p>Node: "} + server.hostname + {"</p>*
> *      <p>Host: "} + bereq.http.host + {"</p>*
> *      <p>URL: "} + bereq.url + {"</p>*
> *      <p>XID: "} + bereq.xid + {"</p>*
> *      <hr>*
> *      <p>Varnish cache server*
> *    </body>*
> *  </html>*
> *  "} );*
> *   return (deliver);*
> *}*
>
> *sub vcl_synth {*
> *    #Redirect using '301 - Permanent Redirect', permanent redirect*
> *    if (resp.status == 851) { *
> *        set resp.http.Location = req.http.x-redir;*
> *        set resp.http.X-Varnish-Redirect = true;*
> *        set resp.status = 301;*
> *        return (deliver);*
> *    }*
> *    #Redirect using '302 - Found', temporary redirect*
> *    if (resp.status == 852) { *
> *        set resp.http.Location = req.http.x-redir;*
> *        set resp.http.X-Varnish-Redirect = true;*
> *        set resp.status = 302;*
> *        return (deliver);*
> *    }*
> *    #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
> dont change method on redirected requests*
> *    if (resp.status == 857) { *
> *        set resp.http.Location = req.http.x-redir;*
> *        set resp.http.X-Varnish-Redirect = true;*
> *        set resp.status = 307;*
> *        return (deliver);*
> *    }*
> *    #Respond with 403 - Forbidden*
> *    if (resp.status == 863) {*
> *        set resp.http.X-Varnish-Error = true;*
> *        set resp.status = 403;*
> *        return (deliver);*
> *    }*
> *}*
>
> *sub vcl_purge {*
> *  if (req.method != "PURGE") {*
> *    set req.http.X-Purge = "Yes";*
> *    return (restart);*
> *  }*
> *}*
>
> ...
>
>
> Karim Ayari
> Universit? Claude Bernard - Lyon 1
> Service ICAP
> Ing?nieur syst?mes
>
> Le 20/02/2023 ? 18:55, Guillaume Quintard a ?crit :
>
> Hello Karim,
>
> You VCL would be useful to debug this (as well as the command line you are
> running Varnish with), but it sounds like Varnish is using the Transient
> storage (
> https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
> to store the file, and as the storage isn't bounded, it explodes.
> We can fix this in a couple of ways, from storing the file in the regular
> cache storage, to using pipe, to waiting a few days for
> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
> to be released.
>
> Question is: should that file be cached?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari at univ-lyon1.fr>
> wrote:
>
>> Hi!
>>
>> I am currently experiencing a memory load problem with video playback.
>>
>> here is the infrastructure :
>>
>> client --> haproxy --> varnish --> moodle workers (x5)
>>
>> a teacher uploaded a 400MB video to Moodle, when we start playing the
>> video with browser player, Varnish consumes all the memory until it runs
>> out and oom killer to kill varnishd. i have no configuration for mp4
>> files in my vcl file, so by default they are not hidden (?). I can't
>> find a solution :(
>>
>> I can give my vcl file if necessary.
>>
>> (I am a beginner on varnish :))
>>
>> thank you for your support.
>>
>> Karim
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/a31ee9fe/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/a31ee9fe/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/a31ee9fe/attachment-0003.png>

From karim.ayari at univ-lyon1.fr  Tue Feb 21 08:05:12 2023
From: karim.ayari at univ-lyon1.fr (Karim Ayari)
Date: Tue, 21 Feb 2023 09:05:12 +0100
Subject: varnish and mp4 files
In-Reply-To: <CAOrPor=zG5PkKr-_3sVnTjfFED_Xepmk5_BMBLEgosZG6r5bLw@mail.gmail.com>
References: <acbd6ad5-cac0-e5d2-864e-6256a1308ba7@univ-lyon1.fr>
 <CAOrPormuFTWqMyRn5=4YA56Ubx3Nu2a197aXE=HcZabbRfCmbA@mail.gmail.com>
 <c7a414f9-0912-632d-da3a-d7d9fb3b2f83@univ-lyon1.fr>
 <CAOrPor=zG5PkKr-_3sVnTjfFED_Xepmk5_BMBLEgosZG6r5bLw@mail.gmail.com>
Message-ID: <8a22d5e0-bfcb-abbe-6c48-11a956b1d364@univ-lyon1.fr>

sorry I thought I noted it in my previous email.

here are the good lines for using pipe :


/if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {//
//?? std.log("ispiped:true");//
//?? return (pipe);//
//}/

thank you!

Karim.



Le 20/02/2023 ? 21:50, Guillaume Quintard a ?crit?:
> Looks like Rainer replied directly to you and not to the list, would 
> you mind sharing/highlighting the fix for people having the same issue?
>
> Cheers,
>
> -- 
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari 
> <karim.ayari at univ-lyon1.fr> wrote:
>
>     thank you both for your replies.I forgot the most important
>     thing:the varnish server has 16 GB of ram and the cache is 1 GB.
>
>     the cache never seems full
>
>
>     the command used : */usr/sbin/varnishd -j unix,user=vcache -F
>     -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S
>     /etc/varnish/secret -s malloc,1g****-p http_max_hdr=96***
>
>     I had tried using pipe it didn't work, but Rainer's lines works fine.
>
>     this solution should suffice because the videos will soon have to
>     be hosted on our video platform.
>
>     /
>     /
>
>     my vcl (file found on github) :
>
>     /
>     /
>
>     /....
>     /
>
>     /sub vcl_recv {//
>     //
>     //??? # Keep client IP//
>     //??? if (req.restarts == 0) {//
>     //??????? if (req.http.x-forwarded-for) {//
>     //??????????? set req.http.X-Forwarded-For =
>     req.http.X-Forwarded-For + ", " + client.ip;//
>     //??????? } else {//
>     //?? ???? unset req.http.X-Forwarded-For;//
>     //??????????? set req.http.X-Forwarded-For = client.ip;//
>     //??????? }//
>     //??? }//
>     //
>     //??? if (req.http.X-Real-IP) {//
>     //??????? set req.http.X-Forwarded-For = req.http.X-Real-IP;//
>     //??? } else {//
>     //??????? set req.http.X-Forwarded-For = client.ip;//
>     //??? }//
>     ////
>     //??? # Only deal with "normal" types//
>     //??? if (req.method != "GET" &&//
>     //????? req.method != "HEAD" &&//
>     //????? req.method != "PUT" &&//
>     //????? req.method != "POST" &&//
>     //????? req.method != "TRACE" &&//
>     //????? req.method != "OPTIONS" &&//
>     //????? req.method != "PATCH" &&//
>     //????? req.method != "DELETE") {//
>     //?? ???? /* Non-RFC2616 or CONNECT which is weird. *///
>     //?? ?/*Why send the packet upstream, while the visitor is using a
>     non-valid HTTP method? *///
>     //?? ?return (synth(404, "Non-valid HTTP method!"));//
>     //????? }//
>     ////
>     //??? # Varnish don't mess with healthchecks//
>     //??? if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
>     "^/healthcheck.php") {//
>     //??????? return (pass);//
>     //??? }//
>     //??? # Pipe requests to backup.php straight to backend - prevents
>     problem with progress bar long polling 503 problem//
>     //??? # This is here because backup.php is POSTing to itself -
>     Filter before !GET&&!HEAD//
>     //??? if (req.url ~ "^/backup/backup.php")//
>     //??? {//
>     //??????? return (pipe);//
>     //??? }//
>     //
>     //??? # Varnish only deals with GET and HEAD by default. If
>     request method is not GET or HEAD, pass request to backend//
>     //??? if (req.method != "GET" && req.method != "HEAD") {//
>     //????? return (pass);//
>     //??? }//
>     //
>     //??? if (req.http.Cookie) {//
>     //????? # Remove any Google Analytics based cookies//
>     //????? set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$",
>     "");//
>     //????? set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$",
>     "");//
>     //????? set req.http.Cookie = regsuball(req.http.Cookie,
>     "__gads=[^;]+(; )?", "");//
>     //????? set req.http.Cookie = regsuball(req.http.Cookie,
>     "__qc.=[^;]+(; )?", "");//
>     //????? set req.http.Cookie = regsuball(req.http.Cookie,
>     "__atuv.=[^;]+(; )?", "");//
>     //????? set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*",
>     "");//
>     //????? if (req.http.Cookie ~ "^\s*$") {//
>     //??????? unset req.http.Cookie;//
>     //????? }//
>     //??? }//
>     //
>     //?### Rules for Moodle ###//
>     //
>     //??? # Perform lookup for selected assets that we know are static
>     but Moodle still needs a Cookie//
>     //??? if(? req.url ~ "^/theme/.+\.?" ||//
>     //?? ? req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||//
>     //???????? req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||//
>     //???????? req.url ~
>     "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||//
>     //?? ? req.url ~
>     "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"//
>     //????? )//
>     //??? {//
>     //???????? # Set internal temporary header, based on which we will
>     do things in vcl_backend_response//
>     //???????? set req.http.X-Long-TTL = "86400";//
>     //???????? return (hash);//
>     //??? }//
>     //??? # Requests containing "Cookie" or "Authorization" headers
>     will not be cached//
>     //??? if (req.http.Authorization || req.http.Cookie) {//
>     //??????? return (pass);//
>     //??? }//
>     //??? # Almost everything in Moodle correctly serves Cache-Control
>     headers, if//
>     //??? # needed, which varnish will honor, but there are some which
>     don't. Rather//
>     //??? # than explicitly finding them all and listing them here we
>     just fail safe//
>     //??? # and don't cache unknown urls that get this far.//
>     //??? return (pass);//
>     //
>     //}//
>     //
>     //sub vcl_backend_response {//
>     //??? # Set backend name//
>     //??? set beresp.http.X-Backend = beresp.backend.name
>     <http://beresp.backend.name>;//
>     //
>     //??? if (beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
>     beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
>     !beresp.http.WWW-Authenticate ) { # If max-age < defined in
>     X-Long-TTL header//
>     //??????? set beresp.http.X-Orig-Pragma = beresp.http.Pragma;
>     unset beresp.http.Pragma;//
>     //??????? set beresp.http.X-Orig-Cache-Control =
>     beresp.http.Cache-Control;//
>     //??????? set beresp.http.Cache-Control = "public,
>     max-age="+bereq.http.X-Long-TTL+", no-transform";//
>     //??????? set beresp.ttl = std.duration(bereq.http.X-Long-TTL +
>     "s", 1s);//
>     //??????? unset bereq.http.X-Long-TTL;//
>     //??? }//
>     //??? else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL
>     && !beresp.http.WWW-Authenticate ) {//
>     //??????? set beresp.http.X-Orig-Pragma = beresp.http.Pragma;
>     unset beresp.http.Pragma;//
>     //??????? set beresp.http.Cache-Control = "public,
>     max-age="+bereq.http.X-Long-TTL+", no-transform";//
>     //??????? set beresp.ttl = std.duration(bereq.http.X-Long-TTL +
>     "s", 1s);//
>     //??????? unset bereq.http.X-Long-TTL;//
>     //??? }//
>     //??? else { # Don't touch headers if max-age > defined in
>     X-Long-TTL header//
>     //??????? unset bereq.http.X-Long-TTL;//
>     //??? }//
>     //??? # Here we set X-Trace header, prepending it to X-Trace
>     header received from backend. Useful for troubleshooting//
>     //??? if (beresp.http.x-trace && !beresp.was_304) {//
>     //??????? set beresp.http.X-Trace = regsub(server.identity,
>     "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
>     <http://beresp.backend.name>,
>     "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;//
>     //??? }//
>     //??? else {//
>     //??????? set beresp.http.X-Trace = regsub(server.identity,
>     "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
>     <http://beresp.backend.name>,
>     "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");//
>     //??? }//
>     //}//
>     //
>     //sub vcl_deliver {//
>     //
>     //# Revert back to original Cache-Control header before delivery
>     to client//
>     //??? if (resp.http.X-Orig-Cache-Control)//
>     //??? {//
>     //??????? set resp.http.Cache-Control =
>     resp.http.X-Orig-Cache-Control;//
>     //??????? unset resp.http.X-Orig-Cache-Control;//
>     //??? }//
>     //??? # Revert back to original Pragma header before delivery to
>     client//
>     //??? if (resp.http.X-Orig-Pragma)//
>     //??? {//
>     //??????? set resp.http.Pragma = resp.http.X-Orig-Pragma;//
>     //??????? unset resp.http.X-Orig-Pragma;//
>     //??? }//
>     //
>     //? if (obj.hits > 0) { # Add debug header to see if it's a
>     HIT/MISS and the number of hits, disable when not needed//
>     //??? set resp.http.X-Cache = "HIT";//
>     //? } else {//
>     //??? set resp.http.X-Cache = "MISS";//
>     //? }//
>     //??? set resp.http.X-Cache-Hits = obj.hits;//
>     //
>     //# If desired "Via: 1.1 Varnish-v4" response header can be
>     removed from response//
>     //??? unset resp.http.Via;//
>     //??? unset resp.http.Server;//
>     //
>     //??? return (deliver);//
>     //}//
>     //
>     //sub vcl_backend_error {//
>     //??? # More comprehensive varnish error page. Display time,
>     instance hostname, host header, url for easier troubleshooting.//
>     //??? set beresp.http.Content-Type = "text/html; charset=utf-8";//
>     //??? set beresp.http.Retry-After = "5";//
>     //??? synthetic( {"//
>     //? <!DOCTYPE html>//
>     //? <html>//
>     //??? <head>//
>     //????? <title>"} + beresp.status + " " + beresp.reason + {"</title>//
>     //??? </head>//
>     //??? <body>//
>     //????? <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>//
>     //????? <p>"} + beresp.reason + {"</p>//
>     //????? <h3>Guru Meditation:</h3>//
>     //????? <p>Time: "} + now + {"</p>//
>     //????? <p>Node: "} + server.hostname + {"</p>//
>     //????? <p>Host: "} + bereq.http.host + {"</p>//
>     //????? <p>URL: "} + bereq.url + {"</p>//
>     //????? <p>XID: "} + bereq.xid + {"</p>//
>     //????? <hr>//
>     //????? <p>Varnish cache server//
>     //??? </body>//
>     //? </html>//
>     //? "} );//
>     //?? return (deliver);//
>     //}//
>     //
>     //sub vcl_synth {//
>     //??? #Redirect using '301 - Permanent Redirect', permanent redirect//
>     //??? if (resp.status == 851) { //
>     //??????? set resp.http.Location = req.http.x-redir;//
>     //??????? set resp.http.X-Varnish-Redirect = true;//
>     //??????? set resp.status = 301;//
>     //??????? return (deliver);//
>     //??? }//
>     //??? #Redirect using '302 - Found', temporary redirect//
>     //??? if (resp.status == 852) { //
>     //??????? set resp.http.Location = req.http.x-redir;//
>     //??????? set resp.http.X-Varnish-Redirect = true;//
>     //??????? set resp.status = 302;//
>     //??????? return (deliver);//
>     //??? }//
>     //??? #Redirect using '307 - Temporary Redirect', !GET&&!HEAD
>     requests, dont change method on redirected requests//
>     //??? if (resp.status == 857) { //
>     //??????? set resp.http.Location = req.http.x-redir;//
>     //??????? set resp.http.X-Varnish-Redirect = true;//
>     //??????? set resp.status = 307;//
>     //??????? return (deliver);//
>     //??? }//
>     //??? #Respond with 403 - Forbidden//
>     //??? if (resp.status == 863) {//
>     //??????? set resp.http.X-Varnish-Error = true;//
>     //??????? set resp.status = 403;//
>     //??????? return (deliver);//
>     //??? }//
>     //}//
>     //
>     //sub vcl_purge {//
>     //? if (req.method != "PURGE") {//
>     //??? set req.http.X-Purge = "Yes";//
>     //??? return (restart);//
>     //? }//
>     //}/
>
>     ...
>
>
>     Le 20/02/2023 ? 18:55, Guillaume Quintard a ?crit?:
>>     Hello Karim,
>>
>>     You VCL would be useful to debug this (as well as the command
>>     line you are running Varnish with), but it sounds like Varnish is
>>     using the Transient storage
>>     (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
>>     to store the file, and as the storage isn't bounded, it explodes.
>>     We can fix this in a couple of ways, from storing the file in the
>>     regular cache storage, to using pipe, to waiting a few days for
>>     https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
>>     to be released.
>>
>>     Question is: should that file be cached?
>>
>>     Cheers,
>>
>>     -- 
>>     Guillaume Quintard
>>
>>
>>     On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari
>>     <karim.ayari at univ-lyon1.fr> wrote:
>>
>>         Hi!
>>
>>         I am currently experiencing a memory load problem with video
>>         playback.
>>
>>         here is the infrastructure :
>>
>>         client --> haproxy --> varnish --> moodle workers (x5)
>>
>>         a teacher uploaded a 400MB video to Moodle, when we start
>>         playing the video with browser player, Varnish consumes all
>>         the memory until it runs out and oom killer to kill
>>         varnishd.i have no configuration for mp4 files in my vcl
>>         file, so by default they are not hidden (?). I can't find a
>>         solution :(
>>
>>         I can give my vcl file if necessary.
>>
>>         (I am a beginner on varnish :))
>>
>>         thank you for your support.
>>
>>         Karim
>>
>>         _______________________________________________
>>         varnish-misc mailing list
>>         varnish-misc at varnish-cache.org
>>         https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4295 bytes
Desc: Signature cryptographique S/MIME
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0001.bin>