From guillaume.quintard at gmail.com Mon Nov 6 06:34:16 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sun, 5 Nov 2023 22:34:16 -0800 Subject: A plea for a more useful and discoverable built-in VCL Message-ID: Hi everybody! A bunch of questions I regularly get regarding Varnish behavior revolve around the built-in vcl, mainly, I get one of these three: - why is Varnish not caching? - how come something is happening in vcl_X even though I don't have it in my vcl? - what on earth is that built-in vcl you are talking about? As usual, I have a half-baked solution with a bunch of problems, which will hopefully inspire smarter people to fix the issue properly. What I came up with is here: https://github.com/varnish/toolbox/tree/verbose_builtin/vcls/verbose_builtin Essentially, use std.log() to explain what the built-in code is doing. At the moment, it's a purely opt-in solution, meaning that you need to know about builtin.vcl to find it, which doesn't really help with discoverability, but I intend on including that code in the docker image, which should raise awareness a bit. The absolute best in my mind would be to have something similar in core, but I can see how importing std would be a hurdle. Maybe as part of packaging, we could include that file in the provided default.vcl? I dismissed the performance penalty of printing a few more lines as negligible, but I could be wrong about that. There's also the question of phrasing, so we can have a message that is concise but also gives enough information to debug the behavior. But that's very minor, and the least of our worries here. Thoughts? -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Nov 8 08:20:56 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 8 Nov 2023 08:20:56 +0000 Subject: A plea for a more useful and discoverable built-in VCL In-Reply-To: References: Message-ID: On Mon, Nov 6, 2023 at 6:36?AM Guillaume Quintard wrote: > > Hi everybody! > > A bunch of questions I regularly get regarding Varnish behavior revolve around the built-in vcl, mainly, I get one of these three: > - why is Varnish not caching? > - how come something is happening in vcl_X even though I don't have it in my vcl? > - what on earth is that built-in vcl you are talking about? > > As usual, I have a half-baked solution with a bunch of problems, which will hopefully inspire smarter people to fix the issue properly. > > What I came up with is here: https://github.com/varnish/toolbox/tree/verbose_builtin/vcls/verbose_builtin > Essentially, use std.log() to explain what the built-in code is doing. > > At the moment, it's a purely opt-in solution, meaning that you need to know about builtin.vcl to find it, which doesn't really help with discoverability, but I intend on including that code in the docker image, which should raise awareness a bit. > The absolute best in my mind would be to have something similar in core, but I can see how importing std would be a hurdle. Maybe as part of packaging, we could include that file in the provided default.vcl? > > I dismissed the performance penalty of printing a few more lines as negligible, but I could be wrong about that. I think we should rather have new VSL tags for the call action in VCL (calling and returning from a non-step subroutine) because this way if they really consume too many resources (vsl_buffer, vsl_space, etc) they could easily be masked. For example: - VCL_sub_call - VCL_sub_return I would probably not log subroutines called by VMODs and leave that as an exercise to VMOD authors. > There's also the question of phrasing, so we can have a message that is concise but also gives enough information to debug the behavior. But that's very minor, and the least of our worries here. > > Thoughts? If you look at the VCL reference documentation between 6.0 and 7.4 you may notice that we went from a monolithic manual to a bunch of them: https://varnish-cache.org/docs/6.0/reference/ https://varnish-cache.org/docs/7.4/reference/ One thing I've wanted to do for a while is to add a vcl-builtin(7) manual, and I think even a small synopsis and a dump of bin/varnishd/builtin.vcl would improve discoverability. I would however prefer to have something smarter than that, but haven't given enough thoughts on how to proceed. For example, the vcl-step(7) and vcl-var(7) manual are authoritative, so updating them changes Varnish's behavior. We could imagine having a vcl-builtin(7) manual where we document and put individual subroutines authoritative snippets, and generate bin/varnishd/builtin.vcl from that. Cheers