From phk at phk.freebsd.dk Mon Jan 8 08:10:00 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 08 Jan 2018 08:10:00 +0000 Subject: this week... Message-ID: <80374.1515399000@critter.freebsd.dk> Happy New Year everybody! This is just a heads-up that life will be a bit crazy at casa-Kamp this week: I have builders running around mon+tue adding a small partition wall and wednesday we get two new kittens to replace our old cat, which had to be put down dec. 30th. At bugwash today we should touch base about all the stuff we wanted for the march 15 release, but havn't finished yet... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Fri Jan 12 22:33:57 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 12 Jan 2018 22:33:57 +0000 Subject: hosed vtest machines Message-ID: <47940.1515796437@critter.freebsd.dk> Those of you running VTEST machines probably have a bunch of varnishtop, hist and stat processes eating up CPU. Please kill them in whatever covenient way you prefer. I found one bug causing it, but more may appear now that I have started testing curses mode of these programs in varnishtest. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Sat Jan 13 00:39:06 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 13 Jan 2018 00:39:06 +0000 Subject: Things you didn't want to know about curses(3) Message-ID: <77435.1515803946@critter.freebsd.dk> I just spent far too much time finding out why the new terminal support in varnishtest::process left hanging varnish{hist|stat|top} processes on test-failure. Remembering the follwing bit of learned wisdom may save you time in a somewhat strange set of circumstances: In curses(3) the wgetch(3) function both returns ERR for timeouts, (if configured with timeout(3)), and for EOF on stdin. In other words, EOF on stdin had better be accompanied by SIGHUP or your cursed program will just spin if it uses timeout(3). One would think somebody had fixed that by now... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Tue Jan 30 11:17:36 2018 From: slink at schokola.de (Nils Goroll) Date: Tue, 30 Jan 2018 12:17:36 +0100 Subject: calling for a new workspace sizing concept - Re: auto-tune ws / headroom poc In-Reply-To: <56D43E68.60606@schokola.de> References: <56D43E68.60606@schokola.de> Message-ID: Hi, we've run into a Workspace overflow in VFP_Push() and basically wasted a couple of hours on exactly the same thing which I had raised almost two years ago: workspace_backend is by far not the amount of space available for VCL and it is anything but obvious to users that it also needs to accommodate the following: (size estimates are for default parameters on 64bit) - bereq0, bereq, beresp: 3 * HTTP_estimate(cache_param->http_max_hdr) = 3 * (sizeof (struct http) + sizeof(txt) * cache_param->http_max_hdr + cache_param->http_max_hdr) ~ 3 * (56 + 16 * 64 + 64) - cache_param->vsl_buffer 4k - cache_param->http_resp_size 32k So, for the default parameters, ~40k of our 64k are already planned to be used by functions unrelated to VCL, the backend request and VFPs. If users tune up any of the three parameters http_max_hdr, vsl_buffer and http_resp_size, they are very likely to run into out-of-workspace conditions and start getting backend 503s. I'm tempted to docfix this, but even telling users the HTTP_estimate formula seems just plain wrong. Similar problems exists on the client side. IMHO, we really should replace the concept of tuning workspaces by tuning headroom to be available for VCL and other dynamic allocations (like VFPs). As my previous attempt to tackle this did not raise much attention, I would want to avoid wasting time on patches before reaching some kind of agreement on this suggestion or any other sensible solution: * forbid any nested sizing parameters as in "if you tune knob a, you also need to tune knob b" -> knob b should be replaced by a knob b' which is independent of knob a and any derivation should be calculated automatically. * make workspace sizes read only parameters which users can query to estimate their memory consumption * replace the workspace tunables with xxx_headroom tuning just the space to remain available on the workspace after deduction of all known allocations Thanks, Nils On 29/02/16 13:49, Nils Goroll wrote: > Hi, > > following up an attempt to discuss this on irc: > > > The following discussion is, for the time being, regarding workpace_backend > only, but the question is relevant for the other workspaces also: > > Relevant portions of workspace_backend are not available for use in VCL, but are > consumed in core code. Setting workspace_backend too low will trigger an > immediate assertion failure in VBO_GetBusyObj(). > > IMHO, it doesn't make sense to handle these assertions differently, because > space for http headers, vsl and the initial read from the backend are mandatory > for varnish to issue backend requests. > > Instead of trying to better handle nonsense workspace_backend sizes, I'd suggest > to make it impossible to tune it stupidly in the first place - and to give > admins a tunable which is easier to understand. > > The attached patch contains a PoC (this is _not_ a finished patch, see below) > suggesting the following changes: > > - make workspace_backend a protected (read-only) parameter > > - add headroom_backend as a new admin-tunable parameter which roughly equals to > "workspace available to VCL" > > - calculate the size of workspace_backend based on all other relevant parameters > > Notes on the patch: > > - I've kept the workspace parameter as protected so the mempool code can stay > as is (pointing to a single parameter for sizing) > > - in VBO_Init() I've taken a lightweight approach at the consistency issue from > parameter changes, and I'd be curious to hear if others agree or think > that we need to make this safer. > > - what is the best way to cross the border between the cache and mgt code? > > Originally I would have preferred to have the sizing code in cache_busyobj.c, > but then we'd need to make mgt_param visible there - which I didn't like. > > But what I have in the patch now doesn't look clean at all, I am really > unsure at this point. > > Thx, Nils > > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > From dridi at varni.sh Tue Jan 30 13:08:00 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 30 Jan 2018 14:08:00 +0100 Subject: calling for a new workspace sizing concept - Re: auto-tune ws / headroom poc In-Reply-To: References: <56D43E68.60606@schokola.de> Message-ID: On Tue, Jan 30, 2018 at 12:17 PM, Nils Goroll wrote: > * forbid any nested sizing parameters as in "if you tune knob a, you also need > to tune knob b" -> knob b should be replaced by a knob b' which is independent > of knob a and any derivation should be calculated automatically. > > * make workspace sizes read only parameters which users can query to estimate > their memory consumption > > * replace the workspace tunables with xxx_headroom tuning just the space > to remain available on the workspace after deduction of all known allocations Hi, Instead to new xxx_headroom knobs, why not recycle the existing workspace_xxx parameters to take their values _in addition to_ related parameters and maybe document it as such? This way the description could tell that workspace_client is the space allocated to VCL processing on the client side, and possibly (would we need to?) mention that the total workspace size of a client transaction is "this much". Knowing the formula would help capacity planning, so documenting it somewhere sounds sensible all things considered. I overall agree that we should prevent users from getting a configuration that guarantees transactions failures. Dridi From slink at schokola.de Tue Jan 30 13:11:46 2018 From: slink at schokola.de (Nils Goroll) Date: Tue, 30 Jan 2018 14:11:46 +0100 Subject: calling for a new workspace sizing concept - Re: auto-tune ws / headroom poc In-Reply-To: References: <56D43E68.60606@schokola.de> Message-ID: <00b4a834-471b-4dd4-548e-76f21d1a7b52@schokola.de> On 30/01/18 14:08, Dridi Boukelmoune wrote: > Instead to new xxx_headroom knobs, why not recycle the existing workspace_xxx > parameters to take their values _in addition to_ related parameters and maybe > document it as such? I wouldn't oppose to this option, but this would lead to a significant increase of actual workspace sizes for those users who are unaware of the change. Using new parameter names and making the old ones read-only would raise awareness for the change. My preference would be the latter, but I'm ok with the former if we reach consensus. Nils From dridi at varni.sh Tue Jan 30 13:15:40 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 30 Jan 2018 14:15:40 +0100 Subject: calling for a new workspace sizing concept - Re: auto-tune ws / headroom poc In-Reply-To: <00b4a834-471b-4dd4-548e-76f21d1a7b52@schokola.de> References: <56D43E68.60606@schokola.de> <00b4a834-471b-4dd4-548e-76f21d1a7b52@schokola.de> Message-ID: On Tue, Jan 30, 2018 at 2:11 PM, Nils Goroll wrote: > On 30/01/18 14:08, Dridi Boukelmoune wrote: >> Instead to new xxx_headroom knobs, why not recycle the existing workspace_xxx >> parameters to take their values _in addition to_ related parameters and maybe >> document it as such? > > I wouldn't oppose to this option, but this would lead to a significant increase > of actual workspace sizes for those users who are unaware of the change. Agreed, but in both cases we'd need to change our defaults to match the change and users should read release notes before upgrading :) > Using new parameter names and making the old ones read-only would raise > awareness for the change. > > My preference would be the latter, but I'm ok with the former if we reach consensus. > > Nils From phk at phk.freebsd.dk Tue Jan 30 13:18:55 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 30 Jan 2018 13:18:55 +0000 Subject: calling for a new workspace sizing concept - Re: auto-tune ws / headroom poc In-Reply-To: References: <56D43E68.60606@schokola.de> Message-ID: <51243.1517318335@critter.freebsd.dk> -------- Let me just lay down a ground rule here: 1. I think this is an important thing to (re)consider, by all means go at it. 2. No commits related to this until after march 15th,. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Wed Jan 31 00:48:42 2018 From: fgsch at lodoss.net (Federico Schwindt) Date: Wed, 31 Jan 2018 00:48:42 +0000 Subject: calling for a new workspace sizing concept - Re: auto-tune ws / headroom poc In-Reply-To: <51243.1517318335@critter.freebsd.dk> References: <56D43E68.60606@schokola.de> <51243.1517318335@critter.freebsd.dk> Message-ID: I prefer less knobs so I'm inclined to agree with Dridi's suggestion. I'd propose to make the math but keep the final workspaces' numbers as they are and bump them after the release. That would keep the footprint similar to what it is today and we could document the change in advance (and warn about the bump). On Tue, Jan 30, 2018 at 1:18 PM, Poul-Henning Kamp wrote: > -------- > > Let me just lay down a ground rule here: > > 1. I think this is an important thing to (re)consider, by all means > go at it. > > 2. No commits related to this until after march 15th,. > > Poul-Henning > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: