[master] 3efdc84bc doc: First stab at built-in VCL split documentation

[Nils Goroll]

commit 3efdc84bc923e1d11f57a41577a8e6e0f0a2a7d1
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Fri Jan 29 20:16:47 2021 +0100

    doc: First stab at built-in VCL split documentation

diff --git a/doc/sphinx/users-guide/vcl-built-in-code.rst b/doc/sphinx/users-guide/vcl-built-in-code.rst
new file mode 100644
index 000000000..2bd72ed50
--- /dev/null
+++ b/doc/sphinx/users-guide/vcl-built-in-code.rst
@@ -0,0 +1,87 @@
+.. _vcl-built-in-code:
+
+Built-in VCL
+============
+
+Whenever a VCL program is loaded, the built-in VCL is appended to it. The
+:ref:`vcl-built-in-subs` have a special property, they can appear multiple
+times and the result is concatenation of all built-in subroutines.
+
+For example, let's take the following snippet::
+
+    sub vcl_recv {
+        # loaded code for vcl_recv
+    }
+
+The effective VCL that is supplied to the compiler looks like::
+
+    sub vcl_recv {
+        # loaded code for vcl_recv
+        # built-in code for vcl_recv
+    }
+
+This is how it is guaranteed that all :ref:`reference-states` have at least
+one ``return (<action>)``.
+
+It is generally recommended not to invariably return from loaded code to
+let Varnish execute the built-in code, because the built-in code provides
+essentially a sensible default behavior for an HTTP cache.
+
+Built-in subroutines split
+--------------------------
+
+It might however not always be practical that the built-in VCL rules take
+effect at the very end of a state, so some subroutines like ``vcl_recv``
+are split into multiple calls to other subroutines.
+
+By convention, those assistant subroutines are named after the variable
+they operate on, like ``req`` or ``beresp``. This allows for instance to
+circumvent default behavior.
+
+For example, ``vcl_recv`` in the built-in VCL prevents caching when clients
+have a cookie. If you can trust your backend to always specify whether a
+response is cacheable or not regardless of whether the request contained a
+cookie you can do this::
+
+    sub vcl_req_cookie {
+        return;
+    }
+
+With this, all other default behaviors from the built-in ``vcl_recv`` are
+executed and only cookie handling is affected.
+
+Another example is how the built-in ``vcl_backend_response`` treats a
+negative TTL as a signal not to cache. It's a historical mechanism to mark
+a response as uncacheable, but only if the built-in ``vcl_backend_response``
+is not circumvented by a ``return (<action>)``.
+
+However, in a multi-tier architecture where a backend might be another
+Varnish server, you might want to cache stale responses to allow the
+delivery of graced objects and enable revalidation on the next fetch. This
+can be done with the following snippet::
+
+    sub vcl_beresp_stale {
+        if (beresp.ttl + beresp.grace > 0s) {
+            return;
+        }
+    }
+
+This granularity, and the general goal of the built-in subroutines split
+is to allow to circumvent a specific aspect of the default rules without
+giving the entire logic up.
+
+Built-in VCL reference
+----------------------
+
+A copy of the ``builtin.vcl`` file might be provided with your Varnish
+installation but :ref:`varnishd(1)` is the reference to determine the code
+that is appended to any loaded VCL.
+
+The VCL compilation happens in two passes:
+
+- the first one compiles the built-in VCL only,
+- and the second pass compiles the concatenation of the loaded and built-in
+  VCLs.
+
+Any VCL subroutine present in the built-in VCL can be extended, in which
+case the loaded VCL code will be executed before the built-in code.
diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index 2f46d8b87..b614a8428 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -5,7 +5,7 @@
 
 .. _vcl-built-in-subs:
 
-Built in subroutines
+Built-in subroutines
 ====================
 
 Various built-in subroutines are called during processing of client-
@@ -25,6 +25,8 @@ Common actions are documented in
 :ref:`user-guide-vcl_actions`. Actions specific to only one or some
 subroutines are documented herein.
 
+A default behavior is provided for all :ref:`reference-states` in the
+:ref:`vcl-built-in-code` code.
 
 client side
 -----------
diff --git a/doc/sphinx/users-guide/vcl-syntax.rst b/doc/sphinx/users-guide/vcl-syntax.rst
index b3334a4b7..2cc6fc6a5 100644
--- a/doc/sphinx/users-guide/vcl-syntax.rst
+++ b/doc/sphinx/users-guide/vcl-syntax.rst
@@ -87,13 +87,15 @@ down for, uhm, examples.
 Built in subroutines
 ~~~~~~~~~~~~~~~~~~~~
 
-Varnish has quite a few built in subroutines that are called for each
-transaction as it flows through Varnish. These builtin subroutines are all
-named ``vcl_*`` and are explained in :ref:`vcl-built-in-subs`.
+Varnish has quite a few built-in subroutines that are called for each
+transaction as it flows through Varnish. These built-in subroutines are
+all named ``vcl_*`` and are explained in :ref:`vcl-built-in-subs`.
 
-Processing in built in subroutines ends with ``return (<action>)``
+Processing in built-in subroutines ends with ``return (<action>)``
 (see :ref:`user-guide-vcl_actions`).
 
+The :ref:`vcl-built-in-code` also contains custom assistant subroutines
+called by the built-in subroutines, also prefixed with ``vcl_``.
 
 Custom subroutines
 ~~~~~~~~~~~~~~~~~~
diff --git a/doc/sphinx/users-guide/vcl.rst b/doc/sphinx/users-guide/vcl.rst
index 57e6f452b..831a24071 100644
--- a/doc/sphinx/users-guide/vcl.rst
+++ b/doc/sphinx/users-guide/vcl.rst
@@ -41,6 +41,7 @@ code commented out in the file `builtin.vcl` that ships with Varnish Cache.
 
    vcl-syntax
    vcl-built-in-subs
+   vcl-built-in-code
    vcl-variables
    vcl-actions
    vcl-backends