When I first started working with Varnish, my only concerns were mostly
configuration and a bit of administration. Depending on your needs (which can
impressively become complex sometimes) you can reach the limits of Varnish in
terms of features (especially if your needs are not directly linked to HTTP
caching). In this case, you will probably need a module or create your own, and
I've already shown you how to
make one. I already knew
that Varnish comes with a test framework, but I didn't expect it could also
enable TDD for VMODs. I was lost when I first tried to read a test case, but I
quickly found that the varnishtest framework is quite powerful and easy to
use!
A test framework ?
When I stated earlier that Varnish comes with a test framework, I meant that you
do have the varnishtest program when you install Varnish. It's not just a
tool for the Varnish development team, nor a tool for VMOD creators. It's also a
tool you can use to reproduce a bug when you file a bug report. If you look at
the tests in Varnish's source tree, you can see a README:
Test-scripts for varnishtest
============================
Naming scheme
-------------
The intent is to be able to run all scripts in lexicographic
order and get a sensible failure mode.
This requires more basic tests to be earlier and more complex
tests to be later in the test sequence, we do this with the
prefix/id letter:
[id]%05d.vtc
id ~ [a] --> varnishtest(1) tests
id ~ [b] --> Basic functionality tests
id ~ [c] --> Complex functionality tests
id ~ [e] --> ESI tests
id ~ [g] --> GZIP tests
id ~ [m] --> VMOD tests
id ~ [p] --> Persistent tests
id ~ [r] --> Regression tests, same number as ticket
id ~ [s] --> Slow tests, expiry, grace etc.
id ~ [t] --> sTreaming tests
id ~ [v] --> VCL tests: execute VRT functions
As of Varnish 3.0.3, released a few days ago, there are 294 tests you can study
before writing your own. And today I'll save you some time by showing the
important parts of the VTC language.
The Varnish Test Case language
Many things in Varnish's architecture are impressive for me. The design of a
DSL for the configuration is one example, but I
didn't expect I would find another DSL for testing when I first used with
Varnish. The VTC is really easy to both write and read, but you need a small
mental shift (at least I needed) because it doesn't follow the
set up/test/assert/tear down or given/when/then patterns. Depending on
your scenario, there might be test preparations, executions and assertions all
over the place. Unlike the VCL, the VTC is not compiled but simply interpreted
on the fly.
First of all, a test has a name:
varnishtest "Test example vmod"
Starting a varnishd instance
The thing about varnishtest is that it launches a real Varnish instance and
interacts with it. It launches the varnishd instance, controls it through
the management process and makes assertions based on the shared memory logs.
It makes tests really slow (like a hundred milliseconds slow every time an
expected value has yet to be found) but it actually tests the real product.
The Varnish instance needs a name. You can optionally configure it or just rely
on the default VCL. If you are testing a VMOD, you can import it with an
absolute file name.
varnish v1 -vcl+backend {
import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
sub vcl_deliver {
set resp.http.hello = example.hello("World");
}
} -start
The instance can be started later...
varnish v1 -vcl+backend {
import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
sub vcl_deliver {
set resp.http.hello = example.hello("World");
}
}
[...]
varnish v1 -start
If you fork the libvmod-example
module, your autotools configuration will run varnishtest with the
vmod_topbuild macro. It works like Java system properties in the command
line: -Dkey=value.
The -vcl+backend directive automatically injects the backend into the VCL,
which most of the time is what you want to do. Sometimes you might have several
backends (load balanced for instance) so you might want to do it manually:
varnish v1 -vcl {
import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
# you'll learn how to run a "s1" server
backend default {
.host = "${s1_addr}";
.port = "${s1_port}";
}
sub vcl_deliver {
set resp.http.hello = example.hello("World");
}
} -start
Now let's learn how varnishtest can mock backends and run clients.
Mocking the backend
The syntax for backends and clients is similar to the syntax for Varnish
instances except that instead of VCL, you'll find actions and assertions.
server s1 {
rxreq
txresp
} -start
In this case, we are starting a server which answers a response to exactly one
request. It could also answer to more requests, it doesn't matter. It would be a
problem if the server received more requests than expected, it would timeout and
the Varnish instance would send a 503 Unavailable response (yet, this is a
behaviour you might need to test). Those can be seen as actions: receiving a
request (rx) and transmitting (tx) a response.
Answering up to two requests:
server s1 {
rxreq
txresp
rxreq
txresp
} -start
In addition to actions, you can add assertions. You can expect the server to
receive a request with a given url, a specific header... Remember that your
client's request goes through a Varnish instance first and the backend request
might change in some way you want to check.
server s1 {
rxreq
expect req.url == "/"
txresp
} -start
It is of course possible to change the empty default response depending on your
use cases.
server s1 {
rxreq
expect req.url == "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
txresp -body "A great article :p"
rxreq
expect req.url == "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
txresp -body "The second part ;)"
} -start
Once you start a server named s1, macros are automatically made available:
${s1_addr} : the IP address (127.0.0.1)${s1_port} : the port (randomly chosen)${s1_sock} : the socket
It is also possible to use a real backend instead of just a mock. Remember that
varnishtest launches a real varnishd instance. You could integrate
Varnish in your CI environment and run your own
integration or regression tests against your actual application.
Running client requests
Once you have a server and a Varnish, you can start running clients and sending
requests. The syntax is the same as for the servers, but instead of expecting
requests and answering responses, clients send requests and expect responses.
client c1 {
txreq -url "/"
rxresp
expect resp.http.hello == "Hello, World"
} -run
And it's easy to run the same client's scenario multiple times:
client c1 {
txreq
rxresp
}
[...]
client c1 -run
client c1 -run
client c1 -run
And of course a single client can send several requests:
clent c1 {
txreq -url "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
rxresp
expect resp.bodylen == 18
txreq -url "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
rxresp
expect resp.bodylen == 18
} -run
We've seen that servers and clients are made to embed a scenario with both
actions and expectations, embeded in the server or client body. Since a Varnish
instance's body contains VCL, is it possible to add assertions ? The answer is
yes.
Varnish assertions
Unlike clients or servers, Varnish assertions are put outside the instance's
declaration body. What can you actually test with Varnish ? Since it's a proxy
to your backend, checking requests and responses is irrelevant. Instead, you
have access to the counters exposed by varnishstat at any time (it doesn't
mean that all counters are relevant for testing purpose).
varnish v1 -expect the_counter == the_value
Now, how does it look like when you write a test case that does all this ?
A more advanced example
varnishtest "Test example vmod even more"
server s1 {
rxreq
expect req.url == "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
txresp -body "A great article :p"
rxreq
expect req.url == "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
txresp -body "The second part ;)"
} -start
varnish v1 -vcl+backend {
import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
sub vcl_deliver {
set resp.http.hello = example.hello("World");
}
} -start
client c1 {
txreq -url "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
rxresp
expect resp.http.hello == "Hello, World"
expect resp.bodylen == 18
txreq -url "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
rxresp
expect resp.http.hello == "Hello, World"
expect resp.bodylen == 18
}
varnish v1 -expect cache_miss == 0
varnish v1 -expect cache_hit == 0
client c1 -run
varnish v1 -expect cache_miss == 2
varnish v1 -expect cache_hit == 0
client c1 -run
client c1 -run
client c1 -run
varnish v1 -expect cache_miss == 2
varnish v1 -expect cache_hit == 6
Conclusion
Test preparations, executions and assertions all over the place, didn't I warn
you ? Yes I did, however, it doesn't mean that you can't write readable tests.
On the contrary, the varnishtest DSL is really expressive, especially once
you know how to read it. It's also easy to write, even though your test case is
exploded (servers + varnish + clients), the isolation of those components makes
it simpler IMHO. I should probably mention that varnishtest is not suited
for performance tests by design.
This is only an introduction to varnishtest but I hope this will answer
questions you might ask yourself when writing your first test cases.
David Harrigan
