Planet Varnish

November 24, 2015

Kristian LyngstølVarnish Foo - Introduction

Posted on 2015-11-24

This is the only chapter written in first person.

I've worked on Varnish since late 2008, first for Redpill Linpro, then Varnish Software, then, after a brief pause, for Redpill Linpro again. Over the years I've written code, written Varnish modules and blog posts, tried to push the boundaries of what Varnish can do, debugged or analyzed countless Varnish sites, probably held more training courses than anyone else, written training material, and helped shape the Varnish community.

Today I find myself in a position where the training material I once maintained is no longer my responsibility. But I still love writing, and there's an obvious need for documentation for Varnish.

I came up with a simple solution: I will write a book. Because I couldn't imagine that I would ever finish it if I attempted writing a whole book in one go, I decided I would publish one chapter at a time on my blog. This is the first chapter of that book.

You will find the source on This is something I am doing on my spare time, and I hope to get help from the Varnish community in the form of feedback. While the format will be that of a book, I intend to keep it alive with updates as long as I can.

I intend to cover as much Varnish-related content as possible, from administration to web development and infrastructure. And my hope is that one day, this will be good enough that it will be worth printing as more than just a leaflet.

I am writing this in my spare time, I retain full ownership of the material. For now, the material is available under a Creative Commons "CC-BY-SA-NC" license. The NC-part of that license will be removed when I feel the material has matured enough and the time is right. To clarify, the "non-commercial" clause is aimed at people wanting to sell the book or use it in commercial training (or similar) - it is not intended to prevent you from reading the material at work.

Target audience and format

This book covers a large spectre of subjects related to Varnish. It is suitable for system administrators, infrastructure architects and web developers. The first few chapters is general enough to be of interest to all, while later chapters specialize on certain aspects of Varnish usage.

Each chapter is intended to stand well on its own, but there will be some cross-references. The book focuses on best practices and good habits that will help you beyond what just a few examples or explanations will do.

Each chapter provides both theory and practical examples. Each example is tested with a recent Varnish Version where relevant, and are based on experience from real-world Varnish installations.

What is Varnish

Varnish is a web server.

Unlike most web servers, Varnish does not read content from a hard drive, or run programs that generates content from SQL databases. Varnish acquires the content from other web servers. Usually it will keep a copy of that content around in memory for a while to avoid fetching the same content multiple times, but not necessarily.

There are numerous reasons you might want Varnish:

  1. Your web server/application is a beastly nightmare where performance is measured in page views per hour - on a good day.
  2. Your content needs to be available from multiple geographically diverse locations.
  3. Your web site consists of numerous different little parts that you need to glue together in a sensible manner.
  4. Your boss bought a service subscription and now has to justify the budget post.
  5. You like Varnish.
  6. ???

Varnish is designed around two simple concepts: Give you the means to fix or work around technical challenges. And speed. Speed was largely handled very early on, and Varnish is quite simply fast. This is achieved by being, at the core, simple. The less you have to do for each request, the more requests you can handle.

The name suggests what it's all about:

From The Collaborative International Dictionary of English v.0.48 [gcide]:

  Varnish \Var"nish\, v. t. [imp. & p. p. {Varnished}; p. pr. &
     vb. n. {Varnishing}.] [Cf. F. vernir, vernisser. See
     {Varnish}, n.]
     [1913 Webster]
     1. To lay varnish on; to cover with a liquid which produces,
        when dry, a hard, glossy surface; as, to varnish a table;
        to varnish a painting.
        [1913 Webster]

     2. To cover or conceal with something that gives a fair
        appearance; to give a fair coloring to by words; to gloss
        over; to palliate; as, to varnish guilt. "Beauty doth
        varnish age." --Shak.
        [1913 Webster]

Varnish can be used to smooth over rough edges in your stack, to give a fair appearance.


The Varnish project began in 2005. The issue to be solved was that of VG, a large Norwegian news site (or alternatively a tiny international site). The first release came in 2006, and worked well for pretty much one site: In 2008, Varnish 2.0 came, which opened Varnish up to more sites, as long as they looked and behaved similar to As time progressed and more people started using Varnish, Varnish has been adapted to a large and varied set of use cases.

From the beginning, the project was administered through Redpill Linpro, with the majority of development being done by Poul-Henning Kamp through his own company and his Varnish Moral License. In 2010, Varnish Software sprung out from Redpill Linpro. Varnish Cache has always been a free software project, and while Varnish Software has been custodians of the infrastructure and large contributors of code and cash, the project is independent.

Varnish Plus was born some time during 2011, all though it didn't go by that name at the time. It was the result of somewhat conflicting interests. Varnish Software had customer obligations that required features, and the development power to implement them, but they did not necessarily align with the goals and time frames of Varnish Cache. Varnish Plus became a commercial test-bed for features that were not /yet/ in Varnish Cache for various reasons. Many of the features have since trickled into Varnish Cache proper in one way or an other (streaming, surrogate keys, and more), and some have still to make it. Some may never make it. This book will focus on Varnish Cache proper, but will reference Varnish Plus where it makes sense.

With Varnish 3.0, released in 2011, varnish modules started becoming a big thing. These are modules that are not part of the Varnish Cache code base, but are loaded at run-time to add features such as cryptographic hash functions (vmod-digest) and memcached. The number of vmods available grew quickly, but even with Varnish 4.1, the biggest issue with them were that they required source-compilation for use. That, however, is being fixed almost as I am writing this sentence.

Varnish would not be where it is today without a large number of people and businesses. Varnish Software have contributed and continues to contribute numerous tools, vmods, and core features. Poul-Henning Kamp is still the gatekeeper of Varnish Cache code, for better or worse, and does the majority of the architectural work. Over the years, there have been too many companies and individuals involved to list them all in a book, so I will leave that to the official Varnish Cache project.

Today, Varnish is used by CDNs and news papers, APIs and blogs.

More than just cache

Varnish caches content, but can do much more. In 2008, it was used to rewrite URLs, normalize HTTP headers and similar things. Today, it is used to implement paywalls (whether you like them or not), API metering, load balancing, CDNs, and more.

Varnish has a powerful configuration language, the Varnish Configuration Language (VCL). VCL isn't parsed the traditional way a configuration file is, but is translated to C code, compiled and linked into the running Varnish. From the beginning, it was possible to bypass the entire translation process and provide C code directly, which was never recommended. With Varnish modules, it's possible to write proper modules to replace the in-line C code that was used in the past.

There is also a often overlooked Varnish agent that provides a HTTP REST interface to managing Varnish. This can be used to extract metrics, review or optionally change configuration, stop and start Varnish, and more. The agent lives on, and is packaged for most distributions today. There's also a commercial administration console that builds further on the agent.

Using Varnish to gracefully handle operational issues is also common. Serving cached content past its expiry time while a web server is down, or switching to a different server, will give your users a better browsing experience. And in a worst case scenario, at least the user can be presented with a real error message instead of a refused or timed out connection.

An often overlooked feature of Varnish is Edge Side Includes. This is a means to build a single HTTP object (like a HTML page) from multiple smaller object, with different caching properties. This lets content writers provide more fine-grained caching strategies without having to be too smart about it.

Where to get help

The official varnish documentation is available both as manual pages (run man -k varnish on a machine with a properly installed Varnish package), and as Sphinx documentation found under

Varnish Software has also publish their official training material, which is called "The Varnish Book" (Not to be confused with THIS book about Varnish). This is available freely through their site at, after registration.

An often overlooked source of information for Varnish are the flow charts/dot-graphs used to document the VCL state engine. The official location for this is only found in the source code of Varnish, under doc/graphviz/. They can be generated simply, assuming you have graphviz installed:

# git clone
Cloning into 'Varnish-Cache'...
# cd Varnish-Cache/
# cd doc/graphviz/
# for a in *dot; do dot -Tpng $a > $(echo $a | sed s/.dot/.png/); done
# ls *png

Alternatively, replace -Tpng and .png with -Tsvg and .svg respectively to get vector graphics, or -Tpdf/.pdf for pdfs.

You've now made three graphs that you might as well print right now and glue to your desk if you will be working with Varnish a lot.

For convenience, the graphs from Varnish 4.1 are included. If you don't quite grasp what these tell you yet, don't be too alarmed. These graphs are provided early as they are useful to have around as reference material. A brief explanation for each is included, mostly to help you in later chapters.



This can be used when writing VCL. You want to look for the blocks that read vcl_ to identify VCL functions. The lines tell you how a return-statement in VCL will affect the VCL state engine at large, and which return statements are available where. You can also see which objects are available where.

This particular graph details the client-specific part of the VCL state engine.



This graph has the same format as the cache_req_fsm.png-one, but from the perspective of a backend request.



Of the three, this is the least practical flow chart, mainly included for completeness. It does not document much related to VCL or practical Varnish usage, but the internal state engine of an HTTP request in Varnish. It can sometimes be helpful for debugging internal Varnish issues.


November 16, 2015

Kristian LyngstølVisualizing VCL

Posted on 2015-11-16

I was preparing to upgrade a customer, and ran across a semi-extensive VCL setup. It quickly became a bit hard to get a decent overview of what was going on.

The actual VCL is fairly simple.

To deal with this, I ended up hacking together a tiny awk/shell script to generate a dot graph of how things were glued together. You can find the script at .

The output is somewhat ugly, but useful.


(Click for full version)

Of note:

  • This is so far Varnish 3.0-ish, mainly because of the error-syntax. (So it'll work for 4.x, just without vcl_error-tracking)
  • Green borders: Found the reference and everything is OK.
  • Black border: The sub was referenced, but not found in any VCL file.
  • Red border: The sub was found, but never referenced (doesn't count for subroutines beginning with vcl_, e.g. vcl_recv, etc)

No idea if it's of interest to anyone but me, but I found it useful.


October 12, 2015

Ingvar HagelundVarnish-4.1.0 released, packages for fedora and epel

Varnish-4.1.0 was recently released, and as usual, I have patched and wrapped up packages for fedora and epel. As 4.1.0 is not api/abi compatible with varnish-4.0, packages for stable releases of epel and fedora are not updated. Varnish-4.1.x will be available in a stable Fedora at latest from f24, though the package recompiles fine on anything from el5 to f23 as well.

Prebuilt packages for epel5, epel6, and epel7 are available here:

If you are a fedora contributor, please test the f23 package. The package should install directly on el7 and all supported fedoras, including f23. Then report feedback and add karma points. With a little luck, varnish-4.1 will go into fedora 23 before it freezes.


Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at

September 25, 2015

Kristian LyngstølMagic Grace

Posted on 2015-09-25

I was hacking together a JavaScript varnishstat implementation for a customer a few days ago when I noticed something strange. I have put Varnish in front of the agent delivering stats, but I'm only caching the statistics for 1 second.

But the cache hit rate was 100%.

And the stats were updating?

Logically speaking, how can you hit cache 100% of the time and still get fresh content all the time?

Enter Grace

Grace mode is a feature Varnish has had since version 2.0 back in 2008. It is a fairly simple mechanic: Add a little bit of extra cache duration to an object. This is the grace period. If a request is made for the object during that grace period, the object is updated and the cached copy is used while updating it.

This reduces the thundering horde problem when a large amount of users request recently expired content, and it can drastically improve user experience when updating content is expensive.

The big change that happened in Varnish 4 was background fetches.

Varnish uses a very simple thread model (so to speak). Essentially, each session is handled by one thread. In prior versions of Varnish, requests to the backend were always tied to a client request.

  • Thread 1: Accept request from client 1
  • Thread 1: Look up content in cache
  • Thread 1: Cache miss
  • Thread 1: Request content from web server
  • Thread 1: Block
  • Thread 1: Get content from web server
  • Thread 1: Respond

If the cache is empty, there isn't much of a reason NOT to do this. Grace mode always complicated this. What PHK did to solve this was, in my opinion, quite brilliant in its simplicity. Even if it was a trade-off.

With grace mode, you HAVE the content, you just need to make sure it's updated. It looked something like this:

  • Thread 1: Accept request from client 1
  • Thread 1: Look up content in cache
  • Thread 1: Cache miss
  • Thread 1: Request content from web server
  • Thread 1: Block
  • Thread 1: Get content from web server
  • Thread 1: Respond

So ... NO CHANGE. For a single client, you don't have grace mode in earlier Varnish versions.

But enter client number 2 (or 3, 4, 5...):

  • Thread 1: Accept request from client 1
  • Thread 1: Look up content in cache
  • Thread 1: Cache miss
  • Thread 1: Request content from web server
  • Thread 1: Block
  • Thread 2: Accept request from client 2
  • Thread 2: Look up content in cache
  • Thread 2: Cache hit - grace copy is now eligible - Respond
  • Thread 1: Get content from web server
  • Thread 1: Respond

So with Varnish 2 and 3, only the first client will block waiting for new content. This is still an issue, but it does the trick for the majority of use cases.

Background fetches!

Background fetches changed all this. It's more complicated in many ways, but from a grace perspective, it massively simplifies everything.

With Varnish 4 you get:

  • Thread 1: Accept request from client 1
  • Thread 1: Look up content in cache
  • Thread 1: Cache hit - grace copy is now eligible - Respond
  • Thread 2: Request content from web server
  • Thread 2: Block
  • Thread 3: Accept request from client 2
  • Thread 3: Look up content in cache
  • Thread 3: Cache hit - grace copy is now eligible - Respond
  • Thread 2: Get content from web server

And so forth. Strictly speaking, I suppose this makes grace /less/ magical...

In other words: The first client will also get a cache hit, but Varnish will update the content in the background for you.

It just works.


What is a cache hit?

If I tell you that I have 100% cache hit rate, how much backend traffic would you expect?

We want to keep track of two ratios:

  • Cache hit rate - how much content is delivered directly from cache (same as today). Target value: 100%.
  • Fetch/request ratio: How many backend fetches do you initiate per client request. Target value: 0%.

For my application, a single user will result in a 100% cache hit rate, but also a fetch/request ratio of 100%. The cache isn't really offloading the backend load significantly until I have multiple users of the app. Mind you, if the application was slow, this would still benefit that one user.

The latter is also interesting from a security point of view. If you find the right type of request, you could end up with more backend fetches than client requests (e.g. due to restarts/retries).

How to use grace

You already have it, most likely. Grace is turned on by default, using a 10 second grace period. For frequently updated content, this is enough.

Varnish 4 changed some of the VCL and parameters related to grace. The important bits are:

  • Use beresp.grace in VCL to adjust grace for an individual object.
  • Use the default_grace parameter to adjust the ... default grace for objects.

If you want to override grace mechanics, you can do so in either vcl_recv by setting req.ttl to define a max TTL to be used for an object, regardless of the actual TTL. That bit is a bit mysterious.

Or you can look at vcl_hit. Here you'll be able to do:

if (obj.ttl + obj.grace > 0s && obj.ttl =< 0s) {
        // We are in grace mode, we have an object though
        if (req.http.x-magic-skip-grace-header ~ "yes") {
                return (miss);
        } else {
                return (delier);

The above example-snippet will evaluate of the object has an expired TTL, but is still in the grace period. If that happens, it looks for a client header called "X-Magic-Skip-Grace-Header" and checks if it contains the string "yes". If so, the request is treated as a cache miss, otherwise, the cached object is delivered.


September 19, 2015

Kristian LyngstølVarnish Wishlist

Posted on 2015-09-19

I recently went back to working for Redpill Linpro, and thus started working with Varnish again, after being on the side lines for a few years.

I've been using Varnish since 2008. And a bit more than just using it too. There's been a lot of great change over time, but there are still things missing. I recently read and while I largely agree with Kacper, I think some of the bigger issues are missing from the list.

So here's my attempt to add to the debate.


Varnish needs TLS/SSL.

It's the elephant in the room that nobody wants to talk about.

The world is not the same as it was in 2006. Varnish is used for more and more sensitive sites. A larger percentage of Varnish installations now have some sort of TLS/SSL termination attached to it.

TLS/SSL has been a controversial issue in the history of Varnish Cache, with PHK (Principal architect of Varnish Cache - being an outspoken opponent of adding TLS in Varnish. There are valid reasons, and heartbleed has most certainly proven many of PHK's grievances right. But what does that matter when we use TLS/SSL anyway? It's already in the stack, we're just closing our eyes to it.

Setting up nginx in front of Varnish to get TLS/SSL, then nginx behind Varnish to get TLS/SSL... That's just silly. Why not just use nginx to cache then? The lack of TLS/SSL in Varnish is a great advertisement for nginx.

There are a lot of things I dislike about TLS/SSL, but we need it anyway. There's the hitch project (, but it's not really enough. We also need TLS/SSL to the backends, and a tunnel-based solution isn't enough. How would you do smart load balancing through that? If we don't add TLS/SSL, we might as well just forget about backend directors all together. And it has to be an integral part of all backends.

We can't have a situation where some backend directors support TLS/SSL and some don't.

Varnish Software is already selling this through Varnish Cache Plus, their proprietary version of Varnish. That is obviously because it's a deal breaker in a lot of situations. The same goes for basically any serious commercial actor out there.

So we need TLS/SSL. And we need it ASAP.


After speaking to PHK, let me clarify: He's not against adding support for TLS, but adding TLS itself. Varnish now supports the PROXY-protool which is added explicitly to improve support for TLS termination. Further such additions would likely be acceptable, always doing the TLS outside of Varnish.

Better procedures for VCL changes

With every Varnish version, VCL (The configuration language for Varnish) changes either a little bit, or a lot. Some of these changes are unavoidable due to internal Varnish changes. Some changes are to tweak the language to be more accurate (e.g. changing req.request to req.method, to reflect that it's the request method).

If Varnish is part of your day-to-day work, then this might not be a huge deal. You probably keep up-to-date on what's going on with Varnish anyway. But most users aren't there. We want Varnish to be a natural part of your stack, not a special thing that requires a "varnish-admin".

This isn't necessarily an easy problem to solve. We want to be able to improve VCL and get rid of old mistakes (e.g., changing req.request to req.method is a good thing for VCL). We've also changed the way to do error messages (or custom varnish-generated messages) numerous times. And how to create hitpass objects (a complicated aspect of any cache).

A few simple suggestions:

  • All VCL changes reviewed in public as a whole before the release process even starts. To avoid having to change it again two versions down the line.
  • Backward compatibility when possible. With warnings or even requiring an extra option to allow it. E.g.: req.request could easily still work, there's no conflict there. Never for forever, but perhaps to the end of a major version. Not everything will be backwards compatible, but some can.

I've had numerous complaints from highly skilled sysadmins who are frustrated by this aspect of Varnish. They just don't want to upgrade because they have to do what feels like arbitrary VCL changes every single time. Let's see if we can at least REDUCE that.


There's a lot of documentation for Varnish, but there's also a lot of bad documentation. Some issues:

  • People Google and end up on random versions on No, telling people "but there's a version right there so it's your own fault!" is not an acceptable solution. Varnish Software them self recently had a link in their Varnish Book where they used a link to "trunk" instead of "4.0", whereupon the "here is a complete list of changes between Varnish 3 and Varnish 4" link was actually a link to changes betwen Varnish 4.0 and the next version of Varnish.

  • "user guide" and "tutorial" and "installation"? Kill at least two and leave the others for blog posts or whatever. Hard enough to maintain one with decent quality.

  • Generated documentation needs to be improved. Example:

            STRING fileread(PRIV_CALL, STRING)
            Reads a file and returns a string with the content. Please
            note that it is not recommended to send variables to this
            function the caching in the function doesn't take
            this into account. Also, files are not re-read.
            set beresp.http.served-by = std.fileread("/etc/hostname");

    PRIV_CALL should clearly not be exposed! Other examples are easy enough to find.

    In addition, the Description is a mixture of reference documentation style and elaboration. Reference documentation should be clearly separated from analysis of consequences so technical users don't have to reverse-engineer a sentence of "don't do this because X" to figure out what the code actually does.

    And where are the details? What happens if the file can't be opened? What are the memory constraints? It says it returns the content of the file as a string, but what happens with binary content? There's clearly some caching of the file, but how does that work? Per session? Per VCL? Does that cache persist when you do varnishadm stop; varnishadm start? That's completely left out.

  • Rants mixed in with documentation? Get rid of "doc/shpinx/phk" ( and instead reference it somewhere else. should not be a weird blog-space. It clutters the documentation space. Varnish is not a small little project any more, it's grown past this.

VMOD packages

Varnish vmods are awesome. You can design some truly neat solutions using Open Source vmods, or proprietary ones.

But there are no even semi-official package repositories for the open source vmods. Varnish Software offers this to customers, but I really want it for the public too. Both for my own needs, and because it's important to improve Varnish and VMOD adaption.

Until you can do "apt-get install varnish-vmod-foo" or something like that, VMODS will not get the attention they deserve.

There are some projects in the works here, though, so stay tuned.


In case you missed it, I want TLS/SSL.

I want to be able to type https://<varnish host>

BTW: Regarding terminology, I decided to go with "TLS/SSL" instead of either "SSL" or "TLS" after some feedback. I suppose "TLS" is correct, but "SSL" is more recognized, whether we like it or not.


August 23, 2015

Kacper WysockiMy Varnish pet peeves

I’ve been meaning to write a blog entry about Varnish for years now. The closest I’ve come is to write a blog about how to make Varnish cache your debian repos, make you a WikiLeaks cache and I’ve released Varnish Secure Firewall, but that without a word on this blog. So? SO? Well, after years it turns out there is a thing or two to say about Varnish. Read on to find out what annoys me and people I meet the most.

varnish on wood

Although you could definitely call me a “Varnish expert” and even a sometimes contributor, and I do develop programs, I cannot call myself a Varnish developer because I’ve shamefully never participated in a Monday evening bug wash. My role in the Varnish world is more… operative. I am often tasked with helping ops people use Varnish correctly, justify its use and cost to their bosses, defend it from expensive and inferior competitors, sit up long nites with load tests just before launch days. I’m the guy that explains the low risk and high reward of putting Varnish in front of your critical site, and the guy that makes it actually be low risk, with long nites on load tests and I’ll be the first guy on the scene when the code has just taken a huge dump on the CEO’s new pet Jaguar. I am also sometimes the guy who tells these stories to the Varnish developers, although of course they also have other sources. The consequences of this .. lifestyle choice .. is that what code I do write is either short and to the point or .. incomplete.

bug wash

I know we all love Varnish, which is why after nearly 7 years of working with this software I’d like to share with you my pet peeves about the project. There aren’t many problems with this lovely and lean piece of software but those which are there are sharp edges that pretty much everyone snubs a toe or snags their head on. Some of them are specific to a certain version, while others are “features” present in nearly all versions.

And for you Varnish devs who will surely read this, I love you all. I write this critique of the software you contribute to, knowing full well that I haven’t filed bug reports on any of these issues and therefore I too am guilty in contributing to the problem and not the solution. I aim to change that starting now :-) Also, I know that some of these issues are better lived with than fixed, the medicine being more hazardous than the disease, so take this as all good cooking; with a grain of salt.

Silent error messages in init scripts

Some genious keeps inserting 1>/dev/null 2>&1 into the startup scripts on most Linux distros. This might be in line with some wacko distro policy but makes conf errors and in particular VCL errors way harder to debug for the common man. Even worse, the `service varnish reload` script called `varnish-vcl-reload -q`, that’s q for please-silence-my-fatal-conf-mistakes, and the best way to fix this is to *edit the init script and remove the offender*. Mind your p’s and q’s eh, it makes me sad every time, but where do I file this particular bug report?

silent but deadly still not adequately documented

People go YEARS using Varnish without discovering watch varnishadm Not to mention that it’s anyone’s guess this has to do with probes, and that there are no other debug.* parameters, except for the totally unrelated debug parameter. Perhaps this was decided to be dev-internal at some point, but the probe status is actually really useful in precisely this form. is still absent from the list and the man pages, while in 4.0 some probe status and backend info has been put into varnishstat, which I am sure to be not the only one being verry thankful for indeed.

Bad naming

Designing a language is tricky.


Explaining why purge is now ban and what is now purge is something else is mindboggling. This issue will be fixed in 10 years when people are no longer running varnish 2.1 anywhere. Explaining all the three-letter acronyms that start with V is just a gas.
Showing someone ban("req.url = "+ req.url) for the first time is bound to make them go “oh” like a racoon just caught sneaking through your garbage.
Grace and Saint mode… that’s biblical, man. Understanding what it does and how to demonstrate the functionality is still for Advanced Users, explaining this to noobs is downright futile, and I am still unsure whether we wouldn’t all be better off for just enabling it by default and forgetting about it.
I suppose if you’re going to be awesome at architecting and writing software, it’s going to get in the way of coming up with really awesome names for things, and I’m actually happy that’s still the way they prioritize what gets done first.

Only for people who grok regex

Sometimes you’ll meet Varnish users who do code but just don’t grok regex. It’s weak, I know, but this language isn’t for them.

Uncertain current working directory

This is a problem on some rigs which have VCL code in stacked layers, or really anywhere where it’s more appropriate to call the VCL a Varnish program, as in “a program written for the Varnish runtime”, rather than simply a configuration for Varnish.

UncertantyYou’ll typically want to organize your VCL in such a way that each VCL is standalone with if-wrappend rules and they’re all included from one main vcl file, stacking all the vcl_recv’s and vcl_fetches .

Because distros don’t agree on where to put varnishd’s current working directory, which happens to be where it’s been launched from, instead of always chdir $(basename $CURRENT_VCL_FILE), you can’t reliably specify include statements with relative paths. This forces us to use hardcoded absolute paths in includes, which is neither pretty nor portable.

Missing default director in 4.0

When translating VCL to 4.0 there is no longer any language for director definitions, which means they are done in vcl_init(), which means your default backend is no longer the director you specified at the top, which means you’ll have to rewrite some logic lest it bite you in the ass.

director.backend() is without string representation, instead of backend_hint,
so cannot do old style name comparisons, ie backends are first-class objects but directors are another class of objects.

the missing director

VCL doesn’t allow unused backends or probes

Adding and removing backends is a routine ordeal in Varnish.
Quite often you’ll find it useful to keep backup backends around that aren’t enabled, either as manual failover backups, because you’re testing something or just because you’re doing something funky. Unfortunately, the VCC is a strict and harsh mistress on this matter: you are forced to comment out or delete unused backends :-(

Workarounds include using the backends inside some dead code or constructs like

	set req.backend_hint = unused;
	set req.backend_hint = default;

It’s impossible to determine how many bugs this error message has avoided by letting you know that backend you just added, er yes that one isn’t in use sir, but you can definitely count the number of Varnish users inconvenienced by having to “comment out that backend they just temporarily removed from the request flow”.

I am sure it is wise to warn about this, but couldn’t it have been just that, a warning? Well, I guess maybe not, considering distro packaging is silencing error messages in init and reload scripts..

To be fair, this is now configurable in Varnish by setting vcc_err_unref to false, but couldn’t this be the default?

saintmode_threshold default considered harmful


If many different URLs keep returning bad data or error codes, you might concievably want the whole backend to be declared sick instead of growing some huge list of sick urls for this backend. What if I told you your developers just deployed an application which generates 50x error codes triggering your saintmode for an infinite amount of URLs? Well, then you have just DoSed yourself because you hit this threshold. I usually enable saintmode only after giving my clients a big fat warning about this one, because quite frankly this easily comes straight out of left field every time. Either saintmode is off, or the treshold is Really Large™ or even ∞, and in only some special cases do you actually want this set to an actual number.

Then again, maybe it is just my clients and the wacky applications they put behind Varnish.

What is graceful about the saint in V4?

While we are on the subject, grace mode being the most often misunderstood feature of Varnish, the thing has changed so radically in Varnish 4 that it is no longer recognizable by users, and they often make completely reasonable but devestating mistakes trying to predict its behavior.

To be clear on what has happened: saint mode is deprecated as a core feature in V4.0, while the new architecture now allows a type of “stale-while-revalidate” logic. A saintmode vmod is slated for Varnish 4.1.

But as of 4.0, say you have a bunch of requests hitting a slow backend. They’ll all queue up while we fetch a new one, right? Well yes, and then they all error out when that request times out, or if the backend fetch errors out. That sucks. So lets turn on grace mode, and get “stale-while-revalidate” and even “stale-if-error” logic, right? And send If-Modified-Since headers too, sweet as.

Now that’s gonna work when the request times out, but you might be surprised that it does not when the request errors out with 50x errors. Since beresp.saint_mode isn’t a thing anymore in V4, those error codes are actually going to knock the old object outta cache and each request is going to break your precious stale-while-error until the backend probe declares the backend sick and your requests become grace candidates.

Ouch, you didn’t mean for it to do that, did you?

The Saint

And if, gods forbid, your apphost returns 404′s when some backend app is not resolving, bam you are in a cascading hell fan fantasy.

What did you want it to do, behave sanely? A backend response always replaces another backend response for the same URL – not counting vary-headers. To get a poor mans saint mode back in Varnish 4.0, you’ll have to return (abandon) those erroneous backend responses.

Evil grace on unloved objects

For frequently accessed URLs grace is fantastic, and will save you loads of grief, and those objects could have large grace times. However, rarely accessed URLs suffer a big penalty under grace, especially when they are dynamic and ment to be updated from backend. If that URL is meant to be refreshed from backend every hour, and Varnish sees many hours between each access, it’s going to serve up that many-hour-old stale object while it revalidates its cache.

stale while revalidate
This diagram might help you understand what happens in the “200 OK” and “50x error” cases of graceful request flow through Varnish 4.0.

Language breaks on major versions

This is a funny one because the first major language break I remember was the one that I caused myself. We were making security.vcl and I was translating rules from mod_security and having trouble with it because Varnish used POSIX regexes at the time, and I was writing this really godaweful script to translate PCRE into POSIX when Kristian who conceived of security.vcl went to Tollef, who were both working in the same department at the time, and asked in his classical broker-no-argument kind of way "why don’t we just support Perl regexes?".
Needless to say, (?i) spent a full 12 months afterwards cursing myself while rewriting tons of nasty client VCL code from POSIX to PCRE and fixing occasional site-devestating bugs related to case-sensitivity.

Of course, Varnish is all the better for the change, and would get no where fast if the devs were to hang on to legacy, but there is a lesson in here somewhere.


So what's a couple of sed 's/req.method/req.request/'s every now and again?
This is actually the main reason I created the VCL.BNF. For one, it got the devs thinking about the grammar itself as an actual thing (which may or may not have resulted in the cleanups that make VCL a very regular and clean language today), but my intent was to write a parser that could parse any version of VCL and spit out any other version of VCL, optionally pruning and pretty-printing of course. That is still really high on my todo list. Funny how my clients will book all my time to convert their code for days but will not spend a dime on me writing code that would basically make the conversion free and painless for everyone forever.

Indeed, most of these issues are really hard to predict consequences of implementation decisions, and I am unsure whether it would be possible to predict these consequences without actually getting snagged by the issues in the first place. So again: varnish devs, I love you, what are your pet peeves? Varnish users, what are your pet peeves?

Errata: vcc_err_unref has existed since Varnish 3.

June 26, 2015

Ingvar Hagelundhitch-1.0.0-beta for Fedora and EPEL

The Varnish project has a new little free software baby arriving soon: Hitch, a scalable TLS proxy. It will also be made available with support by Varnish Software as part of their Varnish Plus product.

A bit of background:

Varnish is a high-performance HTTP accelerator, widely used over the Internet. To use varnish with https, it is often fronted by other general http/proxy servers like nginx or apache, though a more specific proxy-only high-performance tool would be preferable. So they looked at stud.

hitch is a fork of stud. The fork is maintained by the Varnish development team, as stud seems abandoned by its creators, after the project was taken over by Google, with no new commits after 2012.

I wrapped hitch for fedora, epel6 and epel7, and submitted them for Fedora and EPEL. Please test the latest builds and add feedback: . The default config is for a single instance of hitch.

The package has been reviewed and was recently accepted into Fedora and EPEL (bz #1235305). Update august 2015: Packages are pushed for testing. They will trickle down to stable eventually.

Note that there also exists as a fedora package of the (old) version of stud. If you use stud on fedora and want to test hitch, the two packages may coexist, and should be able to install in parallel.

To test hitch in front of varnish, in front of apache, you may do something like this (tested on el7):

  • Install varnish, httpd and hitch
      sudo yum install httpd varnish
      sudo yum --enablerepo=epel-testing install hitch || sudo yum --enablerepo=updates-testing install hitch
  • Start apache
      sudo systemctl start httpd.service
  • Edit the varnish config to point to the local httpd, that is, change the default backend definition in /etc/varnish/default.vcl , like this:
      backend default {
        .host = "";
        .port = "80";
  • Start varnish
      sudo systemctl start varnish.service
  • Add an ssl certificate to the hitch config. For a dummy certificate,
    the certificate from the hitch source may be used:

      sudo wget -O /etc/pki/tls/private/
  • Edit /etc/hitch/hitch.conf. Change the pem-file option to use that cert
      pem-file = "/etc/pki/tls/private/"
  • Start hitch
      sudo systemctl start hitch.service
  • Open your local firewall if necessary, by something like this:
      sudo firewall-cmd --zone=public --add-port=8443/tcp
  • Point your web browser to https://localhost:8443/ . You should be greeted with a warning about a non-official certificate. Past that, you will get the apache frontpage through varnish and hitch.

    Enjoy, and let me hear about any interesting test results.


    Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at

  • May 15, 2015

    Lasse KarstensenIntroducing hitch – a scalable TLS terminating proxy.

    The last couple of weeks we’ve been pretty busy making SSL/TLS support for Varnish Cache Plus 4. Now that the news is out, I can follow up with some notes here.

    The setup will be a TLS terminating proxy in front, speaking PROXY protocol to Varnish. Backend/origin support for SSL/TLS has been added, so VCP can now talk encrypted to your backends.

    On the client-facing side we are forking the abandoned TLS proxy called stud, and giving a new name: hitch.

    hitch will live on github as a standalone open source project, and we are happy to review patches/pull requests made by the community. Here is the source code:

    We’ve picked all the important patches from the flora of forks, and merged it all into a hopefully stable tool. Some of the new stuff includes: TLS1.1, TLS1.2, SNI, wildcard certs, multiple listening sockets. See the CHANGES.rst file updates.

    Varnish Software will provide support on it for commercial uses, under the current Varnish Plus product package.

    April 24, 2015

    Stefan CaunterRogers needs to pay to solve its CFL problem

    So, there’s an article on the Toronto Sun website. Understandably, they won’t publish my comment, which appears below. Spam and pointless bickering is fine, apparently. Sigh. Here is my take. Rogers Communications and the Jays situation in the dome is driving the debate about BMO Field. It has nothing to do with the Argos […]

    April 13, 2015

    Stefan CaunterThe unconditional interest of Leafs fans

    Leafs tickets are seen as an investment to hold, not a conditional payment on success.The Leaf team is an incredibly valuable sport property that is basically destroyed every year by the media that keep them incredibly valuable. How? The players are given exalted status based on next to nothing on an achievement scale. People get […]

    March 05, 2015

    Ingvar Hagelundvarnish-4.0.3 for Fedora and EPEL

    varnish-4.0.3 was released recently. I have wrapped packages for Fedora and EPEL, and requested updates for epel7, f21 and f22. They will trickle down as stable updates within some days. I have also built packages for el6, and after som small patching, even for el5. These builds are based on the Fedora package, but should be only cosmetically different from the el6 and el7 packages available from

    Also note that Red Hat finally caught up, and imported the necessary selinux-policy changes for Varnish from fedora into el7. With selinux-policy-3.13.1-23.el7, Varnish starts fine in enforcing mode. See RHBA-2015-0458.

    My builds for el5 and el6 are available here: Note that they need other packages from EPEL to work.

    Update 1: I also provide an selinux module for those running varnish-4.0 on el6. It should work for all versions of varnish-4.0, including mine and the ones from

    Update 2: Updated builds with a patch for bugzilla ticket 1200034 are pushed for testing in f21, f22 and epel7. el5 and el6 builds are available on link above.



    Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at

    January 19, 2015

    Lasse KarstensenPROXY protocol in Varnish

    Dag has been working implementing support for HAProxy’s PROXY protocol[1] in Varnish. This is a protocol adds a small header on each incoming TCP connection that describes who the real client is, added by (for example) an SSL terminating process. (since srcip is the terminating proxy)

    We’re aiming for merging this into Varnish master (so perhaps in 4.1?) when it is ready.

    The code is still somewhat unfinished, timeouts are lacking and some polishing needed, but it works and can be played with in a development setup.

    Code can be found here:

    I think Dag is using haproxy to test it with. I’ve run it with stunnel (some connection:close issues to figure out still), and I’d love if someone could test it with ELB, stud or other PROXY implementations.


    January 08, 2015

    Ingvar Hagelundrpm packages of vmod-ipcast

    Still on varnish-3.0? Missing the ability to filter X-Forwarded-For through ACLs? Use vmod ipcast by Lasse Karstensen.

    I cleaned up and rolled an rpm package of vmod-ipcast-1.2 for varnish-3.0.6 on el6. It’s available here:

    Note that the usage has changed a bit since the last version. You are now longer permitted to change client.ip (and that’s probably a good thing). Now it’s called like this, returning an IP address object:


    If the string does not resemble an IP address, the fallback ip is returned. Note that if the fallback ip is an unvalid address, varnishd will crash!

    So, if you want to filter X-Forwarded-For through an ACL, you would something like this:

    import ipcast;
    sub vcl_recv {
       # Add some code to sanitize X-Forwarded-For above here, so it resembles one single IP address
       if ( ipcast.ip(req.http.X-Forwarded-For, "") ~ someacl ) {
         # Do something special

    And that’s all for today.

    Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

    Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at

    October 13, 2014

    Lasse KarstensenVarnish VMOD static code analysis

    I recently went looking for something similar to pep8/pylint when writing Varnish VMODs, and ended up with OCLint.

    I can’t really speak to how good it is, but it catches the basic stuff I was interested in.

    The documentation is mostly for cmake, so I’ll give a small tutorial for automake:

  • (download+install oclint to somewhere in $PATH)
  • apt-get install bear
  • cd libvmod-xxx
  • ./; ./configure –prefix=/usr
  • bear make # “build ear” == bear. writes compile_commands.json
  • cd src
  • oclint libvmod-xxx.c # profit
  • Which will tell you about unused variables, useless parentheses, dead code and so on.

    October 03, 2014

    Lasse KarstensenAnnouncing libvmod-tcp: Adjust Varnish congestion control algorithm.

    I’ve uploaded my new TCP VMOD for Varnish 4 to github, you can find it here:

    This VMOD allows you to get the estimated client socket round trip time, and then let you change the TCP connection’s congestion control algorithm if you’re so inclined.

    Research[tm][0] says that Hybla is better for long high latency links, so currently that is what it is used for.

    Here is a quick VCL example:

    if (tcp.get_estimated_rtt() > 300) {
    set req.http.x-tcp = tcp.congestion_algorithm("hybla");

    One thing to note is that VCL handling is very early in the TCP connection lifetime. We’ve only just read and acked the HTTP request. The readings may be off, I’m analyzing this currently.
    (As I understand it the Linux kernel will keep per-ip statistics, so for subsequent requests this should get better and better..)

    0: Esterhuizen, A., and A. E. Krzesinski. “TCP Congestion Control Comparison.” (2012).

    September 30, 2014

    Lasse KarstensenFresh Varnish packages for Debian/Ubuntu and Redhat systems

    We use continuous integration when developing Varnish Cache. This means that we run our internal test suite (varnishtest) on all commits, so we catch our mistakes earlier.

    This pipeline of build jobs sometimes end up with binary packages of Varnish, which may be useful to people when they know they exist. They may not be the easiest to find, which this blog post tries to remedy.

    Development wise, Varnish Cache is developed with GIT with a master branch for development and a set of production branches, currently 3.0 and 4.0.

    Unreleased packages for Varnish master can be found here:

    Unreleased packages of Varnish 4.0 can be found here:

    (There is also a set of 3.0 jobs, but you should really go for 4.0 these days.)

    The latest commits in each of the production branches may contain fixes we’ve added after the last production release, but haven’t cut a formal release for yet. (For example there are some gzip fixes in the 3.0 branch awaiting a 3.0.6 release, which I really should get out soon.)

    Some jobs in the job listing just check that Varnish builds, without creating any output (or artifacts as Jenkins calls it.) This applies for any jobs with “-build-” in the name, for example varnish-4.0-build-el7-x86_64 and varnish-4.0-build-freebsd10-amd64.

    The Debian and Ubuntu packages are all built from one job currently, called varnish-VERSION-deb-debian-wheezy-amd64. Press “Expand all” under artifacts to get the full list.

    Redhat/RHEL packages are built in the different el5/el6/el7 jobs.

    The unreleased packages built for 3.0 and 4.0 are safe. This is the process used to build the officially released packages, just a step earlier in the process. The varnish-master packages are of course failing from time to time, but that is to be expected.

    The version numbers in the packages produced may be a bit strange, but that is what you get with unreleased software builds.

    I’m happy to improve this process and system if it can help you run never versions of Varnish, comments (either here or on IRC) are appreciated.

    June 03, 2014

    Lasse KarstensenWhat happened to ban.url in Varnish 4.0?

    tl;dr; when using Varnish 4 and bans via varnishadm, instead of “ban.url EXPRESSION”, use “ban req.url ~ EXPRESSION”.

    In Varnish 3.0 we had the ban.url command in the varnishadm CLI. This was a shortcut function expanding to the a bit cryptic (but powerful) ban command. In essence ban.url just took your expression, prefixed it with “req.url ~ ” and fed it to ban. No magic.

    We deprecated this in Varnish 4.0, and now everyone has to update their CMS’s plugin for cache  invalidation. Hence this blog post. Perhaps it will help. Perhaps not. :-)

    Some references:

    April 09, 2014

    MacYvesBuilding vagent2 for Varnish Cache 4.0.0 beta 1 for OS X 10.9.2

    For those keen bunnies that wishes to jump in and help us test out varnish cache 4.0.0 Beta 1 with varnish-agent 2, here’s how you do it on OS X 10.9.2 Mavericks.


    Homebrew dependencies

    Install the following with Homebrew

    • automake 1.14.1
    • libtool 2.4.2
    • pkg-config 0.28
    • pcre 8.34
    • libmicrohttpd 0.9.34

    Build varnish cache 4.0.0 beta 1

    1. Download and extract varnish cache 4
    2. run ./
    3. run ./configure
    4. make

    Build varnish-agent2 for varnish cache 4.0.0 beta 1

    1. Clone varnish-agent from repo
    2. Checkout the varnish-4.0-experimental branch
    3. export VARNISHAPI_CFLAGS=-I/tmp/varnish/varnish-4.0.0-beta1/include
    4. export VARNISHAPI_LIBS="-L/tmp/varnish/varnish-4.0.0-beta1/lib/libvarnishapi/.libs -lvarnishapi"
    5. run ./
    6. run ./configure
    7. make

    Note that if you run make install for varnish cache 4 or varnish-agent, it would then install it for you respectively.

    December 19, 2013

    Lasse KarstensenConverting a Varnish 3.0 VMOD to 4.0

    So we’re getting closer to releasing the first proper 4.0 version of Varnish Cache. One of the things we need to fix is to get all the vmod writers to make sure their vmod works with the new version.

    Here are my notes from doing just that, in the hope to make it simpler for others.

    In 4.0, you don’t need the source tree of Varnish any more. The include files will be enough, and pkg-config will find them for you.

    Make sure that /usr/lib/pkgconfig/varnishapi.pc and /usr/share/aclocal/varnish.m4 exists. If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source. (I need multiple varnishd versions when debugging customer problems, so I let them live in /opt/varnishX.Y/ on my laptop)

    Pull/merge the new files from the master branch of libvmod-example.

    Header files: remove bin/varnishd/cache.h and add cache/cache.h.

    Vmod functions are now called with a vrt context as first argument. %s/struct sess \*sp/const struct vrt_ctx \*ctx/g

    The old sess struct has been split, some data is in vrt_ctx->req, and some is in vrt_vtx->req->sp. Look up what is where in cache/cache.h. 

    I’ve put up the 3.0->4.0 diff for vmod_policy.c as a gist:

    There was a bit trouble of finding varnishtest, as src/Makefile was missing the reference entirely. I just fixed it by hand for now. Another thing for the 4.0 todolist, then.

    And finally; 

    lkarsten@immer:~/work/libvmod-policy/src$ make check
    /opt/varnish/bin/varnishtest -Dvarnishd=/opt/varnish/sbin/varnishd -Dvmod_topbuild=/home/lkarsten/work/libvmod-policy tests/test01.vtc
    # top TEST tests/test01.vtc passed (1.574)


    I have a working Varnish 4.0 vmod. :-D

    December 11, 2013

    Lasse KarstensenDNS RBL test address for development

    If you are writing code that checks a DNS real-time blockhole list (RBL), it looks like is the standard address that is always in the black/white -list.

    This is probably know for most sysadmins/security people and whatnot, but wasn’t entirely trivial to find using Google.

    lkarsten@immer:~$ dig @
    ; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55083
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
    ; IN A
    ;; ANSWER SECTION: 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A
    ;; Query time: 17 msec
    ;; SERVER:
    ;; WHEN: Wed Dec 11 14:12:20 2013
    ;; MSG SIZE rcvd: 203

    Good to be able to actually test your code for hits also.

    (this is for libvmod-policy, so you can deny/reject POST/PUT from spammers in Varnish)

    September 25, 2013

    Lasse KarstensenVarnish and Ghost blogging software

    So there is a new shiny blogging platform out called Ghost. Looks pretty good to me.

    If you want to run it behind Varnish, you’ll soon notice it has the usual problem of setting session cookies everywhere leading to 0% hit rate. 

    I have written a Varnish VCL configuration for filtering this in the necessary places, while keeping the admin interface working still.

    You can find it here:

    Have fun.

    September 10, 2013

    Lasse KarstensenTesting VMODs with Travis (

    Travis CI is a service where open source software can run tests automatically on commits. It hooks into github in a silky smooth way.

    If you’re developing a Varnish module (VMOD), you probably have started out with our libvmod-example package. It has all the automake magic you need, as well as some simple test cases to test your vmod. Given that you’ve written som varnishtest testcases for it (you really should), you now can get travis to run them as well!

    I’ve put a small recipe for this into the libvmod-example package.

    Feel free to play around with it, feedback appreciated. For the travis setup bits, consult the travis getting started guide. The final result is something like this, shown for libvmod-cookie:

    September 09, 2013

    MacYvesVagrant, Varnish and vmods

    Development environment has been plaguing us for a while in my product development department. From dependencies hell to complex setup in operations, our development environment has gone through the usual gauntlet of pains and complaints.

    This has changed with Vagrant. It is the single tool that gels the devs with the ops; quintessential devop tool if you will. Not only Vagrant has helped eliminate the “works on my machine” bugs, we use it for automated integration tests. In addition, this one tool has made our development environment setup quick and simple for our HCI guys too.

    We do a lot of integration work with Varnish Cache and I thought I would take this opportunity to share this simple Vagrantfile, as an example, to help get started with installing varnish and libdigest-vmod from source.

    Note that the provisioning process is rather crude in this example. Rather, the intention here is to out outline the steps required to get varnish and vmods installed and running via Vagrant. For production and future maintainability, do use Chef or Puppet as it can be seamlessly integrated within the Vagrantfile.

    # -*- mode: ruby -*-
    # vi: set ft=ruby :


    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

      config.vm.define :varnish do |varnish| = "varnish"
        varnish.vm.box_url = ""
        $script_varnish = <<SCRIPT
    echo Installing dependencies, curl
    sudo apt-get update
    sudo apt-get install curl -y
    sudo apt-get install git -y
    curl | sudo apt-key add -
    echo "deb precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
    echo "deb-src precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
    sudo apt-get update
    echo ==== Compiling and installing Varnish from source ====
    sudo apt-get install build-essential -y
    sudo apt-get build-dep varnish -y
    apt-get source varnish
    cd varnish-3.0.4
    sudo make install
    cd ..
    echo done
    echo ==== Compiling and installing lib-digest vmod from source ===
    git clone
    sudo apt-get install libmhash-dev libmhash2 -y
    cd libvmod-digest
    ./configure VARNISHSRC=/home/vagrant/varnish-3.0.4 VMODDIR=/usr/local/lib/varnish/vmods
    sudo make install
    cd ..
    echo ===== done ====
    echo ===== firing up varnish via commandline ====
    sudo varnishd -a :80 -T :6081 -f /vagrant/test.vcl
    touch varnish_vm

        varnish.vm.provision :shell, :inline => $script_varnish


    July 29, 2013

    Lasse KarstensenBuilding a Varnish VMOD on Debian

    From the tutorials department, here are some quick notes on how to install a Varnish VMOD from source.

    This is slightly complicated because Varnish demands that a VMOD must be built against the same git commit (or release) as the one that is running. This will be relaxed in future versions.

    Current setup is a standalone Varnish VM on Debian Wheezy with Varnish installed from varnish package archives (3.0.4-1~wheezy.)

    1. Get the vmod

    lkarsten@lb1:~$ git clone
    Cloning into 'libvmod-cookie'...
    remote: Counting objects: 253, done.
    remote: Compressing objects: 100% (131/131), done.
    remote: Total 253 (delta 132), reused 232 (delta 112)
    Receiving objects: 100% (253/253), 49.51 KiB, done.
    Resolving deltas: 100% (132/132), done.

    2. Get and configure the source tree for the running Varnish

    Verify first that you have the necessary package repositories enabled:

    lkarsten@lb1:~$ grep varnish /etc/apt/sources.list
    deb wheezy varnish-3.0
    deb-src wheezy varnish-3.0

    After that, continue with the juicy parts:

    lkarsten@lb1:~$ apt-get source varnish 
    Reading package lists... Done 
    Building dependency tree 
    Reading state information... Done 
    NOTICE: 'varnish' packaging is maintained in the 'Git' version control system at: 
    Need to get 2,060 kB of source archives. 
    Get:1 wheezy/varnish-3.0 varnish 3.0.4-1 (dsc) [2,334 B] 
    Get:2 wheezy/varnish-3.0 varnish 3.0.4-1 (tar) [2,044 kB] 
    Get:3 wheezy/varnish-3.0 varnish 3.0.4-1 (diff) [14.1 kB] 
    Fetched 2,060 kB in 0s (11.4 MB/s) 
    gpgv: keyblock resource `/home/lkarsten/.gnupg/trustedkeys.gpg': file open error 
    gpgv: Signature made Fri 14 Jun 2013 11:56:48 CEST using RSA key ID 87218D9C 
    gpgv: Can't check signature: public key not found 
    dpkg-source: warning: failed to verify signature on ./varnish_3.0.4-1.dsc 
    dpkg-source: info: extracting varnish in varnish-3.0.4 
    dpkg-source: info: unpacking varnish_3.0.4.orig.tar.gz 
    dpkg-source: info: applying varnish_3.0.4-1.diff.gz 
    lkarsten@lb1:~$ cd varnish-3.0.4
    lkarsten@lb1:~/varnish-3.0.4$ ./
    lkarsten@lb1:~/varnish-3.0.4$ ./configure --prefix=/usr
    lkarsten@lb1:~/varnish-3.0.4$ make

    If configure or make fails, you might need some additional packages. Run an apt-get build-dep varnish and work from there. (if editline fails on you, remember to rerun configure after installing it)

    3. Build and install the vmod

    lkarsten@lb1:~$ cd libvmod-cookie/
    lkarsten@lb1:~/libvmod-cookie$ ./
    + aclocal -I m4
    + libtoolize --copy --force
    libtoolize: putting auxiliary files in `.'.
    libtoolize: copying file `./'
    libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
    libtoolize: copying file `m4/libtool.m4'
    libtoolize: copying file `m4/ltoptions.m4'
    libtoolize: copying file `m4/ltsugar.m4'
    libtoolize: copying file `m4/ltversion.m4'
    libtoolize: copying file `m4/lt~obsolete.m4'
    + autoheader
    + automake --add-missing --copy --foreign installing `./config.guess' installing `./config.sub' installing `./install-sh' installing `./missing'
    src/ installing `./depcomp'
    + autoconf
    lkarsten@lb1:~/libvmod-cookie$ ./configure VARNISHSRC=~/varnish-3.0.4/
    # and finally
    lkarsten@lb1:~/libvmod-cookie$ make
    libtool: link: ( cd ".libs" && rm -f "" && ln -s "../" "" )
    make[2]: Leaving directory `/home/lkarsten/libvmod-cookie/src'
    make[2]: Entering directory `/home/lkarsten/libvmod-cookie'
    rst2man README.rst vmod_cookie.3
    make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
    make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'
    lkarsten@lb1:~/libvmod-cookie$ sudo make install
    /bin/mkdir -p '/usr/local/share/man/man3'
     /usr/bin/install -c -m 644 vmod_cookie.3 '/usr/local/share/man/man3'
    make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
    make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'

    At this point you should have the two vmod files available for Varnish:

    lkarsten@lb1:~/libvmod-cookie$ ls -l /usr/lib/varnish/vmods/
    total 64
    -rwxr-xr-x 1 root root 966 Jul 29 11:11
    -rwxr-xr-x 1 root root 41538 Jul 29 11:11
    -rw-r--r-- 1 root root 16128 Jun 17 13:38

    And you are done!

    “import cookie” should now work without issue in your /etc/varnish/default.vcl.

    July 22, 2013

    Lasse KarstensenSetting client.ip in Varnish VCL with libvmod-ipcast

    I’ve written a new Varnish 3.0 VMOD called ipcast.

    It has a single function; ipcast.clientip(ipstring) which sets the internal Varnish variable client.ip to whatever IPv4/IPv6 address you give as the argument.

    You need this if you want to do ACL checks on connections done through a load balancer or SSL terminator. In those cases client.ip would be and you get the real client’s IP address in the X-Forwarded-For (or similar) header.

    You can find it here:

    Here is some example VCL to illustrate how it works. I think the regex needs some work, suggestions/pull requests are welcome.

    import ipcast;
    acl friendly_network {
    sub vcl_recv {
        if (req.http.X-Forwarded-For !~ ",") {
            set req.http.xff = req.http.X-Forwarded-For;
        } else {
            set req.http.xff = regsub(req.http.X-Forwarded-For,
                    "^[^,]+.?.?(.*)$", "\1");
        if (ipcast.clientip(req.http.xff) != 0) {
            error 400 "Bad request";
        if (client.ip !~ friendly_network) {
                error 403 "Forbidden";

    July 18, 2013

    cd34Varnish and Node.js

    While working with a client installation they wanted to run Varnish in front of their node.js powered site to eliminate having node serve the static assets. uses HTTP/1.0 and cannot be cached. Minimally these few lines can be added to their respective functions and things will work. Obviously you’ll want to set expires on […]

    June 30, 2013

    ops42Properly redirect to mobile pages

    It is just amazing how much advice and examples one can find for how to redirect to a mobile equivalent of a given HTTP address. Oversimplified, wrong and harmful advice that is. And no, I’m not talking about that 301 vs 302 bullshit. For the love of God, stop listening to those overpaid, know-nothing SEO […]

    March 25, 2013

    Mikko OhtamaaVarnish at the front of WordPress + Apache and Plone CMS virtual hosts

    When moving some sites to a new server I upgraded the Varnish cache server configuration serving this setup. Here are my notes how one can use Varnish at the front of virtual hosting.

    The setup is following

    • Multiple sites are hosted on the same server. The sites are mix of PHP of Plone sites.
    • Varnish accepts HTTP requests in port 80 and forwards request to the corresponding backend through HTTP proxying
    • Our virtual host rules capture domain names with or without www-prefix, or with any subdomain name prefix
    • Apache runs in non-standard port localhost:81, serving PHP and WordPress. WordPress caching rules are defined in Apache <virtualhost> config file.
    • Every Plone site runs in its own port and process. Plone uses VirtualHostMonster to rewrite publicly facing site URLS. Plone caching HTTP headers are set by addon.
    • We do extensive cookie sanitization for PHP (generic), WordPress and Plone.  Google Analytics etc. cookies don’t bust the cache and we can still login to WordPress and Plone as admin
    • As a special trick, there is cleaned cookie debugging trick through HTTP response headers


    Don’t worry, Varnish can handle little load



    • Blazingly fast, as Varnish is
    • With Plone’s, one does not need to touch configuration files but Plone caching HTTP headers can be configured through-the-web


    • Varnish does not have Nginx or Apache style virtual host configuration file facilities by default and making includes is little bit tricky: With many virtualhost the default.vcl config file grows long.
    • Because WordPress cannot do static resource serving as smartly as Plone, which has unique URLs for all static media revisions, you need to purge Varnish manually from command line if you update any static media files like CSS, JS or images.

    Varnish /etc/varnish/default.vcl example for Varnish 3.0:

    # This backend never responds... we get hit in the case of bad virtualhost name
    backend default {
        .host = "";
        .port = "55555";
    backend myplonesite {
        .host = "";
        .port = "6699";
    # Apache running on server port 81
    backend apache {
        .host = "";
        .port = "81";
    # Gues which site / virtualhost we are diving into.
    # Apache, Nginx or Plone directly
    sub choose_backend {
        # WordPress site
        if ( ~ "^(.*\.)?opensourcehacker\.com(:[0-9]+)?$") {
            set req.backend = apache;
        # Example Plone site
        if ( ~ "^(.*\.)?myplonesite\.fi(:[0-9]+)?$") {
            set req.backend = myplonesite;
            # Zope VirtualHostMonster
            set req.url = "/VirtualHostBase/http/" + + ":80/Plone/VirtualHostRoot" + req.url;
    sub vcl_recv {
        # Do Plone cookie sanitization, so cookies do not destroy cacheable anonymous pages.
        # Also, make sure we do not destroy WordPress admin and login cookies in the proces
        if (req.http.Cookie && !(req.url ~ "wp-(login|admin)")) {
            set req.http.Cookie = ";" + req.http.Cookie;
            set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
            set req.http.Cookie = regsuball(req.http.Cookie, ";(statusmessages|__ac|_ZopeId|__cp|php|PHP|wordpress_(.*))=", "; \1=");
            set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
            set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
            if (req.http.Cookie == "") {
                remove req.http.Cookie;
        call choose_backend;
        if (req.request != "GET" &&
          req.request != "HEAD" &&
          req.request != "PUT" &&
          req.request != "POST" &&
          req.request != "TRACE" &&
          req.request != "OPTIONS" &&
          req.request != "DELETE") {
            /* Non-RFC2616 or CONNECT which is weird. */
            return (pipe);
        if (req.request != "GET" && req.request != "HEAD") {
            /* We only deal with GET and HEAD by default */
            return (pass);
        if (req.http.Authorization || req.http.Cookie) {
            /* Not cacheable by default */
            return (pass);
        return (lookup);
    sub vcl_fetch {
        /* Use to see what cookies go through our filtering code to the server */
        /* set beresp.http.X-Varnish-Cookie-Debug = "Cleaned request cookie: " + req.http.Cookie; */
        if (beresp.ttl <= 0s ||
            beresp.http.Set-Cookie ||
            beresp.http.Vary == "*") {
             * Mark as "Hit-For-Pass" for the next 2 minutes
            set beresp.ttl = 120 s;
            return (hit_for_pass);
        return (deliver);
    # Show custom helpful 500 page when the upstream does not respond
    sub vcl_error {
      // Let's deliver a friendlier error page.
      // You can customize this as you wish.
      set obj.http.Content-Type = "text/html; charset=utf-8";
      synthetic {"
      <?xml version="1.0" encoding="utf-8"?>
      <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
          <title>"} + obj.status + " " + obj.response + {"</title>
          <style type="text/css">
          #page {width: 400px; padding: 10px; margin: 20px auto; border: 1px solid black; background-color: #FFF;}
          p {margin-left:20px;}
          body {background-color: #DDD; margin: auto;}
        <div id="page">
        <h1>Sivu ei ole saatavissa</h1>
        <p>Pahoittelemme, mutta palvelua ei ole saatavilla.</p>
        <hr />
        <h4>Debug Info:</h4>
        <pre>Status: "} + obj.status + {"
    Response: "} + obj.response + {"
    XID: "} + req.xid + {"</pre>

    WordPress does not support setting HTTP response headers natively like Plone. We set them in Apache virtual host configuration file in /etc/apache2/sites-enabled:

        LogFormat       combined
        TransferLog     /var/log/apache2/
        # Basic WordPress setup
        Options +Indexes FollowSymLinks +ExecCGI
        DocumentRoot /srv/php/opensourcehacker/wordpress
        <Directory /srv/php/opensourcehacker/wordpress>
            Options FollowSymlinks
            AllowOverride All
        AddType text/css .css
        AddType application/x-httpd-php .php .php3 .php4 .php5
        AddType application/x-httpd-php-source .phps
        # Set expires headers manually
        ExpiresActive On
        ExpiresByType text/html A0
        ExpiresByType image/gif A3600
        ExpiresByType image/png A3600
        ExpiresByType image/image/ A3600
        ExpiresByType image/jpeg A3600
        ExpiresByType text/css A3600
        ExpiresByType text/javascript A3600
        ExpiresByType application/x-javascript A3600

     Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

    February 15, 2013

    Kristian LyngstølThe Architecture the Varnish Agent

    Posted on 2013-02-15

    Designing software architecture is fun.

    The Varnish Agent 2 was written as a replacement for the original Varnish Agent. They both share the same purpose: Expose node-specific Varnish features to a management system. They are design very differently, though.

    In this post I'd like to explain some choices that were made, and show you how to write your own code for the Varnish Agent 2. It's really not that hard.

    The code can be found at:

    Why C ?

    The choice of C as a language was made fairly early. One of the main reasons is that Varnish itself is written in C, as are all the tools for Varnish. This means that the by far best supported APIs for talking to Varnish are written in C.

    But an other reason is because C is a very good language. It has become a false truth that you "never write web apps in C", more or less. There are good reasons for this: It takes time to set things up in C, C isn't very forgiving and perhaps most importantly: people generally suck at C.

    In the end, we chose C because it was the right tool for the job.


    When designing a new system, it's important to know what you're trying to achieve, and perhaps just as important to know what you're /not/ trying to achieve.

    The Varnish Agent is designed to:

    • Manage a single Varnish server.
    • Remove the need for management frontends to know the Varnish CLI language.
    • Expose log data
    • Persist configuration changes
    • Require "0" configuration of the agent itself
    • Ensure that Varnish works on boot, even if there is no management front-end present.
    • Be expandable without major re-factoring.
    • Be easy to expand

    What we did NOT want was:

    • Support for running the agent on a different machine than the Varnish server.
    • Elaborate self-management of the agent (e.g: support for users, and management of them).
    • Mechanisms that are opaque to a system administrator
    • Front-end code mixed with back-end code
    • "Sessions"

    We've achieved pretty much all of these goals.

    The heart of the agent: The module

    At the heart of the agent, there is the module. As of this writing, there are 14 modules written. The average module is 211 lines of C code (including copyright and license). The smallest module, the echo module, is 92 lines of code (the echo plugin is an example plugin with extensive self documentation). The largest modules, the vlog and vcl modules, are both 387 lines of code.

    To make modules useful, I spent most of the initial work on carving out how modules should work. This is currently how it works:

    • You define a module, say, src/modules/foobar.c
    • You write foobar_init(). This function is the only absolutely required part of the function. It will be run in the single-threaded stage of the agent.
    • You either hook into other modules (like the httpd-module), or define a start function.
    • After all plugins are initialized, the start function of each plugin is executed, if present.

    That's it.

    Since a common task is inter-operation between plugins, an IPC mechanism was needed. I threw together a simple message passing mechanism, inspired by varnish. This lives in src/ipc.c and include/ipc.h. The only other way to currently talk to other modules is through httpd_register (and logger(), but that's just a macro for ipc_run()).

    If you want your foobar.c-plugin to talk to the varnish CLI, you want to go through the vadmin-plugin. This is a two-step process:

    int handle;
    void foobar_init(struct agent_core_t *core)
        handle = ipc_register(core, "vadmin");

    This part of the code gives you a socket to talk to the vadmin module. Actually talking to other modules in foobar_init() is not going to work, since the module isn't started yet.

    And proper etiquette is not to use a global variable, but to use the plugin structure for your plugin, present in core:

    struct foobar_priv_t {
            int vadmin;
    void foobar_init(struct agent_core_t *core)
            struct foobar_priv_t *priv = malloc(sizeof(struct echo_priv_t));
            struct agent_plugin_t *plug;
            plug = plugin_find(core,"foobar");
            priv->vadmin = ipc_register(core,"vadmin");
            plug->data = (void *)priv;
            plug->start = NULL;

    In this example, we have a private data structure for the module, which we allocate in the init function. Every function has a generic struct agent_plugin_t data structure already allocated for it and hooked on to the core->plugins list. This allows you to store generic data, as the core-data structure is the one typically passed around.


    The varnish agent uses a lot of assert()s. This is similar to what Varnish does. It lets you, the developer, state that we assume this worked, but if it didn't you really shouldn't just continue. It's excellent for catching obscure bugs before they actually become obscure. And it's excellent for letting you know where you actually need proper error code.

    Let's take a closer look at the generic struct agent_plugin_t:

    struct agent_plugin_t {
            const char *name;
            void *data;
            struct ipc_t *ipc;
            struct agent_plugin_t *next;
            pthread_t *(*start)(struct
                                agent_core_t *core, const
                                char *name);
            pthread_t *thread;

    The name should be obvious. The void *data is left for the plugin to define. It can be ignored if your plugin doesn't need any data at all (what does it do?).

    struct ipc_t *ipc is the IPC-structure for the plugin. This tells you that all plugins have an IPC present. This is to allow you to run ipc_register() before a plugin has initialized itself. Otherwise we'd have to worry a lot more about which order modules were loaded.

    Next is *next. This is simply because the plugins are par of a linked list.

    the start() function-pointer is used to define a function that will start your plugin. This function can do pretty much anything, but have to return fairly fast. If it spawns off a thread, it's expected that it will return the pthread_t * data structure, as the agent will later wait for it to join. Similar, *thread is used for the same purpose.

    Using the IPC

    You've got a handle to work with, let's use it. To do that, let's look at the vping plugin, starting with init and start:

    static pthread_t *
    vping_start(struct agent_core_t *core, const char *name)
            pthread_t *thread = malloc(sizeof (pthread_t));
            return thread;
    vping_init(struct agent_core_t *core)
            struct agent_plugin_t *plug;
            struct vping_priv_t *priv = malloc(sizeof(struct vping_priv_t));
            plug = plugin_find(core,"vping");
            priv->vadmin_sock = ipc_register(core,"vadmin");
            priv->logger = ipc_register(core,"logger");
            plug->data = (void *)priv;
            plug->start = vping_start;

    vping_init() grabs a handle for the vadmin (varnish admin interface) plugin, and the logger. It also assigns vping_start() to relevant pointer.

    vping_start() simply spawns a thread that runs vping_run.

    static void *vping_run(void *data)
            struct agent_core_t *core = (struct agent_core_t *)data;
            struct agent_plugin_t *plug;
            struct vping_priv_t *ping;
            struct ipc_ret_t vret;
            plug = plugin_find(core,"vping");
            ping = (struct vping_priv_t *) plug->data;
            logger(ping->logger, "Health check starting at 30 second intervals");
            while (1) {
                    ipc_run(ping->vadmin_sock, &vret, "ping");
                    if (vret.status != 200)
                            logger(ping->logger, "Ping failed. %d ", vret.status);
                    ipc_run(ping->vadmin_sock, &vret, "status");
                    if (vret.status != 200 || strcmp(vret.answer,"Child in state running"))
                            logger(ping->logger, "%d %s", vret.status, vret.answer);
            return NULL;

    The vping module was the first module written. Written before the varnish admin interface was a module. It simply pings Varnish over the admin interface.

    This also illustrates how to use the logger: Grab a handle, then use logger(handle,fmt,...), similar to how you'd use printf().

    The IPC mechanism returns data through a vret-structure. For vadmin, this is precisely how Varnish would return it.


    ipc_run() dynamically allocates memory for ret->answer. FREE IT.

    The logger also returns a vret-like structure, but the logger() macro handles this for you.

    Hooking up to HTTP!

    Hooking up to HTTP is ridiculously easy.

    Let's look at echo, comments removed:

    struct echo_priv_t {
            int logger;
    static unsigned int echo_reply(struct httpd_request *request, void *data)
            struct echo_priv_t *echo = data;
            logger(echo->logger, "Responding to request");
            send_response(request->connection, 200, request->data, request->ndata);
            return 0;
    void echo_init(struct agent_core_t *core)
            struct echo_priv_t *priv = malloc(sizeof(struct echo_priv_t));
            struct agent_plugin_t *plug;
            plug = plugin_find(core,"echo");
            priv->logger = ipc_register(core,"logger");
            plug->data = (void *)priv;
            plug->start = NULL;
            httpd_register_url(core, "/echo", M_POST | M_PUT | M_GET, echo_reply, priv);

    This is the ENTIRE echo plugin. httpd_register_url() is the key here. It register a url-base, /echo in this case, and a set of request methods (POST, PUT and GET in this case. DELETE is also supported). A callback to execute and some optional private data.

    The echo_reply function is now executed every time a POST, PUT or GET request is received for URLs starting with /echo.

    You can respond with send_response() as demonstrated above, or the shorthands send_response_ok(request->connection, "Things are all OK!"); and send_response_fail(request->connection, "THINGS WENT BAD");.


    Currently all http requests are handled in a single thread. This means you really really shouldn't block.

    But make sure it's written with thread safety in mind. We might switch to a multi-threaded request handler in the future.

    Know your HTTP

    "REST"-interfaces are great, if implemented correctly. A short reminder:

    • GET requests are idempotent and should not cause side effects. They should be purely informational.
    • PUT requests are idempotent, but can cause side effects. Example: PUT /start can be run multiple times.
    • POST requests do not have to be idempotent, and can cause side effects. Example: POST /vcl/ will upload new copies of the VCL.
    • DELETE requests are idempotent, and can have side effects. Example: DELETE /vcl/foobar.

    Test your code!

    Unused code is broken code. Untested code is also broken code.

    Pretty much all functionality is tested. Take a look in tests/.

    If your code is to be included in an official release, someone has to write test cases.

    I also advise you to add something in html/index.html to test it if that's feasible. It also tends to be quite fun.

    Getting started

    To get started, grab the code and get crackin'.

    I advise you to read include/*.h thoroughly.


    February 11, 2013

    MacYvesDirrty Hax0r in Perl 5.12 for varnish-agent

    Right, so I am not particularly proud of this but again, more of a note to self than anything. Secondly, I am most certainly not a Perl expert. This little hack should probably never be used or put into any production Mac setup. If you know of a proper fix, please do let me know.

    Lastly, this is varnish-agent and not varnish-agent2. The former is written in Perl while the latter is written in C and uses Varnish API directly.

    OK! Now that the disclaimer is out of the way, onward with the dodgy! So when running varnish-agent on OSX 10.8.2, one may encounter the following issue:

    Can't kill a non-numeric process ID at /Library/Perl/5.12/File/ line 124.

    The quick fix is proposed here involves adding this line to the the file on line 124.

    return undef unless $pid > 0;

    So your subroutine in will end up looking more like this.

    sub running {
       my $self = shift;
       my $pid  = $self->_get_pid_from_file;
       return undef unless $pid > 0;
       return   kill(0, $pid)
          ? $pid
          : undef;

    February 06, 2013

    Mikko OhtamaaVarnish shell singleliners: reload config, purge cache and test hits

    Varnish is a server-side caching daemon. On our production server, Varnish listens to port HTTP 80 and serves at the production server front end cache; we use it mainly to serve JS, CSS and static images blazingly fast.

    This blog post is based on default Varnish Ubuntu / Debian installation using apt-get. These instructions were tested on Ubuntu 12.04 LTS and Varnish 3.0.2.

    1. Reload edited Varnish configs

    Varnish caching rules live in /etc/varnish folder. The config entry point (main file) is /etc/varnish/default.vcl. The daemon itself (ports, etc.) is configured by /etc/defaults/varnish.

    Varnish is controlled by an utility program varnishadm.You can use it in a console mode or issue direct command evaluations (think shell, MySQL client). On Ubuntu / Debian default installation varnishadm command as is is enough to control Varnish. However, on custom setup, you might need to guide it to a special console port or point it to a secret file.

    Varnish config load is 2 stage process:

    • Parse and load cfg file to a Varnish memory and give it a handle you can later refer to it
    • Activate config by handle (only possible if step 1 success)

    Below is an one liner shell script which generates a random handle and uses it to load the config if the config parses successfully.

    HANDLE=varnish-cfg-$RANDOM ; \
      varnishadm vcl.load $HANDLE /etc/varnish/default.vcl && \
      varnishadm vcl.use $HANDLE

    2. Purging Varnish cache from command line

    Another useful snippet is to purge all Varnish cache from command line (invalidate all the cache):

    varnishadm "ban.url ."  # Matches all URLs

    Note: Command is purge.url in Varnish 2.x.

    The cache is kept as shared memorymapped file in /var/lib/varnish/$INSTANCE/varnish_storage.bin. When Varnish is running it should map 1 GB (default) of your virtual memory to this file (as seen in ps, top).

    You could also ban by a hostname:

    varnishadm "ban =="

    Here is a shell transcript where we observe that ban works as intended using wget utility.

    # Go to /tmp because wget leaves files around
    cd /tmp
    # 1st load: uncached file, one X-Varnish stamp
    wget -S
    --2013-02-06 20:02:18--
    Resolving (
    Connecting to (||:80... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Server: Apache/2.2.22 (Ubuntu)
      Last-Modified: Sun, 14 Aug 2011 22:55:01 GMT
      ETag: "2000893-108ec-4aa7f09555b40"
      Cache-Control: max-age=3600
      Expires: Wed, 06 Feb 2013 23:02:19 GMT
      Content-Type: image/jpeg
      Content-Length: 67820
      Accept-Ranges: bytes
      Date: Wed, 06 Feb 2013 22:02:19 GMT
      X-Varnish: 705602514
    # 2st load: cached file, two X-Varnish stamps
    wget -S
    --2013-02-06 20:02:21--
    Resolving (
    Connecting to (||:80... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Server: Apache/2.2.22 (Ubuntu)
      Last-Modified: Sun, 14 Aug 2011 22:55:01 GMT
      ETag: "2000893-108ec-4aa7f09555b40"
      Cache-Control: max-age=3600
      Expires: Wed, 06 Feb 2013 23:02:19 GMT
      Content-Type: image/jpeg
      Content-Length: 67820
      Accept-Ranges: bytes
      Date: Wed, 06 Feb 2013 22:02:22 GMT
      X-Varnish: 705602515 705602514
    # Purge
    varnishadm "ban.url ."
    # It's non-cached again
    wget -S
    --2013-02-06 20:02:34--
    Resolving (
    Connecting to (||:80... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Server: Apache/2.2.22 (Ubuntu)
      Last-Modified: Sun, 14 Aug 2011 22:55:01 GMT
      ETag: "2000893-108ec-4aa7f09555b40"
      Cache-Control: max-age=3600
      Expires: Wed, 06 Feb 2013 23:02:35 GMT
      Content-Type: image/jpeg
      Content-Length: 67820
      Accept-Ranges: bytes
      Date: Wed, 06 Feb 2013 22:02:35 GMT
      X-Varnish: 705602516

    3. Restart Varnish on Ubuntu

    This forces config flush, not sure about whether cache file storage gets reset(?).

    service varnish restart

    4. Further ideas

    If someone knowns where to get Varnish VCL syntax highlighter for Sublime Text 2 (TextMate) that would make my life easier, used in the combination SFTP plug-in.


     Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

    MacYvesBuilding on varnish-agent2

    The varnish-agent2 existing code base is pretty solid,  rather beautiful I might add. These simple words, keep it simple, has been realised by the following rule of thumb.

    – Close to 0 configuration
    – “Just works”
    – Maintainable
    – Generic
    – Stateless

    For those that are keen to get started on the varnish-agent2 code base, I hope that this document will be of some use. I have used a tiny subset of the Concurrent Object Modeling and architectural design mEThod (COMET), particularly the Requirements Modeling, Analysis Modeling and snippets of the actual Design Model.

    Lastly, this document mostly serves as a mental note for myself :)

    Requirements Modeling

    The requirement for the vagent2 is to provide an extendible restful interface for varnish cache. In addition, vagent2 act as the point of integration for varnish cache with other systems – e.g. an administrative or monitoring system.

    The use cases are simple right now, and is depicted in the use case diagram below.

    vagent2 use cases

    vagent2 use cases

    Analysis Modeling

    vagent2 is designed to support the full spectrum of HTTP method types. A user of the vagent2 will issue these HTTP requests and receive JSON data as response where applicable. Furthermore, the vagent2 is built with modules in mind to address a potentially expanding feature set. Lastly, each modules should be able to communicate and reused by another module.

    IPC lies at the heart of the varnish-agent2 code base and message passing is the norm here for implementing an event driven model. Each module follows vagent2’s plugin paradigm, and comes equipped with the module’s own private data set. The plugin therefore follows the IPC set of callback methods, such as ipc_start and ipc_run. These methods are assigned to the appropriate functions within each module.

    For the module to be exposed as a restful interface, a simple httpd_register method will hook in the module’s reply method of choice, and expose it appropriately.

    For any module, the basic dependencies is depicted below.

    Module breakdown

    basic module dependencies

    Static Modeling

    varnish-agent2 ships with a few core modules, such as the logger, shmlog access, httpd and ipc modules. vstatus and vadmin provides access to the shmlog via Varnish API. Note that as of writing this blog, varnish 3.0.3 was used.

    These aforementioned modules provide the building blocks for managing varnish cache. For an overview of the static models, see the class diagram below.

    Static Model

    Static overview of vagent2

    Dynamic Modeling

    The process of initialising a module is rather straightforward. First, add a new init method for the new module in plugins.h, ensure that you have called the init method in main.c, and of course, allocated some memory for the new plugin in main.c too.

    This new module must provide an implementation of the new init method. See diagram below depicting vban’s initialisation process.

    vban initialisation process

    vban initialisation process

    Once initialised, by hooking onto struct agent_plugin* structure, the new module will correspond to the IPC life cycle.

    plugin->start  = ipc_start;
    plugin->ipc->priv = your_private_plugin_data;
    plugin->ipc->cb = your_plugin_callback;

    plugin->start is called when your plugin starts. Note that you need to assign a start method if you want the IPC to execute your callback.

    plug->ipc->priv refers to a persisted data for your plugin. This can be anything, and as a rule of thumb, this is a good place to hold references to other modules.

    plug->ipc-cb refers to the callback method of when ipc_run is issued by another module.

    A sample execution path

    To tie it all together, the collaboration diagram below illustrate the execution path of issuing a ban to the vagent2. Note that ipc is used to reach the vadmin module.

    Issue a ban

    Issue a ban

    January 31, 2013

    Kristian LyngstølThe Varnish Agent 2.1

    Posted on 2013-01-31

    We just released the Varnish Agent 2.1.

    (Nice when you can start a blog post with some copy/paste!)

    Two-ish weeks ago we released the first version of the new Varnish Agent, and now I have the pleasure of releasing a slightly more polished variant.

    The work I've put in with it the last couple of weeks has gone towards increasing stability, resilience and fault tolerance. Some changes:

    • Fixed several memory leaks
    • Fixed JSON formatting that broke the log output
    • Add privilege separation
    • Handle varnishd restart much better
    • Handle changing of the -T option without restarting the agent
    • Log assert errors to syslog
    • Add site specific javascript

    For a complete-ish log, see the closed tickets for the 2.1 milestone on github.

    This underlines what we seek to achieve with the agent: A rock stable operational service that just works.

    If you've got any features you'd like to see in the agent, this is the time to bring them forth!

    I've already started working on 2.2 which will include a much more powerful API for the varnishlog data (see docs/LOG-API.rst in the repo), and improved HTTP handling, including authentication.

    So head over to the demo, play with it, if you break it, let me know! Try to install the packages and tell me about any part of the installation process that you feel is awkward or not quite right.


    January 22, 2013

    Kristian LyngstølThe Varnish Agent

    Posted on 2013-01-22

    We just released the Varnish Agent 2.0.

    The Varnish Agent is a HTTP REST interface to control Varnish. It also provides a proof of concept front-end in html/JavaScript. In other words: A fully functional Web UI for Varnish.

    We use the agent to interface between our commercial Varnish Administration Console and Varnish. This is the first agent written in C and the first version exposing a HTTP REST interface, so while 2.0 might suggest some maturity, it might be wiser to consider it a tech preview.


    I've written the agent for the last few weeks, and it's been quite fun. This is the first time I've ever written JavaScript, and it was initially just an after thought that quickly turned into something quite fun.

    Some features:

    • Mostly self documenting, so it should be simple to integrate in other environments.
    • Close to 0 configuration. In fact, it currently requires 0 configuration, but might eventually require a tiny bit for authentication.
    • "Unit tests" for most functionality.
    • Upload and download VCL. Uploaded VCL is stored to disk.
    • Deploy vcl (e.g: use it). A hard link points to the most recently deployed VCL, allowing you to use that on varnish boot.
    • Show and change parameters.
    • Stop/start Varnish, show/delete panics, show status, etc
    • Varnishstat: Retrieve varnishstat data in json format
    • Varnishlog data: Retrieve varnishlog data in json format (Historic data only atm).
    • Modularised.

    I've had a lot of fun hacking on this and I hope you will have some fun playing with it too!


    January 19, 2013

    ops42When speed doesn’t matter

    Let’s talk about speed. Speed is important. Varnish is a synonym for speed. That far, that good. But am I really the only one who doesn’t get why it would be so important to be able to PURGE 4000 objects a second? Really, who the fuck cares? Wouldn’t it show a bigger problem if I […]

    January 09, 2013

    MacYvesBuilding Varnish Cache 3.0.3 from source in Mountain Lion, OSX 10.8.2

    If you want to install Varnish Cache in OSX, I highly recommend using Homebrew’s recipe for installing Varnish. Easy!

    But if you want to build and run Varnish Cache from github, here is a little step-by-step checklist for you.

    A shopping list of applications you’ll need for your Mountain Lion:

    • Homebrew 0.9.3
    • Xcode 4.5.2, via Appstore
    • Command Line Tool for Mountain Lion – November 1st 2012

    You’ll need the following dependencies via Homebrew

    • automake, note that this has to be version 1.12. I have tested it with 1.13 and you will run into obsolete AM_CONFIG_HEADER. 
    • libtool
    • pcre
    • pkg-config

    You’ll need docutils for rst2man

    Run the following steps:

    1. brew install the above dependencies
    2. for automake, you will need to switch it to 1.12. See below
    3. install docutils
    4. git checkout 3.0.3 branch of Varnish Cache
    5. run ./
    6. run ./configure
    7. make install

    To install another version of automake with Homebrew:

    1. Unlink the existing 1.13 version – brew unlink automake
    2. Get the list of versions available for automake – brew versions automake
    3. Copy the 1.12 information from git checkout…, for example git checkout 1e5eb62 /usr/local/Library/Formula/automake.rb
    4. cd into /usr/local/, or where ever else you have Homebrew storing your formula
    5. Paste the copied git information and this will check out the appropriate version of automake for you
    6. Install 1.12 – brew install automake
    7. Voila! You now have 1.12 version and you can switch between 1.13 or 1.12 by simply doing brew switch automake 1.13

    November 13, 2012

    ops42Varnish is smarter than you

    I just realized that a week without offending your intelligence is no good week by my standards. So here goes. Did you know that Varnish works out of the box? Provided, of course, you specify a backend to use. In a perfect world your backend would actually respond with a proper Cache-Control Header and Varnish […]

    November 06, 2012

    ops42Neuter Varnish’s keep-alive connections to skip that pesky session replication

    Varnish is all about speed. You know that. I know that. Heck, everybody knows that. That’s what a catchphrase is for. To that end, Varnish will keep backend connections open using keep-alive to shave off that precious milliseconds it would waste opening new ones all the time. In a world without your average application server […]

    November 05, 2012

    ops42How to rid the interwebs of millions of posts about Varnish Purging

    If I had to complain about something – and let’s face it, that’s what I do – I’d have to say it’s the millions of posts explaining the technique to purge something in Varnish. If it’s explained in the docs then spare me the details about how it’s supposedly done. Either stay quiet or link […]

    October 29, 2012

    ops42Varnish needs to compensate for the shortcomings of oversimplified Apache configurations

    I am in the fortunate situation that, for the most part, I can trust my backend responses regarding the Cache-Control headers. This spares me the tedious work of setting a proper ttl and cleaning up the response headers. Let’s face it, the cache layer cannot (and should not) always know how long a certain response […]

    ops42Varnish the gatekeeper, Rule #1

    I know, it just seems too obvious to be worth mentioning, but if I have learned anything from scanning through access log files from a website with 3-digit million requests a day, it is that there is nothing that won’t be thrown your way. So better be safe than sorry and make sure only valid […]

    ops42One Varnish configuration to rule them all

    As part of my day job I am “the Varnish guy” and thus have to keep track of quite some servers for the development, QA, staging and production environment. Of course there are technical differences, yet I like to have “the one” Varnish configuration for all of them. For Varnish this means different backend and […]

    September 19, 2012

    cd34AngularJS – a first step

    If you’ve not heard of AngularJS, I’m not surprised. It moves MVC or MV* to the browser and provides a unique, lightweight, powerful way to write apps that execute on the client side. Forms, validation, controllers, routes are all handled fairly easily. The one thing I look for in every framework I use is a […]

    September 04, 2012

    Tollef Fog HeenDriving Jenkins using YAML and a bit of python

    We recently switched from Buildbot to Jenkins at work, for building Varnish on various platforms. Buildbot worked-ish, but was a bit fiddly to get going on some platforms such as Mac OS and Solaris. Where buildbot has a daemon on each node that is responsible for contacting the central host, Jenkins uses SSH as the transport and centrally manages retries if a host goes down or is rebooted.

    All in all, we are pretty happy with Jenkins, except for one thing: The job configurations are a bunch of XML files and the way you are supposed to configure this is through a web interface. That doesn't scale particularly well when you want to build many very similar jobs. We want to build multiple branches, some which are not public and we want to build on many slaves. The latter we could partially solve with matrix builds, except that will fail the entire build if a single slave fails with an error that works on retry. As the number of slaves increases, such failures become more common.

    To solve this, I hacked together a crude tool that takes a yaml file and writes the XML files. It's not anywhere near as well structured and pretty as liw's jenkinstool, but it is quite good at translating the YAML into a bunch of XML files. I don't know if it's useful for anybody else, there is no documentation and so on, but if you want to take a look, it's on github.

    Feedback is most welcome, as usual. Patches even more so.

    June 19, 2012

    Kristian LyngstølCaching popular content longer

    Posted on 2012-06-19

    The following VCL demonstrates how you can tell Varnish to cache popular content longer than less popular content. Discussion follows.

    sub vcl_hit {
            if (obj.hits == 500) {
                    set obj.ttl = 10w;
    sub vcl_fetch {
            set beresp.ttl = 20m;

    In this example all content starts out with a 20 minute TTL (set beresp.ttl = 20m;). If an object is hit exactly 500 times (if (obj.hits == 500) { , we increase the ttl to 10 weeks (set obj.ttl = 10w;).


    You have to use obj.hits == 500, not obj.hits > 500. If you use the latter, you will increase the TTL every time an object is hit once it's been hit 500 times.

    This is one of the snippets of VCL I'm quite proud of. Unlike the evil backend hack ( which is also quite clever, this ttl-adjustment is far less hacky.

    This allows you to let less requested content expire faster to get more out of a full cache. There's nothing wrong with using this in multiple tiers either:

    sub vcl_hit {
            if (obj.hits == 500) {
                    set obj.ttl = 3h;
            } elsif (obj.hits == 10000) {
                    set obj.ttl = 2d;
            } elsif (obj.hits == 1000000) {
                    set obj.ttl = 4w;
    sub vcl_fetch {
            set beresp.ttl = 20m;

    I'm not convinced the 4 week part is useful here, but that's up to you to decide.

    Of course, setting the ttl to 20 minutes by hand in vcl_fetch is optional. You could just rely on the default mechanisms Varnish offer (

    These are the things to keep in mind:

    • If you run out of cache, varnish will throw out the content that has been inactive for the longest period of time, not necessarily the oldest content.
    • If your cache is full and you have mixed object sizes, Varnish' LRU logic can struggle. If it's not too hard to avoid, you want to avoid LRU nuking.
    • Finding the right numbers (500 hits? 100 hits? 20 hits? 10000 hits?) will depend heavily on your own setup. Experiment.
    • varnishlog -O -i TxHeader -I X-Hits: 500 combined with sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Hits = obj.hits; } } will output a line every time an object has been hit 500 times (well, technically it'll output once you've /sent/ an object that has been hit 500 times).
    • 20 minutes is an example. Experiment.

    Last, I'd just like to say that this is a solution I came up with for a customer, pretty much on the spot. We had never thought about it until the customer asked, but VCL made it possible and we know how to use it. What I'm trying to say is: Buy our stuff! ;) We are Varnish Software: Really though, if you have a Varnish-question, we can probably help you out.


    April 25, 2012

    Kristian LyngstølAnnouncing Spew

    Posted on 2012-04-25

    I just pushed my HTTP request spewer, spew, to github.

    It's Linux-specific, since it uses epoll, and the http.c-code is still nasty, but it's also fast.

    A reminder of what it can do:

    The feature list contains:

    • Spew opens N connections, sends M requests over each connection and then re-opens the connection and repeats.
    • Fast request generation
    • "Keep-Alive support" (sort of!)
    • No analysis at all.
    • Native IPv6 support
    • Configurable, both on the command line and config file
    • It survives despite what happens if you bring your HTTP server down for a moment. It might need some time to catch up, though.

    Most of the boilerplate code is actually from an old defunct project I abandoned in 2009. All the stuff that deals with options and config files and debug messages and whatnot. The only thing I've done recently is src/http.c and integration.

    Also: I know the code is still horrible. Patches are welcome, as are requests, (constructive!) comments, etc.


    April 23, 2012

    Kristian LyngstølNo Stress

    Posted on 2012-04-23

    After my last post about testing Varnish (, and a few years of frustration, I decided to take a look at what is actually possible.

    So this is an example:

    What you are seeing is Varnish doing 183k req/s on my home machine. The important thing, however, is that the tool generating this load, cleverly called a.out, is running at 22% CPU load, and it's the single-threaded result of one day of dirty hacking. Compare this to httperf which is hard to get over 100k req/s, or siege which kills itself at about 15k req/s on the same machine.

    This being a prototype, the code will never see the light of day. Trust me when I say it's horrible - that's what you get from a day of fiddling. However, it has demonstrated to me what's possible, and I might re-start this project now that I have an idea of what I want to do.

    As for how a.out works? It's connection-oriented, so it maintains N open connections at any given time and spews M requests over each connection in rather large (configurable) bursts. It also manages to NOT die if you stop the server for a while (httperf doesn't like this and siege doesn't really need any help to murder itself). It collects roughly 0 statistics and it does not really care about response.

    So no, it's not very good. But it's fast, and that was what I set out to achieve.

    Update: So this tool is a bit more powerful. I've been able to do 280-290k req/s with a single process (and thread). This is the same machine I did the 275k req/s record with using httperf, but that required two extra machines to generate traffic.... Will be interesting to try booting those tomorrow. Comments

    April 19, 2012

    Kristian LyngstølTesting Varnish

    Posted on 2012-04-19

    • How do I benchmark Varnish?
    • How do I make sure Varnish is ready for production?
    • How do I test Varnish?

    These are questions I see people ask frequently, and there is no simple answer. In this blog post, I'll go through some of what you could do to test a Varnish-site.

    What this blog post is not

    This is not about benchmarking. I don't do benchmarking, never have. Why not? Because it's exceedingly hard and very few people succeed at proper benchmarking.

    Neither is this a blog post about testing functionality on your site. You should be doing that already. I'll only say that you should test functionality, and it's often best done by browsing the site.

    Also, don't expect it to be all that complete. Ask questions in comments and I might expand upon it!

    What to test

    Despite what most people ask about, the tools you chose are not nearly as important as what you want to test.

    If you are hosting videos, I doubt testing request/second is a sensible metric. There are a few things you need to ask yourself:

    • What is going to be the bounding factor of my site? Bandwidth? Disk I/O? CPU? Memory? Something else?
    • How much of my site is it possible to cache?
    • Is there any way to tell ahead of time?

    These questions are important and they relate.

    If your site is already in production under some different architecture, you are in luck. Your access logs can tell you a lot about the traffic pattern you can expect. This is a great start.

    If this is a new site, though, it can be harder to estimate the answers. I recommend starting with How much of my site is it possible to cache?. If your site is mostly static content, then Varnish will be able to help you a lot, assuming you set things up accordingly. If it's a site for logged in users, you have a much harder task. It's still possible to cache content with Varnish, but it's much harder. The details of how to do that is beyond the scope of this post.

    As long as you can cache the majority of the content, chances are you will not be CPU bound as far as Varnish is concerned.

    Getting a baseline

    Testing a Varnish-site can be really fast or you can use the next six months doing it. Let's start by getting a baseline.

    I usually start out by something truly simple: Look at varnishstat and varnishlog while you use a browser to browse the site. It's important that this is not a script, because your users are likely using browsers too and you want to catch all the stuff they catch, like cookies.

    To set this up, the best way is to modify /etc/hosts (or the Windows equivalent (there is one, all the viruses uses it)). The reason you don't want to just add a test-domain is because your site will go on-line using a real domain, not a test-domain. A typical /etc/hosts file could look like this for me:       localhost freud
    # The following lines are desirable for IPv6 capable hosts
    ::1     ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters

    Even better is if this is an external server. Then make sure you block any access to other port-80 services for that site. This will ensure that you don't miss any sub-domains.

    What you are looking for is cache hits, misses and hitpasses. This should reveal if the site handles cookies properly or not. You may also want to fire up a different browser.

    You also want to keep a look out for:

    • Vary headers.
    • Strange or unexpected Cache-Control headers
    • Set-Cookie headers.
    • And of course: 404s or other errors.

    Once you've got this nailed down. If you doubt the speed of Varnish, we can always throw in wget too:

    wget --delete-after -p -r

    This will give you a recursive request for with all prerequisites (CSS, images, etc). It's not very useful in itself but it will give you a feel for how fast the site is without Varnish and then after you've cached it with Varnish. You can easily run multiple wget-commands in parallel to gauge the bandwidth usage:

    while sleep 1; do wget --delete-after -p -r; done &
    while sleep 1; do wget --delete-after -p -r; done &
    while sleep 1; do wget --delete-after -p -r; done &

    Ideally this should be network-bound, but realistically speaking, wget is not /that/ fast when it comes to tiny requests.


    Keep in mind that you are likely going to be hitting a DNS server frequently, specially if you don't use /etc/hosts. I've had DNS servers running at 50-70% CPU when I've done stress testing in the past, which means the DNS server is affecting the test more than you want it to.

    So far none of these tricks have been very fancy.

    Bringing out the big guns

    So you wont reach 275 kreq/s using wget. I'm not sure that should be a goal either, but it's worth while taking a look at.

    If you are moving on to testing just Varnish, not the site itself, then it's time to move away from browsers and wget. There are several tools available for this, and I tend to prefer httperf. It's not a good tool by any sensible measure, but it's a fast one. The best way to learn httperf is to stick all the arguments into a text file and set up a shell script that randomly picks them until you find something that works. The manual pages are unhelpful at best.

    An alternative to httperf is siege. I'm sure siege is great, if you don't mind that it'll run into a wall and kill itself long before your web server. If you want further proof, take a look at this part of siegerc, documenting Keep-Alive:

    # Connection directive. Options "close" and "keep-alive"
    # Starting with release 2.57b3, siege implements persistent.
    # connections in accordance to RFC 2068 using both chunked
    # encoding and content-length directives to determine the.
    # page size. To run siege with persistent connections set
    # the connection directive to keep-alive. (Default close)
    # CAUTION: use the keep-alive directive with care.
    # DOUBLE CAUTION: this directive does not work well on HPUX
    # TRIPLE CAUTION: don't use keep-alives until further notice
    # ex: connection = close
    #     connection = keep-alive
    connection = close

    A stress testing tool that doesn't support keep-alive properly isn't very helpful. Whenever I use siege, it tends to max out at about 5000-10000 requests/second.

    There's also Apache Bench, commonly known as just ab. I've rarely used it, but what little use I've seen from it has not been impressive. It supports KeepAlive, but my brief look at it showed no way to control the KeepAlive-ness. From basic tests of it, it also seemed slightly slower than httperf. It does seem better today than it was the first time I looked at it, though. For this blog posts, I'll use httperf simply because it's the tool I'm most familiar with and which have given me the right combination of control and performance.

    However, httperf has several flaws:

    • It is single-threaded. It can do multiple concurrent requests, but only on a single thread. This can be leveraged by running multiple instances.
    • The documentation, while mostly complete, does not really answer enough questions.
    • It tends to max out at 1022-ish concurrent connections due to an internal limit. This might be possible to avoid if you compile it yourself. I've never bothered.
    • Bussy-loops! Beware that httperf using 100% cpu does NOT mean that it is running at full capacity.
    • No graceful slowing down if you try to hit a --rate that's too fast. It'll simply give connection errors instead.

    The trick to httperf is to use --rate when you can. A typical httperf command might look like this (run on my laptop):

    $ httperf --rate 2000 --num-conns=10000
            --num-calls 20 --burst-length 20 --server localhost
            --port 8080 --uri /misc/dummy.png
    httperf --client=0/1 --server=localhost --port=8080
            --uri=/misc/dummy.png --rate=2000 --send-buffer=4096
            --recv-buffer=16384 --num-conns=10000 --num-calls=20
    httperf: warning: open file limit > FD_SETSIZE; limiting max. # of
             open files to FD_SETSIZE
    Maximum connect burst length: 20
    Total: connections 10000 requests 200000 replies 200000 test-duration 7.076 s
    Connection rate: 1413.2 conn/s (0.7 ms/conn, <=266 concurrent connections)
    Connection time [ms]: min 1.6 avg 70.4 max 3049.8 median 33.5 stddev 286.4
    Connection time [ms]: connect 27.9
    Connection length [replies/conn]: 20.000
    Request rate: 28264.9 req/s (0.0 ms/req)
    Request size [B]: 76.0
    Reply rate [replies/s]: min 39514.8 avg 39514.8 max 39514.8 stddev 0.0 (1 samples)
    Reply time [ms]: response 35.9 transfer 0.0
    Reply size [B]: header 317.0 content 178.0 footer 0.0 (total 495.0)
    Reply status: 1xx=0 2xx=200000 3xx=0 4xx=0 5xx=0
    CPU time [s]: user 1.33 system 5.61 (user 18.8% system 79.3% total 98.0%)
    Net I/O: 15761.0 KB/s (129.1*10^6 bps)
    Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
    Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

    Note that httperf will echo the command you ran it with back to you, with all options expanded. I took the liberty of formatting the output a bit more to make it easier to read. The options I use here are:

    • --rate 2000 - Tries to open 2000 connections per second.
    • --num-conns=10000 - Open a total of 10 000 connections for this test.
    • --num-calls=20 - Perform 20 requests per connection. (so a total of 10 000*20 requests (200 000)).
    • --burst-length 20 - Pipeline the requests. This is mainly to speed up httperf itself since it's much faster to send all 20 requests in one go than send them individually. Varnish handles it correctly anyway.
    • The rest should be self explanatory.

    The first thing you should look at in the output is Errors:. If you get errors, there's a very good chance you were too optimistic with your --rate setting. Also note that the uri matters greatly. /misc/dummy.png is just that: a dummy-png I have to test. (see for yourself at Let's try the same with the front page:

    $ httperf --rate 2000 --num-conns=10000 --num-calls 20
            --burst-length 20 --server localhost --port 8080 --uri /
    httperf --client=0/1 --server=localhost --port=8080 --uri=/
            --rate=2000 --send-buffer=4096 --recv-buffer=16384
            --num-conns=10000 --num-calls=20 --burst-length=20
    httperf: warning: open file limit > FD_SETSIZE; limiting max. # of
             open files to FD_SETSIZE
    Maximum connect burst length: 42
    Total: connections 1738 requests 34760 replies 34760 test-duration 10.589 s
    Connection rate: 164.1 conn/s (6.1 ms/conn, <=1018 concurrent connections)
    Connection time [ms]: min 477.3 avg 4592.4 max 8549.5 median 5276.5 stddev 2233.3
    Connection time [ms]: connect 8.7
    Connection length [replies/conn]: 20.000
    Request rate: 3282.8 req/s (0.3 ms/req)
    Request size [B]: 62.0
    Reply rate [replies/s]: min 3077.3 avg 3311.2 max 3545.1 stddev 330.8 (2 samples)
    Reply time [ms]: response 3772.9 transfer 49.8
    Reply size [B]: header 326.0 content 38915.0 footer 2.0 (total 39243.0)
    Reply status: 1xx=0 2xx=34760 3xx=0 4xx=0 5xx=0
    CPU time [s]: user 0.61 system 9.54 (user 5.8% system 90.1% total 95.9%)
    Net I/O: 125998.3 KB/s (1032.2*10^6 bps)
    Errors: total 8262 client-timo 0 socket-timo 0 connrefused 0 connreset 0
    Errors: fd-unavail 8262 addrunavail 0 ftab-full 0 other 0

    Now see how the errors piled up. This is because we exceeded the performance httperf could offer. Yeah, httperf is far from perfect. Also note the bandwidth usage and CPU usage. I'm not sure if it's a coincidence that we're so close to gigabit, since this is a Varnish server running on localhost.

    What you also may want to look at is reply status, to check the status codes. You also want to pay attention to connection times. Let's take a look at the first example again:

    Connection rate: 1413.2 conn/s (0.7 ms/conn, <=266 concurrent connections)
    Connection time [ms]: min 1.6 avg 70.4 max 3049.8 median 33.5 stddev 286.4
    Connection time [ms]: connect 27.9
    Connection length [replies/conn]: 20.000

    This tells me the average connection time was 70.4ms, with a maximum at 3049.8ms. 3 seconds is quite a long time. You may want to look at that. What I do when I debug stuff like this is make sure that I rule out the tool itself as the source of worry. There is no 100% accurate method of doing this, but given the CPU load of httperf at the time, it's reasonable to assume httperf is part of the problem here. You can experiment by slightly adjusting the --rate option to see if you're close to the breaking point of httperf.

    You also want to watch varnishstat during these tests.

    So what did you just learn?

    Frankly very little.

    Sure, this means I can run Varnish at around 30k req/s on my laptop, testing FROM my laptop too. But this is not that helpful.

    What settings should I use?

    Well, first of all, running 20 requests over a single connection is pointless. There's almost no browser or site out there which will cause this to happen. Depending on the site, numbers between 4 and 10 requests per connection is more realistic.

    If all you want is a big number, then tons of requests over a single connection is fine. But it has nothing to do with reality.

    You can get httperf to do some pretty cool things if you invest time in setting up thorough tests. It can generate URLs, for instance, if that's your thing. Or simulate sessions where it asks for one page, then three other pages over the same connection X amount of time later, etc etc. This is were the six-month testing period comes into play.

    I consider it a much better practice to look at access logs you have and use something simpler to iterate the list. wget can do it, and I know several newspapers that use curl for just this purpose. It was actually curl that first showed me what happens when Varnish becomes CPU bound without having a session_linger set (this is set by default now, but for the curious, what happened was that the request rate dropped to a 20th of what it was a moment before, due to context switching).


    Test your site and by all means test Varnish, but do not assume that just because httperf or some other tool gives you 80 000 requests/second that this will match real-life traffic.

    Proper testing is an art and this is just a small look at some techniques I hope people find interesting.


    March 28, 2012

    Kristian LyngstølThe Varnish Book

    Posted on 2012-03-28

    In 2008 Tollef Fog Heen wrote the initial slides used for our first Varnish System Administration training course. Since then, I've been the principal maintainer and author of the material. We contracted Jérôme Renard ( to adapt the course for web developers in 2011 and I've spent some time pulling it all together into one coherent book after that.

    Today we make the Varnish Book available on-line under a Creative Commons CC-BY-NC-SA license.

    See for the sphinx-rendered HTML variant.

    The All-important content!

    The book contains all the content we use for both the system administration course and the web developer course. While each of those courses omit some chapters (Sysadmin omits HTTP and Content Composition while Webdev omits the Tuning and Saving a Request chapters), the content is structured with this in mind.

    The book does not teach you everything about Varnish.

    If you read the book and perform the exercises (without cheating!) you will:

    • Understand basic Varnish configuration and administration
    • Know when modifying parameters is sensible
    • Have extensive understanding of VCL as a state engine and language
    • Understand how caching in HTTP affects Varnish, intermediary caches and browser caches
    • Know how to use Varnish to recover from or prevent error-situations
    • Have in-depth knowledge of the different methods of cache-invalidation that Varnish can provide
    • Know several good ways to approach cookies and user-specific content
    • Know of Edge Side Includes
    • Much more!

    I've gradually restructured the book to be about specific tasks instead of specific features. Because of that, there are several features or techniques that are intentionally left out or down-played to make room for more vital concepts. This has been a gradual transition, and I'm still not entirely satisfied, but I believe this approach to learning Varnish is much more effective than trying to just teach you the facts.

    One of my favorite examples of this is probably the Cache Invalidation chapter. We used to cover the equivalent of purging in the VCL chapters, since it is a simple enough feature, then cover banning in a separate chapter. The problem with that mentality is that when you are setting up Varnish, you don't think "I need to write VCL". You think "I need to figure out how to invalidate my cache" or "how do I make Varnish fall back to a different web server if the first one failed".

    I have learned a great deal about Varnish, about how people learn and about the web in general while holding these training courses and writing this book. I hope that by releasing it in the open, even more people will get to know Varnish.

    Future Content

    The book will continue to change. We at Varnish Software ( will update it for new versions of Varnish and take care of general maintenance.

    I hope that we will also get some significant feedback from all you people out there who will read it. We appreciate everything from general evaluation and proof reading to more in-depth discussions.

    One of the more recent topics I want to cover in the book is Varnish Modules. This is still quite new, so I'm in no rush. I still haven't decided what that chapter should cover. It might be about available vmods and practical usage of them, or we might go more in depth and talk about how to start writing your own. I really don't know.

    An other topic I really wish to expand upon is content composition. The material Jerome provided for us was excellent, but I wish to go broader and also make it available in a couple of other languages than just PHP. There is some work in this area already, I just can't say much more about it yet...

    You will probably also see rewrites of the first chapter and the Programs-chapter in the near future. They are both overdue for a brush-up.

    In the end, though, this a book that will continue to evolve as long as people take interest. What it covers will be defined largely based on feedback from course participants, feedback from people reading it on-line and the resources needed to implement those changes.


    We chose a CC-BY-NC-SA license because we both want to make the material available to as many people as possible, and make sure that we don't put our self out of the training business by providing a ready-made course for potential competitors.

    Being of of those people who actually read licenses and try to interpret their legal ramifications, I've obviously also read the CC-BY-NC-SA license we use. It is (intentionally) vague when it comes to specifying what "non-commercial" means. What I interpret it as with regards to our training material is that you can read it as much as you want regardless of whether you are at work or what commercial benefits you have from understanding the content. You can also hold courses in non-profit settings (your local LUG for instance), and some internal training will probably be a non-issue too. However, the moment you offer training for a profit to other parties, you're violating the license. You'll also be violating it if you print and sell the book for a profit. Printing and selling it to cover the cost of printing is allowed (it's one of the few things where the license actually clarifies this).

    Since we are using a "NC"-license, we'll also be asking for copyright assignment and/or a contributor's agreement if you wish to contribute significantly. This is so we can use your material in commercial activities. Exactly how this will be done is not yet clarified.

    One last point: If you are contributing to the documentation we keep at, we will not consider it a breach of license if you borrow ideas from the book. Our goal is to make sure the book interacts well with the other documentation while covering our expenses.


    March 18, 2012

    cd34REMOTE_ADDR handling with Varnish and Load Balancers

    While working with the ever present spam issue on this blog, I’ve started to have issues with many of the plugins not using the correct IP address lookup. While each plugin author can be contacted, trackbacks and comments through WordPress still have the Varnish server’s IP address. In our vcl, in vcl_recv, we put the […]

    March 08, 2012

    Ingvar Hagelundvarnish-3.0.2 for fedora

    I finally got around to wrap up varnish-3.0.2 for fedora 17 and rawhide. Please test and report karma.

    In this release, I have merged changes from the upstream rpm, and added native systemd support for f17 and rawhide. It also builds nicely for epel5 and epel6, providing packages quite similar to those available from the varnish project repo.

    As epel does not allow changes in a package API after release, varnish-3.0.2 won’t be available through epel5 or epel6, so use the varnish project repo, or my precompiled packages for epel 4, 5 and 6 available here.

    As always, feedback is very welcome.

    January 28, 2012

    cd34First beta of my new webapp,

    After a late night working until 2:45am on the finishing touches, my phone alerted me to the fact it was 5:35am and time for the first Live Photostream to take place. The stream? Roger Waters – The Wall, from the Burswood Dome in Perth, AUSTRALIA. Special thanks to Paul Andrews for taking the pictures and […]

    January 10, 2012

    cd34Finally, a formal release for my WordPress + Varnish + ESI plugin

    A while back I wrote a plugin to take care of a particular client traffic problem. As the traffic came in very quickly and unexpectedly, I had only minutes to come up with a solution. As I knew Varnish pretty well, my initial reaction was to put the site behind Varnish. But, there’s a problem […]

    December 01, 2011

    Kristian LyngstølVarnish Training

    Posted on 2011-12-01

    As anyone who's worked with me should realize by now, I'm big on documentation, be it source code comments or other types of documentation. The only reason I'm not more active in the documentation section of Varnish Cache is because I've maintained our (Varnish Software's) training material ever since Tollef Fog Heen wrote the initial slides in 2009.

    I've held the course more times than I can remember, and usually done improvements after every course. Others have also held the course, including Redpill Linpro's Kacper Wysocki, maintainer of security.vcl ( and Magnus Hagander (Postgresql god/swede). Feedback and gradual improvements have turned a set of slides into a pretty good course material.

    We recently started holding on-line courses too. This revealed several new challenges. The obvious challenges are things like getting basic voice communication working (it sounds easy, but you'd be amazed...). It was also interesting when I held the course in my apartment on Australian time, and my ISP decided to perform maintenance on my cable modem (it was 2AM local, after all). So I've had to hold the course on a 3G connection, communicating with Australia. Fun. Then there's the lack of or severely reduced feedback, which presents challenges in how we do exercises and generally deal with the course. In a class room I can easily determine if the participants are able to keep up, if I'm going too slow or too fast and whether or not a subject is more interesting than an other. All of that is, at best, very difficult in an on-line session.

    The last few weeks I've finally gotten around to merging the sysadmin course with the web development course that Jérôme Renard has written for us. It proved the perfect opportunity to give the course an other big update. While the course was already updated for Varnish 3, I've made several other Varnish 3-related additions. More importantly is that the flow of the course has changed from one oriented on Varnish functionality to tasks you wish to accomplish with Varnish. Instead of teaching you about Varnish architecture first, then Varnish parameters, the course now has a chapter devoted to Tuning.

    Instead of just throwing in purging or banning when talking about VCL, there's now a chapter called Cache Invalidation, that attempts to give broader understanding of the alternatives you have and when to use which solution. Similarly, there's a chapter called Saving The Request, which starts out with the core Grace mechanisms, moves on to health checks, saint mode, req.hash_always_miss, directors and more.

    There are several reasons I write about this. First of all: I'm very excited about the material. I've worked on it regularly for several years, doing everything from hacking rst2s5, tweaking the type setting and design to updating the content, reorganizing the course and of course holding it. It may seem like one big marketing stunt, but I can promise you that I never blog about something I'm not passionate about, regardless of whether it is work-related or not.

    The other reason is that I'm holding the course next week. This will be the first time we hold it using the changed structure. I would have wanted to hold it in a class room first, but holding it on-line is still exciting.

    If you wish to participate, head over to and convince your boss it will be awesome!


    November 16, 2011

    Mikko OhtamaaVarnish, caching and HTTP cookies

    These short notes related on caching and HTTP cookies are based on my experience with Varnish and Plone CMS and WordPress.

    Sanifying cookies for caching

    Any cookie set on the server side (session cookie) or on the client-side (e.g. Google Analytics Javascript cookies) is poison for caching the anonymous visitor content.

    Common cookies for all CMS systems are usually

    • Session cookie (anonymous user session): ZopeId,  PHPSESSID
    • Logged in user cookie: __ac (Plone)
    • Active language cookie: I18N_LANGUAGE (Plone)
    • Analytics cookies (Google Analytics et. al.): various ugly cookies
    • Some other status information e.g. status message: statusmessages (Plone)

    HTTP caching needs to deal with both HTTP request and response cookie handling

    • HTTP request Cookie header. The browser sending HTTP request with Cookie header confuses Varnish cache look-up. This header can be set by Javascript also, not just by the server. Cookie can be preprocessed in vcl_recv.
    • HTTP response Set-Cookie header. This is server-side cookie set. If your server is setting cookies Varnish does not cache these responses by default. Howerver, this might be desirable behavior if e.g. multi-lingual content is served from one URL with language cookies. Set-Cookie can be post-processed in vcl_fetch.

    Example how remove all Plone related cookies besides ones dealing with the logged in users (content authors):

    sub vcl_recv {
      if (req.http.Cookie) {
          # (logged in user, status message - NO session storage or language cookie)
          set req.http.Cookie = ";" req.http.Cookie;
          set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
          set req.http.Cookie = regsuball(req.http.Cookie, ";(statusmessages|__ac)=", "; \1=");
          set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
          set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
          if (req.http.Cookie == "") {
              remove req.http.Cookie;
    # Let's not remove Set-Cookie header in VCL fetch
    sub vcl_fetch {
        # Here we could unset cookies explicitly,
        # but we assume extension does it jobs
        # and no extra cookies fall through for HTTP responses we'd like to cache
        # (like images)
        if (!obj.cacheable) {
            return (pass);
        if (obj.http.Set-Cookie) {
            return (pass);
        set obj.prefetch =  -30s;
        return (deliver);

    Another example how to purge Google Analytics cookies only and leave other cookies intact:

    sub vcl_recv {
             # Remove Google Analytics cookies - will prevent caching of anon content
             # when using GA Javascript. Also you will lose the information of
             # time spend on the site etc..
             if (req.http.cookie) {
                set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
                if (req.http.cookie ~ "^ *$") {
                    remove req.http.cookie;

     Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

    November 11, 2011

    Stefan CaunterSet up Internal Load Balancer Pools on a Barracuda

    Often we call services internally, whether it is a mail gateway or a web service. In production, if high availability is a requirement, we want to create a virtual ip for the service and add two or more physical servers as listeners. I usually use layer 7 service pools when network load is low, and […]

    September 26, 2011

    Stefan Caunterfms4.0 JSEngine out of memory? Change tag to ScriptEngine

    When you get “JavaScript runtime is out of memory; server shutting down instance” errors in your FMS logs, and your app won’t stay loaded, and none of your clients can connect, you need to increase the amount of memory available to the script engine. In the docs for FMS 4.0, the JSEngine tag is “deprecated”. […]

    September 08, 2011

    Mikko OhtamaaPurging Varnish cache from Python web application

    Varnish is a popular front end caching server. If you are using it at the front of your web site you have situations where you wish to force the cached content out of the cache E.g. site pages have been updated.

    Below is an example how to create an action to purge selected pages from the Varnish cache. The example shows how to purge the whole cache, but you can use regular expressions to limit the effect of the purge.

    This tutorial has been written for Ubuntu 8.04, Python 2.6, Varnish 2.x and Plone 4.x. However, it should be trivial to apply the instructions for other systems.

    We use Requests library which is a great improvement over Python stdlib’s infamous urllib. After you try Request you never want to use urllib or curl again.

    Configuring Varnish

    First you need to allow HTTP PURGE request handler in default.vcl from localhost. We’ll create a special PURGE command which takes URLs to be purged out of the cache in a special header:

    acl purge_hosts {
            # XXX: Add your local computer public IP here if you
            # want to test the code against the production server
            # from the development instance
    sub vcl_recv {
            # PURGE requests call purge() with a regular expression payload from
            if (req.request == "PURGE") {
                    # Check IP for access control list
                    if (!client.ip ~ purge_hosts) {
                            error 405 "Not allowed.";
                    # Purge for the current host using reg-ex from X-Purge-Regex header
                    purge(" == " " && req.url ~ " req.http.X-Purge-Regex);
                    error 200 "Purged.";

    Creating a view

    Then let’s create a Plone view which will make a request from Plone to Varnish (upstream localhost:80) and issue PURGE command. We do this using Requests Python lib.

    This is useful when very fine tuned per page purgingn fails and the site admins simply want to make sure that everything from the front end cache will be gone.

    This view is created so that it can be hooked in portal_actions menu and only the site admins have permission to call it.

    Example view code:

    import requests
    from Products.CMFCore.interfaces import ISiteRoot
    from five import grok
    # XXX: The followig monkey-patch is unneeded soon
    # as the author updated Requets library to handle this case
    from requests.models import Request
    if not "PURGE" in Request._METHODS:
        lst = list(Request._METHODS)
        Request._METHODS = tuple(lst)
    class Purge(grok.CodeView):
        Purge upstream cache from all entries.
        This is ideal to hook up for admins e.g. through portal_actions menu.
        You can access it as admin::
        # Onlyl site admins can use this
        def render(self):
            Call the parent cache using Requets Python library and issue PURGE command for all URLs.
            Pipe through the response as is.
            site_url = ""
            # We are cleaning everything, but we could limit the effect of purging here
            headers = {
                       # Match all pages
                       "X-Purge-Regex" : ".*"
            resp = requests.request("PURGE", site_url, headers=headers)
            self.request.response["Content-type"] = "text/plain"
            text = []
            text.append("HTTP " + str(resp.status_code))
            # Dump response headers as is to the Plone user,
            # so he/she can diagnose the problem
            for key, value in resp.headers.items():
                text.append(str(key) + ": " + str(value))
            # Add payload message from the server (if any)
            return "\n".join(text)

    Now if you are a Plone site admin and visit in URL:

    You’ll clear the front end cache.

    More info

     Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

    August 26, 2011

    Ingvar Hagelundrpm packages of varnish-3.0.0

    Varnish is a state of the art http accelerator, or frontside cache, if you like.

    varnish-3.0.0 was released some weeks ago. I have built packages for Fedora and epel4/5/6. Packages may be found at the usual The rhel packages require some dependencies pulled from epel.

    Varnish Software produces their own packages, based on the specfile I maintain for Fedora. The changes from their rpm spec are mostly cosmetic to fit better to Fedora’s packaging standards.

    July 21, 2011

    cd34W3 Total Cache and Varnish

    Last week I got called into a firestorm to fix a set of machines that were having problems. As Varnish was in the mix, the first thing I noticed was the hit rate was extremely low as Varnish’s VCL wasn’t really configured well for WordPress. Since WordPress uses a lot of cookies and Varnish passes […]