Planet Varnish

April 16, 2014

Per BuerThread organization in Varnish Cache 4.0

One of the two biggest changes in Varnish 4.0 is how the threads work. Varnish uses threads for doing all the heavy lifting and it seems to be working out quite well. In Varnish 3.0 one thread would service each client, doing whatever that client wanted it to do. Within reason, obviously. These are very decent threads. The thread would deliver from cache, fetch content from the backend, pipe, etc.

April 14, 2014

Yves HwangFrontend/Senior Software Developer wanted in our London or Oslo office

Exciting open source company seeks Frontend/Senior Software Developer in our London or Oslo office.

April 10, 2014

Hrafnhildur SmaradottirVarnish Cache 4.0 is released!

Varnish Cache 4.0 is out! The Varnish team is thrilled and has just celebrated with a sip of bubbly. We are happy to announce this release to our community and the world. It includes some awesome enhancements for all of you to enjoy!

Dr CarterVarnish 4.0.0 released

This is a major feature release with a lot of big changes from the last version released 2.5 years ago.

Some prominent examples include:

  • Full support for streaming objects through from the backend on a cache miss. Bytes will be sent to 1..n requesting clients as they come in from the backend server.
  • Background (re)fetch of expired objects. On a cache miss where a stale copy is available, serve the client the stale copy while fetching an updated copy from the backend in the background.
  • New varnishlog query language, allowing automatic grouping of requests when debugging ESI or a failed backend request. (among much more)
  • Comprehensive request timestamp and byte counters.
  • Security improvements, including disabling of run-time changes to security sensitive parameters.

Download at

Fuller list of changes:

<script async="async" src=""></script> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script>

April 09, 2014

Ruben RomeroVarnish 4.0 Q&amp;A on Performance, VMODs, SSL, IMS, SWR and more...

During our "What's coming in Varnish 4.0?" Hangout (see the video now) two weeks ago we got some questions. I am getting back to you with some answers.

MacYvesBuilding vagent2 for Varnish Cache 4.0.0 beta 1 for OS X 10.9.2

For those keen bunnies that wishes to jump in and help us test out varnish cache 4.0.0 Beta 1 with varnish-agent 2, here’s how you do it on OS X 10.9.2 Mavericks.


Homebrew dependencies

Install the following with Homebrew

  • automake 1.14.1
  • libtool 2.4.2
  • pkg-config 0.28
  • pcre 8.34
  • libmicrohttpd 0.9.34

Build varnish cache 4.0.0 beta 1

  1. Download and extract varnish cache 4
  2. run ./
  3. run ./configure
  4. make

Build varnish-agent2 for varnish cache 4.0.0 beta 1

  1. Clone varnish-agent from repo
  2. Checkout the varnish-4.0-experimental branch
  3. export VARNISHAPI_CFLAGS=-I/tmp/varnish/varnish-4.0.0-beta1/include
  4. export VARNISHAPI_LIBS="-L/tmp/varnish/varnish-4.0.0-beta1/lib/libvarnishapi/.libs -lvarnishapi"
  5. run ./
  6. run ./configure
  7. make

Note that if you run make install for varnish cache 4 or varnish-agent, it would then install it for you respectively.

Per BuerLogging in Varnish 4.0

The logging in Varnish Cache is one of the unique features of Varnish that in my mind sets it apart from the rest of the software world. It combines both logging in great detail with both performance, manageability, sensible privacy defaults and access to great debugging help. The shared memory log, logs everything that happens without the need to adjust log levels or without significantly affecting performance, eliminating the need for turning on any debug switch. Many of you have surely been in a situation where you have an application that is misbehaving. Then you increase the log level to debug and the problem magically disappears - obviously caused by the application slowing down just enough that the race condition you encountered earlier is now gone. The weakness of logging everything is that so much information is available that the administrator can sometimes be overwhelmed by all the information presented. It’s a figurative firehose of information and drinking from it can be painfull.  Martin has reimplemented a new logging framework in Varnish Cache 4.0. Out of all the new stuff in Varnish Cache 4.0 this might be the most significant one. It’s also the most complex feature, requiring quite a bit of time to fully understand how it works.

Per BuerLocking down Varnish 4.0 - high security installations of Varnish

Varnish Cache 4.0 is just around the corner. We did a beta release a couple of weeks back and the feedback has been pretty good and I think the release will be out later this week. Anyways, there are a lot of canges in Varnish 4.0.  A couple of the changes in Varnish Cache 4.0 are security related ones, giving you the option to lock down a Varnish installation to make it ultra secure.

April 01, 2014

Per BuerVarnish Cache 4.0 beta 1 is out

The beta is out. Varnish Cache 4.0 is just around the corner and we need a bit of help to cross the finishing line. The beta seems to be pretty solid, we've given it a fair beating in out testing facilities as well as some pretty rough production workload. The final production tests last week where done in cooperation with A-Media and Redpill Linpro and where quite encouraging. Varnish 4.0 ran stable on their workload happily chugging away 1300 requests a second. If you have the option of giving Varnish Cache 4.0 a spin on your website, or giving it parts of your traffic we would be grateful. Crashes are very unlikely at this point, but just to be safe, we would recommend having a load balancer in front of the beta so you can gradually increase the load. If you need any help with converting the VCL, please let us know.

March 28, 2014

Per BuerWhen communities feel betrayed

This week the Internet has been all up in arms about Face acquiring Oculus Rift. Many people have been angry about Facebook buying this company, something we did not see when Facebook acquired WhatsApp or Instagram.

March 25, 2014

Yves HwangVCL change management and contiguous integration with your Varnish using VAC API and Git

Managing multiple Varnish instances and their respective VCL is made significantly easier with Varnish Administration Console (VAC) through its API. This blog post aims to illustrate an example of VCL change management and continuous integration with multiple Varnish instances using VAC API and a little magic from Git.

March 24, 2014

Ruben RomeroJoin our Varnish 4.0 Hangout on Wednesday

Two days to go for our Varnish 4.0 Hangout with the Varnish Core Developer team. Make sure you join us. RSVP now!

March 19, 2014

Yves HwangVAC 3

Software releases are not won by any single effort, but they are by processes, persistence, quality, and commitment. Most of all, solid teamwork. A release constitutes a bundling of various Git commits, neatly cherry picked to form a release candidate. From code changes to test automation, to build pipelines, the piece of software is scrutinised closely at every step.

March 17, 2014

Ruben RomeroVarnish Core Devs tell you what is coming in Varnish 4.0 (w/Q&amp;A)

Hangout On Air with PHK, Per, Martin & Lasse. We are now eagerly preparing for the release of Varnish 4.0. The release is planned for Q2 and to give you a sneak preview of what is coming we have scheduled a Google Hangout-on-Air session with Varnish’s Chief Architect, Poul-Henning Kamp, Varnish Software’s Founder and CTO, Per Buer, as well as our key Varnish Developers, Martin Blix-Grydeland and Lasse Karstensen.

March 13, 2014

Yves HwangReal-time statistics API with VAC

Varnish Administration Console (VAC) is broken up into two major components. Its Management UI, and a RESTful API, with the UI utilising the API. There are a diverse number of features supported by the API, for example, change management, statistics and etc. This blog post aims to showcase the Real-time Statistics API.

March 07, 2014

Hrafnhildur SmaradottirUsing Varnish Tools to visualize your most popular content

We just love it when our customers use our products in innovative ways that really help them create business value. One of the most recent examples of such use is how our customer (Norway’s most popular internet news provider) is using Varnish Tools in combination with D3 to visualize their most popular online content.

February 21, 2014

ZenikaCreating a Varnish 4 module

Don't let the title fool you, Varnish 4 hasn't been released yet. However, the master branch (which holds Varnish 4 developments) in its current state should be close enough to what the first release will look like. I've been bitten by API changes several times, but that's the deal when writing code against work in progress[1].


[1] Last tested against the revision d0c0ee9

What's new in Varnish 4 ? I will simply copy/paste an answer from Varnish Software (blame my laziness on this one): The 4.0 release is underway and should be released after the summer, with a prerelease during the summer, hopefully. The main new features are: New logging framework (see this blog post... Read Creating a Varnish 4 module

February 11, 2014

Yves HwangAdding Varnish Cache related statistics to your APM

This blog post aims to introduce AppDynamics Varnish Extension for gathering cache related statistics. At the core of AppDynamics Varnish Extensions is varnishstat data that are fed back to AppDynamics Pro via varnish-agent in order to provide key statistics for a complete performance overview of your web stack.

February 04, 2014

Yves HwangVarnish Hashninja explained

Recently we have had a few questions about the Varnish Hashninja VMOD, and how this all ties into Varnish Plus. This blog post aims to demystify the ever-so-elusive Hashninja, and help the readers find the best fit for their cache invalidation solution.

January 16, 2014

Per BuerIntroducing Varnish Plus

This month we’re launching Varnish Plus. The rationale behind Varnish Plus is to combine all our services and products in one offering, market it under one name. So, when you become our customer, you get access to everything you need in one place.

January 13, 2014

Per BuerAre you a C programmer in Oslo?

We're looking for a C-programmer in Oslo to work on Varnish and related stuff.

Hrafnhildur SmaradottirVarnish’s role in the tech platform redesign

The recently did a redesign of their website. Or rather, they did a complete reinvention of the website’s entire digital platform. In a blog post on their Open blog, they introduce the technology they used in the redesign. The objective with the redesign was to have a faster running code base and to make future changes easier.

December 19, 2013

Lasse KarstensenConverting a Varnish 3.0 VMOD to 4.0

So we’re getting closer to releasing the first proper 4.0 version of Varnish Cache. One of the things we need to fix is to get all the vmod writers to make sure their vmod works with the new version.

Here are my notes from doing just that, in the hope to make it simpler for others.

In 4.0, you don’t need the source tree of Varnish any more. The include files will be enough, and pkg-config will find them for you.

Make sure that /usr/lib/pkgconfig/varnishapi.pc and /usr/share/aclocal/varnish.m4 exists. If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source. (I need multiple varnishd versions when debugging customer problems, so I let them live in /opt/varnishX.Y/ on my laptop)

Pull/merge the new files from the master branch of libvmod-example.

Header files: remove bin/varnishd/cache.h and add cache/cache.h.

Vmod functions are now called with a vrt context as first argument. %s/struct sess \*sp/const struct vrt_ctx \*ctx/g

The old sess struct has been split, some data is in vrt_ctx->req, and some is in vrt_vtx->req->sp. Look up what is where in cache/cache.h. 

I’ve put up the 3.0->4.0 diff for vmod_policy.c as a gist:

There was a bit trouble of finding varnishtest, as src/Makefile was missing the reference entirely. I just fixed it by hand for now. Another thing for the 4.0 todolist, then.

And finally; 

lkarsten@immer:~/work/libvmod-policy/src$ make check
/opt/varnish/bin/varnishtest -Dvarnishd=/opt/varnish/sbin/varnishd -Dvmod_topbuild=/home/lkarsten/work/libvmod-policy tests/test01.vtc
# top TEST tests/test01.vtc passed (1.574)


I have a working Varnish 4.0 vmod. :-D

December 11, 2013

Lasse KarstensenDNS RBL test address for development

If you are writing code that checks a DNS real-time blockhole list (RBL), it looks like is the standard address that is always in the black/white -list.

This is probably know for most sysadmins/security people and whatnot, but wasn’t entirely trivial to find using Google.

lkarsten@immer:~$ dig @
; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55083
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
; IN A
;; ANSWER SECTION: 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A 2562 IN A
;; Query time: 17 msec
;; WHEN: Wed Dec 11 14:12:20 2013
;; MSG SIZE rcvd: 203

Good to be able to actually test your code for hits also.

(this is for libvmod-policy, so you can deny/reject POST/PUT from spammers in Varnish)

November 29, 2013

Per Buer404 handling in Varnish Cache

When you have lots and lots of data, it’s hard to update it in a timely manner. Sometimes all you need is a bit of time while the systems in place are updating. A typical situation is the one where you have two backends serving the same data. However, one of them might lag behind the other one. When a request comes in to the one that is lagging it will throw a 404 Not Found. The solution is simple to describe; when you get a 404 try the other server.

November 21, 2013

Hrafnhildur SmaradottirVarnish Software wins Red Herring 100 Global Award

Last night, at an award ceremony in LA, California, we were awarded the Red Herring 100 Global award, a prize that recognizes Varnish Software as one of the top tech companies in the world.

November 14, 2013

Hrafnhildur SmaradottirHow does the Varnish Paywall really work?

Our paid-content solution, the Varnish Paywall, has provided real value to our customers. Some of them have seen a fast increase in number of subscribers. Others have with the help of the Varnish Paywall implemented more effective gateways to their online content and seen an increase in traffic.

November 05, 2013

Per BuerUsing pipe in Varnish

Quite often we see people using return(pipe) in VCL and then running in to various problems. Let me give you some information on what return(pipe) actually does and when it makes sense to use it and when it doesn’t.

October 28, 2013

Per BuerLive AB testing with Varnish and VCS

In this post we take a look at how to do AB-testing with Varnish and VCS. How to split users into groups, how to record success conversions and how get a live feed of data from the experiement.

October 25, 2013

Hrafnhildur SmaradottirVarnish Software is a Finalist for the 2013 Red Herring 100 Global Award

Varnish Software has been selected as a candidate for Red Herring's 2013 Top 100 Global award. The award honors this year’s most audacious and far reaching private technology companies and entrepreneurs from across the globe.

October 24, 2013

Yves HwangWeb traffic analysis with Varnish and VCS

This blog post aims to illustrate how web traffic statistics can be easily gathered in real-time and without involvement of any 3rd party vendors or the use of cookies. Once these statistical metrics are defined, conducting A/B testing through Varnish is also outlined.

October 22, 2013

Yves HwangVCS 1.3.0 is out!

Varnish Custom Statistic 1.3.0 has been released. It is available for RHEL6, Precise, Squeeze and Wheezy (amd64) today.

October 14, 2013

Hrafnhildur SmaradottirVarnish receives the Global Hot 100 Award

Varnish Software was honored to receive the Global Hot 100 Award at The World Summit on Innovation & Entrepreneurship (THEWSIE) in New York last Friday.

October 08, 2013

Hrafnhildur SmaradottirHow Newsday and News12 bumped up both traffic and speed with Varnish

When Dennis Elder, Newday's vice president for Digital Development, contacted us last year the news sites he manages, including Newsday and News12, were not performing optimally when it came to capacity and speed. Dennis wanted to use Varnish to solve those issues and he also had hopes that Varnish would be able to help accelerate the paid-content infrastructure on his sites.

September 25, 2013

Lasse KarstensenVarnish and Ghost blogging software

So there is a new shiny blogging platform out called Ghost. Looks pretty good to me.

If you want to run it behind Varnish, you’ll soon notice it has the usual problem of setting session cookies everywhere leading to 0% hit rate. 

I have written a Varnish VCL configuration for filtering this in the necessary places, while keeping the admin interface working still.

You can find it here:

Have fun.

September 24, 2013

Hrafnhildur SmaradottirOur October webinars

We have scheduled two webinars for October. They are both on the topic of caching. Surprise, surprise!

September 19, 2013

Hrafnhildur SmaradottirInfoWorld names Varnish Cache Best Open Source Software 2013

InfoWorld recognizes the best open source software for business each year by handing out the Bossies (Best of Open Source Software awards) to a handpicked selection of open source applications.

September 18, 2013

Hrafnhildur SmaradottirThe value of using Varnish as a part of your e-commerce infrastructure, part 2

In my previous post on this topic I told you about our e-commerce customer TradeMe and how they have used Varnish to boost the user experience on their site to help them outperform their competitors. In this article I will explain how you can use the Varnish Hashninja to further enhance your web performance by making your e-commerce website more efficient than you thought possible.

September 13, 2013

Yves HwangSimple scales better and faster in the real world

The Super Fast Purger was conceived as a plugin for the Varnish Administration Console. The current implementation is capable of receiving and distributing 45.000 cache invalidation requests per second with ease. We believe that the upper limit of the throughput for the Super Fast Purger is yet to be determined as the software implementation was limited to network and hardware capabilities during benchmarking.

September 10, 2013

Lasse KarstensenTesting VMODs with Travis (

Travis CI is a service where open source software can run tests automatically on commits. It hooks into github in a silky smooth way.

If you’re developing a Varnish module (VMOD), you probably have started out with our libvmod-example package. It has all the automake magic you need, as well as some simple test cases to test your vmod. Given that you’ve written som varnishtest testcases for it (you really should), you now can get travis to run them as well!

I’ve put a small recipe for this into the libvmod-example package.

Feel free to play around with it, feedback appreciated. For the travis setup bits, consult the travis getting started guide. The final result is something like this, shown for libvmod-cookie:

September 09, 2013

MacYvesVagrant, Varnish and vmods

Development environment has been plaguing us for a while in my product development department. From dependencies hell to complex setup in operations, our development environment has gone through the usual gauntlet of pains and complaints.

This has changed with Vagrant. It is the single tool that gels the devs with the ops; quintessential devop tool if you will. Not only Vagrant has helped eliminate the “works on my machine” bugs, we use it for automated integration tests. In addition, this one tool has made our development environment setup quick and simple for our HCI guys too.

We do a lot of integration work with Varnish Cache and I thought I would take this opportunity to share this simple Vagrantfile, as an example, to help get started with installing varnish and libdigest-vmod from source.

Note that the provisioning process is rather crude in this example. Rather, the intention here is to out outline the steps required to get varnish and vmods installed and running via Vagrant. For production and future maintainability, do use Chef or Puppet as it can be seamlessly integrated within the Vagrantfile.

# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.define :varnish do |varnish| = "varnish"
    varnish.vm.box_url = ""
    $script_varnish = <<SCRIPT
echo Installing dependencies, curl
sudo apt-get update
sudo apt-get install curl -y
sudo apt-get install git -y
curl | sudo apt-key add -
echo "deb precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
echo "deb-src precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
echo ==== Compiling and installing Varnish from source ====
sudo apt-get install build-essential -y
sudo apt-get build-dep varnish -y
apt-get source varnish
cd varnish-3.0.4
sudo make install
cd ..
echo done
echo ==== Compiling and installing lib-digest vmod from source ===
git clone
sudo apt-get install libmhash-dev libmhash2 -y
cd libvmod-digest
./configure VARNISHSRC=/home/vagrant/varnish-3.0.4 VMODDIR=/usr/local/lib/varnish/vmods
sudo make install
cd ..
echo ===== done ====
echo ===== firing up varnish via commandline ====
sudo varnishd -a :80 -T :6081 -f /vagrant/test.vcl
touch varnish_vm

    varnish.vm.provision :shell, :inline => $script_varnish


September 05, 2013

Hrafnhildur SmaradottirVarnish Software nominated as one of the World’s Most-Innovative Emerging Companies

We are thrilled to be able to share with you that we have been named a Global Hot 100 Company by The World Summit on Innovation & Entrepreneurship (THEWSIE).

August 01, 2013

ZenikaCreating a Varnish module

Warning: Parameter 1 to rsExtPost::getExcerpt() expected to be a reference, value given in /home/zenikaco/public_html/inc/clearbricks/dblayer/dblayer.php on line 618

Warning: Parameter 1 to rsExtPost::getContent() expected to be a reference, value given in /home/zenikaco/public_html/inc/clearbricks/dblayer/dblayer.php on line 618

July 29, 2013

Lasse KarstensenBuilding a Varnish VMOD on Debian

From the tutorials department, here are some quick notes on how to install a Varnish VMOD from source.

This is slightly complicated because Varnish demands that a VMOD must be built against the same git commit (or release) as the one that is running. This will be relaxed in future versions.

Current setup is a standalone Varnish VM on Debian Wheezy with Varnish installed from varnish package archives (3.0.4-1~wheezy.)

1. Get the vmod

lkarsten@lb1:~$ git clone
Cloning into 'libvmod-cookie'...
remote: Counting objects: 253, done.
remote: Compressing objects: 100% (131/131), done.
remote: Total 253 (delta 132), reused 232 (delta 112)
Receiving objects: 100% (253/253), 49.51 KiB, done.
Resolving deltas: 100% (132/132), done.

2. Get and configure the source tree for the running Varnish

Verify first that you have the necessary package repositories enabled:

lkarsten@lb1:~$ grep varnish /etc/apt/sources.list
deb wheezy varnish-3.0
deb-src wheezy varnish-3.0

After that, continue with the juicy parts:

lkarsten@lb1:~$ apt-get source varnish 
Reading package lists... Done 
Building dependency tree 
Reading state information... Done 
NOTICE: 'varnish' packaging is maintained in the 'Git' version control system at: 
Need to get 2,060 kB of source archives. 
Get:1 wheezy/varnish-3.0 varnish 3.0.4-1 (dsc) [2,334 B] 
Get:2 wheezy/varnish-3.0 varnish 3.0.4-1 (tar) [2,044 kB] 
Get:3 wheezy/varnish-3.0 varnish 3.0.4-1 (diff) [14.1 kB] 
Fetched 2,060 kB in 0s (11.4 MB/s) 
gpgv: keyblock resource `/home/lkarsten/.gnupg/trustedkeys.gpg': file open error 
gpgv: Signature made Fri 14 Jun 2013 11:56:48 CEST using RSA key ID 87218D9C 
gpgv: Can't check signature: public key not found 
dpkg-source: warning: failed to verify signature on ./varnish_3.0.4-1.dsc 
dpkg-source: info: extracting varnish in varnish-3.0.4 
dpkg-source: info: unpacking varnish_3.0.4.orig.tar.gz 
dpkg-source: info: applying varnish_3.0.4-1.diff.gz 
lkarsten@lb1:~$ cd varnish-3.0.4
lkarsten@lb1:~/varnish-3.0.4$ ./
lkarsten@lb1:~/varnish-3.0.4$ ./configure --prefix=/usr
lkarsten@lb1:~/varnish-3.0.4$ make

If configure or make fails, you might need some additional packages. Run an apt-get build-dep varnish and work from there. (if editline fails on you, remember to rerun configure after installing it)

3. Build and install the vmod

lkarsten@lb1:~$ cd libvmod-cookie/
lkarsten@lb1:~/libvmod-cookie$ ./
+ aclocal -I m4
+ libtoolize --copy --force
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
+ autoheader
+ automake --add-missing --copy --foreign installing `./config.guess' installing `./config.sub' installing `./install-sh' installing `./missing'
src/ installing `./depcomp'
+ autoconf
lkarsten@lb1:~/libvmod-cookie$ ./configure VARNISHSRC=~/varnish-3.0.4/
# and finally
lkarsten@lb1:~/libvmod-cookie$ make
libtool: link: ( cd ".libs" && rm -f "" && ln -s "../" "" )
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie/src'
make[2]: Entering directory `/home/lkarsten/libvmod-cookie'
rst2man README.rst vmod_cookie.3
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'
lkarsten@lb1:~/libvmod-cookie$ sudo make install
/bin/mkdir -p '/usr/local/share/man/man3'
 /usr/bin/install -c -m 644 vmod_cookie.3 '/usr/local/share/man/man3'
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'

At this point you should have the two vmod files available for Varnish:

lkarsten@lb1:~/libvmod-cookie$ ls -l /usr/lib/varnish/vmods/
total 64
-rwxr-xr-x 1 root root 966 Jul 29 11:11
-rwxr-xr-x 1 root root 41538 Jul 29 11:11
-rw-r--r-- 1 root root 16128 Jun 17 13:38

And you are done!

“import cookie” should now work without issue in your /etc/varnish/default.vcl.

July 23, 2013

Per BuerThe essential vmods all Varnish users should know about

Approximately two years ago we released Varnish Cache 3.0. In my view the most important feature was the support for modules. Thanks to the hello world vmod, which almost all of the modules out there are based on, it’s rather easy to write a trivial module. It is obviously more complicated to write more complex modules and since there is no sandboxing involved we see modules wrecking havoc in varnishd regularly. Memory leaks and segfaults are found in modules from time to time. However, there are modules that can add really wonderful functionality without destabilizing your varnishd. In my opinion these are some of the modules all Varnish users should know about.

July 22, 2013

Lasse KarstensenSetting client.ip in Varnish VCL with libvmod-ipcast

I’ve written a new Varnish 3.0 VMOD called ipcast.

It has a single function; ipcast.clientip(ipstring) which sets the internal Varnish variable client.ip to whatever IPv4/IPv6 address you give as the argument.

You need this if you want to do ACL checks on connections done through a load balancer or SSL terminator. In those cases client.ip would be and you get the real client’s IP address in the X-Forwarded-For (or similar) header.

You can find it here:

Here is some example VCL to illustrate how it works. I think the regex needs some work, suggestions/pull requests are welcome.

import ipcast;
acl friendly_network {
sub vcl_recv {
    if (req.http.X-Forwarded-For !~ ",") {
        set req.http.xff = req.http.X-Forwarded-For;
    } else {
        set req.http.xff = regsub(req.http.X-Forwarded-For,
                "^[^,]+.?.?(.*)$", "\1");

    if (ipcast.clientip(req.http.xff) != 0) {
        error 400 "Bad request";

    if (client.ip !~ friendly_network) {
            error 403 "Forbidden";

July 18, 2013

cd34Varnish and Node.js

While working with a client installation they wanted to run Varnish in front of their node.js powered site to eliminate having node serve the static assets. uses HTTP/1.0 and cannot be cached. Minimally these few lines can be added to their respective functions and things will work. Obviously you’ll want to set expires on […]

July 15, 2013

Lasse Karstensenlibvmod-cookie; simpler cookie handling in Varnish VCL

I’ve written a Varnish module (VMOD) to simplify handling of HTTP cookies in Varnish VCL.

In essence it does the usual stuff you’d usually do with regular expressions, but with much simpler syntax and readability.

It works by parsing the req.http.Cookie into a small data structure. Afterwards you can add/delete/filter the list, and finally ask for a combined string back again.

Here is the list of functions:

  • cookie.parse(cookiestring) – parse the http.req.Cookie string.
  • cookie.get(cookiename) – get the value of a single parsed cookie.
  • cookie.set(cookiename, cookievalue) – set the value of a cookie.
  • cookie.delete(cookiename) – delete a single cookie.
  • cookie.filter_except(filterstring). Delete all cookies but the ones matching the names in the comma-separated filterstring (“cookie1,cookie2,cookie3″.) This should be quite useful.
  • cookie.get_string() – return the parsed string.

You can find it here:

June 30, 2013

ops42Properly redirect to mobile pages

It is just amazing how much advice and examples one can find for how to redirect to a mobile equivalent of a given HTTP address. Oversimplified, wrong and harmful advice that is. And no, I’m not talking about that 301 vs 302 bullshit. For the love of God, stop listening to those overpaid, know-nothing SEO […]

June 17, 2013

Dr CarterVarnish 3.0.4 released

Summary of changes from 3.0.3 to 3.0.4

  • The ACL code had a bug which could lead to false negatives. This has been assigned CVE-2013-4090.
  • Varnish will now return an error if the client sends multiple Host headers.
  • If the backend sent invalid gzip while using ESI, Varnish would in some cases assert. It now works correctly.
  • TCP_NODELAY is now enabled, which should lead to performance improvements in some cases.

Download here.

Fuller list of changes:

June 13, 2013

Per BuerSpeculative Lock Elision in Varnish Cache

This year we’ve seen quite an interesting announcement from Intel. Their Haswell processors are now on sale. Usually we haven’t cared much about what Intel releases as new features in CPUs don’t always impact the performance on server software that much. Haswell, on the other hand, now has a feature, Transactional memory, that is pretty cool and might actually influence Varnish's performance quite a bit. Here is why.

Per BuerAdvanced cache invalidation strategies

“There are only two hard things in Computer Science: cache invalidation and naming things.” -- Phil Karlton Today, I’ll write about the hardest one of these two; cache invalidation. This used to be rather straightforward. Most websites had a rather simple structure between their content repository and their URL structure. This made it rather simple to purge URLs when the content changed.

May 27, 2013

Per BuerQ&amp;A from the &quot;You can cache everything&quot; webinar

We lost the audio on the Q&A session of the "You can cache everything" webinar. Here are the questions we got along with their answers.

May 15, 2013

Per BuerGetting live statistics from Varnish with HLS/HDS

The latter years we’ve seen a surge in the use of HTTP based transports for online video. HTTP Live Streaming (HLS), HTTP Dynamic Streaming (HDS) and Silverlight video (Smooth streaming) are seeing quite a bit of usage. These streaming standards are more or less made for caching. However, due to the stateless nature of the HTTP protocol there is a gap between the streams themselves and the underlying traffic. This makes it hard to get proper insight into what is actually going on in the video distribution system.

April 23, 2013

Ruben RomeroFlying without turbines: Compiler-less Varnish Cache

Varnish Cache uses the gcc compiler by default. If you need it to run without the compiler, for whatever reason, we can provide you with a version of Varnish that does not have that requirement.

April 15, 2013

Ruben RomeroVarnish User Group Meeting 7 in NYC #VUG7

We are gearing up for the 7th Varnish User Group Meeting on May 30th-31st 2013 in New York City, USA co-sponsored by Vimeo, 10gen and Varnish Software. The Varnish Administration Course will be held on June 3rd & 4th. Make sure you get a seat by booking it now.

March 25, 2013

Mikko OhtamaaVarnish at the front of WordPress + Apache and Plone CMS virtual hosts

When moving some sites to a new server I upgraded the Varnish cache server configuration serving this setup. Here are my notes how one can use Varnish at the front of virtual hosting.

The setup is following

  • Multiple sites are hosted on the same server. The sites are mix of PHP of Plone sites.
  • Varnish accepts HTTP requests in port 80 and forwards request to the corresponding backend through HTTP proxying
  • Our virtual host rules capture domain names with or without www-prefix, or with any subdomain name prefix
  • Apache runs in non-standard port localhost:81, serving PHP and WordPress. WordPress caching rules are defined in Apache <virtualhost> config file.
  • Every Plone site runs in its own port and process. Plone uses VirtualHostMonster to rewrite publicly facing site URLS. Plone caching HTTP headers are set by addon.
  • We do extensive cookie sanitization for PHP (generic), WordPress and Plone.  Google Analytics etc. cookies don’t bust the cache and we can still login to WordPress and Plone as admin
  • As a special trick, there is cleaned cookie debugging trick through HTTP response headers


Don’t worry, Varnish can handle little load



  • Blazingly fast, as Varnish is
  • With Plone’s, one does not need to touch configuration files but Plone caching HTTP headers can be configured through-the-web


  • Varnish does not have Nginx or Apache style virtual host configuration file facilities by default and making includes is little bit tricky: With many virtualhost the default.vcl config file grows long.
  • Because WordPress cannot do static resource serving as smartly as Plone, which has unique URLs for all static media revisions, you need to purge Varnish manually from command line if you update any static media files like CSS, JS or images.

Varnish /etc/varnish/default.vcl example for Varnish 3.0:

# This backend never responds... we get hit in the case of bad virtualhost name
backend default {
    .host = "";
    .port = "55555";

backend myplonesite {
    .host = "";
    .port = "6699";

# Apache running on server port 81
backend apache {
    .host = "";
    .port = "81";

# Gues which site / virtualhost we are diving into.
# Apache, Nginx or Plone directly
sub choose_backend {

    # WordPress site
    if ( ~ "^(.*\.)?opensourcehacker\.com(:[0-9]+)?$") {
        set req.backend = apache;

    # Example Plone site
    if ( ~ "^(.*\.)?myplonesite\.fi(:[0-9]+)?$") {
        set req.backend = myplonesite;

        # Zope VirtualHostMonster
        set req.url = "/VirtualHostBase/http/" + + ":80/Plone/VirtualHostRoot" + req.url;



sub vcl_recv {

    # Do Plone cookie sanitization, so cookies do not destroy cacheable anonymous pages.
    # Also, make sure we do not destroy WordPress admin and login cookies in the proces
    if (req.http.Cookie && !(req.url ~ "wp-(login|admin)")) {
        set req.http.Cookie = ";" + req.http.Cookie;
        set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
        set req.http.Cookie = regsuball(req.http.Cookie, ";(statusmessages|__ac|_ZopeId|__cp|php|PHP|wordpress_(.*))=", "; \1=");
        set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

        if (req.http.Cookie == "") {
            remove req.http.Cookie;

    call choose_backend;

    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    return (lookup);

sub vcl_fetch {

    /* Use to see what cookies go through our filtering code to the server */
    /* set beresp.http.X-Varnish-Cookie-Debug = "Cleaned request cookie: " + req.http.Cookie; */

    if (beresp.ttl <= 0s ||
        beresp.http.Set-Cookie ||
        beresp.http.Vary == "*") {
         * Mark as "Hit-For-Pass" for the next 2 minutes
        set beresp.ttl = 120 s;
        return (hit_for_pass);
    return (deliver);

# Show custom helpful 500 page when the upstream does not respond
sub vcl_error {
  // Let's deliver a friendlier error page.
  // You can customize this as you wish.
  set obj.http.Content-Type = "text/html; charset=utf-8";
  synthetic {"
  <?xml version="1.0" encoding="utf-8"?>
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
      <title>"} + obj.status + " " + obj.response + {"</title>
      <style type="text/css">
      #page {width: 400px; padding: 10px; margin: 20px auto; border: 1px solid black; background-color: #FFF;}
      p {margin-left:20px;}
      body {background-color: #DDD; margin: auto;}
    <div id="page">
    <h1>Sivu ei ole saatavissa</h1>
    <p>Pahoittelemme, mutta palvelua ei ole saatavilla.</p>
    <hr />
    <h4>Debug Info:</h4>
    <pre>Status: "} + obj.status + {"
Response: "} + obj.response + {"
XID: "} + req.xid + {"</pre>

WordPress does not support setting HTTP response headers natively like Plone. We set them in Apache virtual host configuration file in /etc/apache2/sites-enabled:



    LogFormat       combined
    TransferLog     /var/log/apache2/

    # Basic WordPress setup

    Options +Indexes FollowSymLinks +ExecCGI

    DocumentRoot /srv/php/opensourcehacker/wordpress

    <Directory /srv/php/opensourcehacker/wordpress>
        Options FollowSymlinks
        AllowOverride All

    AddType text/css .css
    AddType application/x-httpd-php .php .php3 .php4 .php5
    AddType application/x-httpd-php-source .phps

    # Set expires headers manually
    ExpiresActive On
    ExpiresByType text/html A0
    ExpiresByType image/gif A3600
    ExpiresByType image/png A3600
    ExpiresByType image/image/ A3600
    ExpiresByType image/jpeg A3600
    ExpiresByType text/css A3600
    ExpiresByType text/javascript A3600
    ExpiresByType application/x-javascript A3600


Buy open source friendly bitcoins  Subscribe to this blog in a reader Follow me on Twitter Follow me on Facebook Follow me Google+

March 20, 2013

Lasse KarstensenVarnish paywall presentation

I held a technical presentation for Redpill Linpro about running a paywall/metered access with Varnish yesterday.

Slides as PDF are here: RL-paywall-2013-03

Main talking points were how it runs within Varnish, what extra vmods are in use, and finally how it is integrated with your backend/cms.

To keep the VS marketing department happy (Hi!), here is the link to our product page: :-)



February 27, 2013

Lasse KarstensenVarnish development news (Feb 2013)

An update from git:

  • Tab completion in varnishadm. (e6a88da)
  • TCP_KEEPALIVE is now enabled for all connections on operating systems that supports it. This avoids a situation where Varnish would keep retrying failing slow backend request for clients that had given up and closed the connection. (f91fc4d)
  • [3.0] The TCP_NODELAY patch introduced in September was backported into 3.0. This disables Nagle’s algorithm on connections, and avoids a situation where the last chunk of a chunked response from ESI/internal gzip got buffered by the server’s IP stack before being sent to the client. (ef5e9e0)
  • Default VCL now takes Surrogate-Control: no-store and Cache-Control: no-cache/no-store/private into account when computing TTL on a backend response. (81006ea)
  • Per took a stab at the documentation, rework underway to fit Scoof’s proposed new model.

We (Varnish Software) are looking for a junior frontend developer in Oslo, Norway. Skiing skills optional. The web page might not be updated yet, but please apply here:

February 15, 2013

Kristian LyngstølThe Architecture the Varnish Agent

Posted on 2013-02-15

Designing software architecture is fun.

The Varnish Agent 2 was written as a replacement for the original Varnish Agent. They both share the same purpose: Expose node-specific Varnish features to a management system. They are design very differently, though.

In this post I'd like to explain some choices that were made, and show you how to write your own code for the Varnish Agent 2. It's really not that hard.

The code can be found at:

Why C ?

The choice of C as a language was made fairly early. One of the main reasons is that Varnish itself is written in C, as are all the tools for Varnish. This means that the by far best supported APIs for talking to Varnish are written in C.

But an other reason is because C is a very good language. It has become a false truth that you "never write web apps in C", more or less. There are good reasons for this: It takes time to set things up in C, C isn't very forgiving and perhaps most importantly: people generally suck at C.

In the end, we chose C because it was the right tool for the job.


When designing a new system, it's important to know what you're trying to achieve, and perhaps just as important to know what you're /not/ trying to achieve.

The Varnish Agent is designed to:

  • Manage a single Varnish server.
  • Remove the need for management frontends to know the Varnish CLI language.
  • Expose log data
  • Persist configuration changes
  • Require "0" configuration of the agent itself
  • Ensure that Varnish works on boot, even if there is no management front-end present.
  • Be expandable without major re-factoring.
  • Be easy to expand

What we did NOT want was:

  • Support for running the agent on a different machine than the Varnish server.
  • Elaborate self-management of the agent (e.g: support for users, and management of them).
  • Mechanisms that are opaque to a system administrator
  • Front-end code mixed with back-end code
  • "Sessions"

We've achieved pretty much all of these goals.

The heart of the agent: The module

At the heart of the agent, there is the module. As of this writing, there are 14 modules written. The average module is 211 lines of C code (including copyright and license). The smallest module, the echo module, is 92 lines of code (the echo plugin is an example plugin with extensive self documentation). The largest modules, the vlog and vcl modules, are both 387 lines of code.

To make modules useful, I spent most of the initial work on carving out how modules should work. This is currently how it works:

  • You define a module, say, src/modules/foobar.c
  • You write foobar_init(). This function is the only absolutely required part of the function. It will be run in the single-threaded stage of the agent.
  • You either hook into other modules (like the httpd-module), or define a start function.
  • After all plugins are initialized, the start function of each plugin is executed, if present.

That's it.

Since a common task is inter-operation between plugins, an IPC mechanism was needed. I threw together a simple message passing mechanism, inspired by varnish. This lives in src/ipc.c and include/ipc.h. The only other way to currently talk to other modules is through httpd_register (and logger(), but that's just a macro for ipc_run()).

If you want your foobar.c-plugin to talk to the varnish CLI, you want to go through the vadmin-plugin. This is a two-step process:

int handle;

void foobar_init(struct agent_core_t *core)
    handle = ipc_register(core, "vadmin");

This part of the code gives you a socket to talk to the vadmin module. Actually talking to other modules in foobar_init() is not going to work, since the module isn't started yet.

And proper etiquette is not to use a global variable, but to use the plugin structure for your plugin, present in core:

struct foobar_priv_t {
        int vadmin;
void foobar_init(struct agent_core_t *core)
        struct foobar_priv_t *priv = malloc(sizeof(struct echo_priv_t));
        struct agent_plugin_t *plug;
        plug = plugin_find(core,"foobar");
        priv->vadmin = ipc_register(core,"vadmin");
        plug->data = (void *)priv;
        plug->start = NULL;

In this example, we have a private data structure for the module, which we allocate in the init function. Every function has a generic struct agent_plugin_t data structure already allocated for it and hooked on to the core->plugins list. This allows you to store generic data, as the core-data structure is the one typically passed around.


The varnish agent uses a lot of assert()s. This is similar to what Varnish does. It lets you, the developer, state that we assume this worked, but if it didn't you really shouldn't just continue. It's excellent for catching obscure bugs before they actually become obscure. And it's excellent for letting you know where you actually need proper error code.

Let's take a closer look at the generic struct agent_plugin_t:

struct agent_plugin_t {
        const char *name;
        void *data;
        struct ipc_t *ipc;
        struct agent_plugin_t *next;
        pthread_t *(*start)(struct
                            agent_core_t *core, const
                            char *name);
        pthread_t *thread;

The name should be obvious. The void *data is left for the plugin to define. It can be ignored if your plugin doesn't need any data at all (what does it do?).

struct ipc_t *ipc is the IPC-structure for the plugin. This tells you that all plugins have an IPC present. This is to allow you to run ipc_register() before a plugin has initialized itself. Otherwise we'd have to worry a lot more about which order modules were loaded.

Next is *next. This is simply because the plugins are par of a linked list.

the start() function-pointer is used to define a function that will start your plugin. This function can do pretty much anything, but have to return fairly fast. If it spawns off a thread, it's expected that it will return the pthread_t * data structure, as the agent will later wait for it to join. Similar, *thread is used for the same purpose.

Using the IPC

You've got a handle to work with, let's use it. To do that, let's look at the vping plugin, starting with init and start:

static pthread_t *
vping_start(struct agent_core_t *core, const char *name)
        pthread_t *thread = malloc(sizeof (pthread_t));
        return thread;

vping_init(struct agent_core_t *core)
        struct agent_plugin_t *plug;
        struct vping_priv_t *priv = malloc(sizeof(struct vping_priv_t));
        plug = plugin_find(core,"vping");

        priv->vadmin_sock = ipc_register(core,"vadmin");
        priv->logger = ipc_register(core,"logger");
        plug->data = (void *)priv;
        plug->start = vping_start;

vping_init() grabs a handle for the vadmin (varnish admin interface) plugin, and the logger. It also assigns vping_start() to relevant pointer.

vping_start() simply spawns a thread that runs vping_run.

static void *vping_run(void *data)
        struct agent_core_t *core = (struct agent_core_t *)data;
        struct agent_plugin_t *plug;
        struct vping_priv_t *ping;
        struct ipc_ret_t vret;

        plug = plugin_find(core,"vping");
        ping = (struct vping_priv_t *) plug->data;

        logger(ping->logger, "Health check starting at 30 second intervals");
        while (1) {
                ipc_run(ping->vadmin_sock, &vret, "ping");
                if (vret.status != 200)
                        logger(ping->logger, "Ping failed. %d ", vret.status);

                ipc_run(ping->vadmin_sock, &vret, "status");
                if (vret.status != 200 || strcmp(vret.answer,"Child in state running"))
                        logger(ping->logger, "%d %s", vret.status, vret.answer);
        return NULL;

The vping module was the first module written. Written before the varnish admin interface was a module. It simply pings Varnish over the admin interface.

This also illustrates how to use the logger: Grab a handle, then use logger(handle,fmt,...), similar to how you'd use printf().

The IPC mechanism returns data through a vret-structure. For vadmin, this is precisely how Varnish would return it.


ipc_run() dynamically allocates memory for ret->answer. FREE IT.

The logger also returns a vret-like structure, but the logger() macro handles this for you.

Hooking up to HTTP!

Hooking up to HTTP is ridiculously easy.

Let's look at echo, comments removed:

struct echo_priv_t {
        int logger;

static unsigned int echo_reply(struct httpd_request *request, void *data)
        struct echo_priv_t *echo = data;
        logger(echo->logger, "Responding to request");
        send_response(request->connection, 200, request->data, request->ndata);
        return 0;

void echo_init(struct agent_core_t *core)
        struct echo_priv_t *priv = malloc(sizeof(struct echo_priv_t));
        struct agent_plugin_t *plug;
        plug = plugin_find(core,"echo");
        priv->logger = ipc_register(core,"logger");
        plug->data = (void *)priv;
        plug->start = NULL;
        httpd_register_url(core, "/echo", M_POST | M_PUT | M_GET, echo_reply, priv);

This is the ENTIRE echo plugin. httpd_register_url() is the key here. It register a url-base, /echo in this case, and a set of request methods (POST, PUT and GET in this case. DELETE is also supported). A callback to execute and some optional private data.

The echo_reply function is now executed every time a POST, PUT or GET request is received for URLs starting with /echo.

You can respond with send_response() as demonstrated above, or the shorthands send_response_ok(request->connection, "Things are all OK!"); and send_response_fail(request->connection, "THINGS WENT BAD");.


Currently all http requests are handled in a single thread. This means you really really shouldn't block.

But make sure it's written with thread safety in mind. We might switch to a multi-threaded request handler in the future.

Know your HTTP

"REST"-interfaces are great, if implemented correctly. A short reminder:

  • GET requests are idempotent and should not cause side effects. They should be purely informational.
  • PUT requests are idempotent, but can cause side effects. Example: PUT /start can be run multiple times.
  • POST requests do not have to be idempotent, and can cause side effects. Example: POST /vcl/ will upload new copies of the VCL.
  • DELETE requests are idempotent, and can have side effects. Example: DELETE /vcl/foobar.

Test your code!

Unused code is broken code. Untested code is also broken code.

Pretty much all functionality is tested. Take a look in tests/.

If your code is to be included in an official release, someone has to write test cases.

I also advise you to add something in html/index.html to test it if that's feasible. It also tends to be quite fun.

Getting started

To get started, grab the code and get crackin'.

I advise you to read include/*.h thoroughly.