Planet Varnish

June 26, 2014

Per BuerStill using the GET and HEAD commands? Meet httpie.

I've been using libwww-perl's GET, POST and HEAD commands for the last ten years at least. Typing "GET -Used" is more or less muscle memory by now and my attempts and getting friendly with the curl command line has never bore fruit. libwww-perl is pretty ancient and it has some annoying snags and those of my friend who are using curl are not raving about their experiences either.  Thanks to a post on hacker news a couple of week back I found something new and it is has changed a very, very, very small part of my life. Somewhat.

June 25, 2014

Per BuerPartitioning your Varnish Cache

If you have multiple virtual hosts being handled by the same server you some times want some sort of QoS to apply to your caching. You might want to reserve a certain amount of memory for each virtual hosts. Let me show you how it can be done.

June 24, 2014

Per BuerNotes from the last Varnish Developer Day

Once every quarter there is a VDD - Varnish Developer Day. We typically gather between 8-16 people that are interested in Varnish development and discuss where we are and where we wanna go.  The last VDD was held in Stockholm, in the offices of Redpill Linpro in Solna. Roughly the outcome was something like this.

June 23, 2014

Per BuerAdding headers to gain insight into VCL

Varnish logs a lot. Sometimes it is a bit too much and verifying that your VCL works the way it is supposed to can be a bit of a bother, especially on a busy server. The new logging in Varnish 4  help a lot, but a lot of the time its easier to just a add a header or two to indicate what is happening.

June 20, 2014

Per BuerGrace in Varnish 4 - Stale-while-revalidate semantics in Varnish

Grace mode has since its inception in Varnish 2.1 been a key feature in Varnish. Initially a feature to mitigate thread pile-ups and the resulting thundering herd was soon adopted to other uses, mainly having Varnish to continue to serve requests when the backend got in trouble. As part of the rework done with the threading model in Varnish 4.0 things changed and grace got somewhat changed semantics. The main change being Varnish is now capable of delivering a stale object and issue an asynchronous refresh request, thereby removing the penalty the first user gets when hitting a stale object.

June 03, 2014

Lasse KarstensenWhat happened to ban.url in Varnish 4.0?

tl;dr; when using Varnish 4 and bans via varnishadm, instead of “ban.url EXPRESSION”, use “ban req.url ~ EXPRESSION”.

In Varnish 3.0 we had the ban.url command in the varnishadm CLI. This was a shortcut function expanding to the a bit cryptic (but powerful) ban command. In essence ban.url just took your expression, prefixed it with “req.url ~ ” and fed it to ban. No magic.

We deprecated this in Varnish 4.0, and now everyone has to update their CMS’s plugin for cache  invalidation. Hence this blog post. Perhaps it will help. Perhaps not. :-)

Some references:


May 16, 2014

Per BuerGetting virtual hosts right with Varnish Cache.

When answering questions in the forums, the mailing lists or our support, one of the most common topics to come up is virtual host. Virtual hosts are tricky and with Varnish and Apache/Nginx it is common to misconfigure it. Here I’ll explain how they actually work, how to verify it is working and how to set it up.

April 16, 2014

Per BuerThread organization in Varnish Cache 4.0

One of the two biggest changes in Varnish 4.0 is how the threads work. Varnish uses threads for doing all the heavy lifting and it seems to be working out quite well. In Varnish 3.0 one thread would service each client, doing whatever that client wanted it to do. Within reason, obviously. These are very decent threads. The thread would deliver from cache, fetch content from the backend, pipe, etc.

April 14, 2014

Yves HwangFrontend/Senior Software Developer wanted in our London or Oslo office

Exciting open source company seeks Frontend/Senior Software Developer in our London or Oslo office.

April 10, 2014

Hrafnhildur SmaradottirVarnish Cache 4.0 is released!

Varnish Cache 4.0 is out! The Varnish team is thrilled and has just celebrated with a sip of bubbly. We are happy to announce this release to our community and the world. It includes some awesome enhancements for all of you to enjoy!

Dr CarterVarnish 4.0.0 released

This is a major feature release with a lot of big changes from the last version released 2.5 years ago.

Some prominent examples include:

  • Full support for streaming objects through from the backend on a cache miss. Bytes will be sent to 1..n requesting clients as they come in from the backend server.
  • Background (re)fetch of expired objects. On a cache miss where a stale copy is available, serve the client the stale copy while fetching an updated copy from the backend in the background.
  • New varnishlog query language, allowing automatic grouping of requests when debugging ESI or a failed backend request. (among much more)
  • Comprehensive request timestamp and byte counters.
  • Security improvements, including disabling of run-time changes to security sensitive parameters.

Download at https://repo.varnish-cache.org/source/varnish-4.0.0.tar.gz.

Fuller list of changes: https://www.varnish-cache.org/trac/browser/doc/changes.rst

<script async="async" src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script>

April 09, 2014

Ruben RomeroVarnish 4.0 Q&amp;A on Performance, VMODs, SSL, IMS, SWR and more...

During our "What's coming in Varnish 4.0?" Hangout (see the video now) two weeks ago we got some questions. I am getting back to you with some answers.

MacYvesBuilding vagent2 for Varnish Cache 4.0.0 beta 1 for OS X 10.9.2

For those keen bunnies that wishes to jump in and help us test out varnish cache 4.0.0 Beta 1 with varnish-agent 2, here’s how you do it on OS X 10.9.2 Mavericks.

Prerequisites

Homebrew dependencies

Install the following with Homebrew

  • automake 1.14.1
  • libtool 2.4.2
  • pkg-config 0.28
  • pcre 8.34
  • libmicrohttpd 0.9.34

Build varnish cache 4.0.0 beta 1

  1. Download and extract varnish cache 4 https://repo.varnish-cache.org/source/varnish-4.0.0-beta1.tar.gz
  2. run ./autogen.sh
  3. run ./configure
  4. make

Build varnish-agent2 for varnish cache 4.0.0 beta 1

  1. Clone varnish-agent from repo https://github.com/varnish/vagent2
  2. Checkout the varnish-4.0-experimental branch
  3. export VARNISHAPI_CFLAGS=-I/tmp/varnish/varnish-4.0.0-beta1/include
  4. export VARNISHAPI_LIBS="-L/tmp/varnish/varnish-4.0.0-beta1/lib/libvarnishapi/.libs -lvarnishapi"
  5. run ./autogen.sh
  6. run ./configure
  7. make

Note that if you run make install for varnish cache 4 or varnish-agent, it would then install it for you respectively.


Per BuerLogging in Varnish 4.0

The logging in Varnish Cache is one of the unique features of Varnish that in my mind sets it apart from the rest of the software world. It combines both logging in great detail with both performance, manageability, sensible privacy defaults and access to great debugging help. The shared memory log, logs everything that happens without the need to adjust log levels or without significantly affecting performance, eliminating the need for turning on any debug switch. Many of you have surely been in a situation where you have an application that is misbehaving. Then you increase the log level to debug and the problem magically disappears - obviously caused by the application slowing down just enough that the race condition you encountered earlier is now gone. The weakness of logging everything is that so much information is available that the administrator can sometimes be overwhelmed by all the information presented. It’s a figurative firehose of information and drinking from it can be painfull.  Martin has reimplemented a new logging framework in Varnish Cache 4.0. Out of all the new stuff in Varnish Cache 4.0 this might be the most significant one. It’s also the most complex feature, requiring quite a bit of time to fully understand how it works.

Per BuerLocking down Varnish 4.0 - high security installations of Varnish

Varnish Cache 4.0 is just around the corner. We did a beta release a couple of weeks back and the feedback has been pretty good and I think the release will be out later this week. Anyways, there are a lot of canges in Varnish 4.0.  A couple of the changes in Varnish Cache 4.0 are security related ones, giving you the option to lock down a Varnish installation to make it ultra secure.

April 01, 2014

Per BuerVarnish Cache 4.0 beta 1 is out

The beta is out. Varnish Cache 4.0 is just around the corner and we need a bit of help to cross the finishing line. The beta seems to be pretty solid, we've given it a fair beating in out testing facilities as well as some pretty rough production workload. The final production tests last week where done in cooperation with A-Media and Redpill Linpro and where quite encouraging. Varnish 4.0 ran stable on their workload happily chugging away 1300 requests a second. If you have the option of giving Varnish Cache 4.0 a spin on your website, or giving it parts of your traffic we would be grateful. Crashes are very unlikely at this point, but just to be safe, we would recommend having a load balancer in front of the beta so you can gradually increase the load. If you need any help with converting the VCL, please let us know.

March 28, 2014

Per BuerWhen communities feel betrayed

This week the Internet has been all up in arms about Face acquiring Oculus Rift. Many people have been angry about Facebook buying this company, something we did not see when Facebook acquired WhatsApp or Instagram.

March 25, 2014

Yves HwangVCL change management and contiguous integration with your Varnish using VAC API and Git

Managing multiple Varnish instances and their respective VCL is made significantly easier with Varnish Administration Console (VAC) through its API. This blog post aims to illustrate an example of VCL change management and continuous integration with multiple Varnish instances using VAC API and a little magic from Git.

March 24, 2014

Ruben RomeroJoin our Varnish 4.0 Hangout on Wednesday

Two days to go for our Varnish 4.0 Hangout with the Varnish Core Developer team. Make sure you join us. RSVP now!

March 19, 2014

Yves HwangVAC 3

Software releases are not won by any single effort, but they are by processes, persistence, quality, and commitment. Most of all, solid teamwork. A release constitutes a bundling of various Git commits, neatly cherry picked to form a release candidate. From code changes to test automation, to build pipelines, the piece of software is scrutinised closely at every step.

March 17, 2014

Ruben RomeroVarnish Core Devs tell you what is coming in Varnish 4.0 (w/Q&amp;A)

Hangout On Air with PHK, Per, Martin & Lasse. We are now eagerly preparing for the release of Varnish 4.0. The release is planned for Q2 and to give you a sneak preview of what is coming we have scheduled a Google Hangout-on-Air session with Varnish’s Chief Architect, Poul-Henning Kamp, Varnish Software’s Founder and CTO, Per Buer, as well as our key Varnish Developers, Martin Blix-Grydeland and Lasse Karstensen.

March 13, 2014

Yves HwangReal-time statistics API with VAC

Varnish Administration Console (VAC) is broken up into two major components. Its Management UI, and a RESTful API, with the UI utilising the API. There are a diverse number of features supported by the API, for example, change management, statistics and etc. This blog post aims to showcase the Real-time Statistics API.

March 07, 2014

Hrafnhildur SmaradottirUsing Varnish Tools to visualize your most popular content

We just love it when our customers use our products in innovative ways that really help them create business value. One of the most recent examples of such use is how our customer vg.no. (Norway’s most popular internet news provider) is using Varnish Tools in combination with D3 to visualize their most popular online content.

February 21, 2014

ZenikaCreating a Varnish 4 module

Don't let the title fool you, Varnish 4 hasn't been released yet. However, the master branch (which holds Varnish 4 developments) in its current state should be close enough to what the first release will look like. I've been bitten by API changes several times, but that's the deal when writing code against work in progress[1].

Note

[1] Last tested against the revision d0c0ee9

What's new in Varnish 4 ? I will simply copy/paste an answer from Varnish Software (blame my laziness on this one): The 4.0 release is underway and should be released after the summer, with a prerelease during the summer, hopefully. The main new features are: New logging framework (see this blog post... Read Creating a Varnish 4 module

February 11, 2014

Yves HwangAdding Varnish Cache related statistics to your APM

This blog post aims to introduce AppDynamics Varnish Extension for gathering cache related statistics. At the core of AppDynamics Varnish Extensions is varnishstat data that are fed back to AppDynamics Pro via varnish-agent in order to provide key statistics for a complete performance overview of your web stack.

February 04, 2014

Yves HwangVarnish Hashninja explained

Recently we have had a few questions about the Varnish Hashninja VMOD, and how this all ties into Varnish Plus. This blog post aims to demystify the ever-so-elusive Hashninja, and help the readers find the best fit for their cache invalidation solution.

January 16, 2014

Per BuerIntroducing Varnish Plus

This month we’re launching Varnish Plus. The rationale behind Varnish Plus is to combine all our services and products in one offering, market it under one name. So, when you become our customer, you get access to everything you need in one place.

January 13, 2014

Per BuerAre you a C programmer in Oslo?

We're looking for a C-programmer in Oslo to work on Varnish and related stuff.

Hrafnhildur SmaradottirVarnish’s role in the NYTimes.com tech platform redesign

The NYTimes.com recently did a redesign of their website. Or rather, they did a complete reinvention of the website’s entire digital platform. In a blog post on their Open blog, they introduce the technology they used in the redesign. The objective with the redesign was to have a faster running code base and to make future changes easier.

December 19, 2013

Lasse KarstensenConverting a Varnish 3.0 VMOD to 4.0

So we’re getting closer to releasing the first proper 4.0 version of Varnish Cache. One of the things we need to fix is to get all the vmod writers to make sure their vmod works with the new version.

Here are my notes from doing just that, in the hope to make it simpler for others.

In 4.0, you don’t need the source tree of Varnish any more. The include files will be enough, and pkg-config will find them for you.

Make sure that /usr/lib/pkgconfig/varnishapi.pc and /usr/share/aclocal/varnish.m4 exists. If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source. (I need multiple varnishd versions when debugging customer problems, so I let them live in /opt/varnishX.Y/ on my laptop)

Pull/merge the new Makefile.am files from the master branch of libvmod-example.

Header files: remove bin/varnishd/cache.h and add cache/cache.h.

Vmod functions are now called with a vrt context as first argument. %s/struct sess \*sp/const struct vrt_ctx \*ctx/g

The old sess struct has been split, some data is in vrt_ctx->req, and some is in vrt_vtx->req->sp. Look up what is where in cache/cache.h. 

I’ve put up the 3.0->4.0 diff for vmod_policy.c as a gist: https://gist.github.com/lkarsten/8039861

There was a bit trouble of finding varnishtest, as src/Makefile was missing the reference entirely. I just fixed it by hand for now. Another thing for the 4.0 todolist, then.

And finally; 

lkarsten@immer:~/work/libvmod-policy/src$ make check
/opt/varnish/bin/varnishtest -Dvarnishd=/opt/varnish/sbin/varnishd -Dvmod_topbuild=/home/lkarsten/work/libvmod-policy tests/test01.vtc
# top TEST tests/test01.vtc passed (1.574)

 

I have a working Varnish 4.0 vmod. :-D


December 11, 2013

Lasse KarstensenDNS RBL test address for development

If you are writing code that checks a DNS real-time blockhole list (RBL), it looks like 127.0.0.2 is the standard address that is always in the black/white -list.

This is probably know for most sysadmins/security people and whatnot, but wasn’t entirely trivial to find using Google.

lkarsten@immer:~$ dig 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55083
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;2.0.0.127.dnsbl.sorbs.net. IN A
;; ANSWER SECTION:
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.10
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.5
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.7
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.2
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.3
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.9
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.14
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.4
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.6
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.8
;; Query time: 17 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Dec 11 14:12:20 2013
;; MSG SIZE rcvd: 203
lkarsten@immer:~$

Good to be able to actually test your code for hits also.

(this is for libvmod-policy, so you can deny/reject POST/PUT from spammers in Varnish)


November 29, 2013

Per Buer404 handling in Varnish Cache

When you have lots and lots of data, it’s hard to update it in a timely manner. Sometimes all you need is a bit of time while the systems in place are updating. A typical situation is the one where you have two backends serving the same data. However, one of them might lag behind the other one. When a request comes in to the one that is lagging it will throw a 404 Not Found. The solution is simple to describe; when you get a 404 try the other server.

November 21, 2013

Hrafnhildur SmaradottirVarnish Software wins Red Herring 100 Global Award

Last night, at an award ceremony in LA, California, we were awarded the Red Herring 100 Global award, a prize that recognizes Varnish Software as one of the top tech companies in the world.

November 14, 2013

Hrafnhildur SmaradottirHow does the Varnish Paywall really work?

Our paid-content solution, the Varnish Paywall, has provided real value to our customers. Some of them have seen a fast increase in number of subscribers. Others have with the help of the Varnish Paywall implemented more effective gateways to their online content and seen an increase in traffic.

November 05, 2013

Per BuerUsing pipe in Varnish

Quite often we see people using return(pipe) in VCL and then running in to various problems. Let me give you some information on what return(pipe) actually does and when it makes sense to use it and when it doesn’t.

October 28, 2013

Per BuerLive AB testing with Varnish and VCS

In this post we take a look at how to do AB-testing with Varnish and VCS. How to split users into groups, how to record success conversions and how get a live feed of data from the experiement.

October 25, 2013

Hrafnhildur SmaradottirVarnish Software is a Finalist for the 2013 Red Herring 100 Global Award

Varnish Software has been selected as a candidate for Red Herring's 2013 Top 100 Global award. The award honors this year’s most audacious and far reaching private technology companies and entrepreneurs from across the globe.

October 24, 2013

Yves HwangWeb traffic analysis with Varnish and VCS

This blog post aims to illustrate how web traffic statistics can be easily gathered in real-time and without involvement of any 3rd party vendors or the use of cookies. Once these statistical metrics are defined, conducting A/B testing through Varnish is also outlined.

October 22, 2013

Yves HwangVCS 1.3.0 is out!

Varnish Custom Statistic 1.3.0 has been released. It is available for RHEL6, Precise, Squeeze and Wheezy (amd64) today.

October 14, 2013

Hrafnhildur SmaradottirVarnish receives the Global Hot 100 Award

Varnish Software was honored to receive the Global Hot 100 Award at The World Summit on Innovation & Entrepreneurship (THEWSIE) in New York last Friday.

October 08, 2013

Hrafnhildur SmaradottirHow Newsday and News12 bumped up both traffic and speed with Varnish

When Dennis Elder, Newday's vice president for Digital Development, contacted us last year the news sites he manages, including Newsday and News12, were not performing optimally when it came to capacity and speed. Dennis wanted to use Varnish to solve those issues and he also had hopes that Varnish would be able to help accelerate the paid-content infrastructure on his sites.

September 25, 2013

Lasse KarstensenVarnish and Ghost blogging software

So there is a new shiny blogging platform out called Ghost. Looks pretty good to me.

If you want to run it behind Varnish, you’ll soon notice it has the usual problem of setting session cookies everywhere leading to 0% hit rate. 

I have written a Varnish VCL configuration for filtering this in the necessary places, while keeping the admin interface working still.

You can find it here:

https://gist.github.com/lkarsten/6683179

Have fun.


September 24, 2013

Hrafnhildur SmaradottirOur October webinars

We have scheduled two webinars for October. They are both on the topic of caching. Surprise, surprise!

September 19, 2013

Hrafnhildur SmaradottirInfoWorld names Varnish Cache Best Open Source Software 2013

InfoWorld recognizes the best open source software for business each year by handing out the Bossies (Best of Open Source Software awards) to a handpicked selection of open source applications.

September 18, 2013

Hrafnhildur SmaradottirThe value of using Varnish as a part of your e-commerce infrastructure, part 2

In my previous post on this topic I told you about our e-commerce customer TradeMe and how they have used Varnish to boost the user experience on their site to help them outperform their competitors. In this article I will explain how you can use the Varnish Hashninja to further enhance your web performance by making your e-commerce website more efficient than you thought possible.

September 13, 2013

Yves HwangSimple scales better and faster in the real world

The Super Fast Purger was conceived as a plugin for the Varnish Administration Console. The current implementation is capable of receiving and distributing 45.000 cache invalidation requests per second with ease. We believe that the upper limit of the throughput for the Super Fast Purger is yet to be determined as the software implementation was limited to network and hardware capabilities during benchmarking.

September 10, 2013

Lasse KarstensenTesting VMODs with Travis (travis-ci.org)

Travis CI is a service where open source software can run tests automatically on commits. It hooks into github in a silky smooth way.

If you’re developing a Varnish module (VMOD), you probably have started out with our libvmod-example package. It has all the automake magic you need, as well as some simple test cases to test your vmod. Given that you’ve written som varnishtest testcases for it (you really should), you now can get travis to run them as well!

I’ve put a small recipe for this into the libvmod-example package.

Feel free to play around with it, feedback appreciated. For the travis setup bits, consult the travis getting started guide. The final result is something like this, shown for libvmod-cookie:

https://travis-ci.org/lkarsten/libvmod-cookie


September 09, 2013

MacYvesVagrant, Varnish and vmods

Development environment has been plaguing us for a while in my product development department. From dependencies hell to complex setup in operations, our development environment has gone through the usual gauntlet of pains and complaints.

This has changed with Vagrant. It is the single tool that gels the devs with the ops; quintessential devop tool if you will. Not only Vagrant has helped eliminate the “works on my machine” bugs, we use it for automated integration tests. In addition, this one tool has made our development environment setup quick and simple for our HCI guys too.

We do a lot of integration work with Varnish Cache and I thought I would take this opportunity to share this simple Vagrantfile, as an example, to help get started with installing varnish and libdigest-vmod from source.

Note that the provisioning process is rather crude in this example. Rather, the intention here is to out outline the steps required to get varnish and vmods installed and running via Vagrant. For production and future maintainability, do use Chef or Puppet as it can be seamlessly integrated within the Vagrantfile.


# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.define :varnish do |varnish|
    varnish.vm.box = "varnish"
    varnish.vm.box_url = "http://files.vagrantup.com/precise64.box"
    $script_varnish = <<SCRIPT
echo Installing dependencies, curl
sudo apt-get update
sudo apt-get install curl -y
sudo apt-get install git -y
curl http://repo.varnish-cache.org/debian/GPG-key.txt | sudo apt-key add -
echo "deb http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
echo ==== Compiling and installing Varnish from source ====
sudo apt-get install build-essential -y
sudo apt-get build-dep varnish -y
apt-get source varnish
cd varnish-3.0.4
./autogen.sh
./configure
make
sudo make install
cd ..
echo done
echo ==== Compiling and installing lib-digest vmod from source ===
git clone https://github.com/varnish/libvmod-digest.git
sudo apt-get install libmhash-dev libmhash2 -y
cd libvmod-digest
./autogen.sh
./configure VARNISHSRC=/home/vagrant/varnish-3.0.4 VMODDIR=/usr/local/lib/varnish/vmods
sudo make install
cd ..
echo ===== done ====
echo ===== firing up varnish via commandline ====
sudo varnishd -a :80 -T :6081 -f /vagrant/test.vcl
touch varnish_vm
SCRIPT

    varnish.vm.provision :shell, :inline => $script_varnish
  end

end


September 05, 2013

Hrafnhildur SmaradottirVarnish Software nominated as one of the World’s Most-Innovative Emerging Companies

We are thrilled to be able to share with you that we have been named a Global Hot 100 Company by The World Summit on Innovation & Entrepreneurship (THEWSIE).

August 01, 2013

ZenikaCreating a Varnish module


Warning: Parameter 1 to rsExtPost::getExcerpt() expected to be a reference, value given in /home/zenikaco/public_html/inc/clearbricks/dblayer/dblayer.php on line 618

Warning: Parameter 1 to rsExtPost::getContent() expected to be a reference, value given in /home/zenikaco/public_html/inc/clearbricks/dblayer/dblayer.php on line 618

July 29, 2013

Lasse KarstensenBuilding a Varnish VMOD on Debian

From the tutorials department, here are some quick notes on how to install a Varnish VMOD from source.

This is slightly complicated because Varnish demands that a VMOD must be built against the same git commit (or release) as the one that is running. This will be relaxed in future versions.

Current setup is a standalone Varnish VM on Debian Wheezy with Varnish installed from varnish package archives (3.0.4-1~wheezy.)

1. Get the vmod

lkarsten@lb1:~$ git clone https://github.com/lkarsten/libvmod-cookie.git
Cloning into 'libvmod-cookie'...
remote: Counting objects: 253, done.
remote: Compressing objects: 100% (131/131), done.
remote: Total 253 (delta 132), reused 232 (delta 112)
Receiving objects: 100% (253/253), 49.51 KiB, done.
Resolving deltas: 100% (132/132), done.
lkarsten@lb1:~$

2. Get and configure the source tree for the running Varnish

Verify first that you have the necessary package repositories enabled:

lkarsten@lb1:~$ grep varnish /etc/apt/sources.list
deb http://repo.varnish-cache.org/debian/ wheezy varnish-3.0
deb-src http://repo.varnish-cache.org/debian/ wheezy varnish-3.0
lkarsten@lb1:~$

After that, continue with the juicy parts:

lkarsten@lb1:~$ apt-get source varnish 
Reading package lists... Done 
Building dependency tree 
Reading state information... Done 
NOTICE: 'varnish' packaging is maintained in the 'Git' version control system at: 
git://git.debian.org/pkg-varnish/pkg-varnish.git 
Need to get 2,060 kB of source archives. 
Get:1 http://repo.varnish-cache.org/debian/ wheezy/varnish-3.0 varnish 3.0.4-1 (dsc) [2,334 B] 
Get:2 http://repo.varnish-cache.org/debian/ wheezy/varnish-3.0 varnish 3.0.4-1 (tar) [2,044 kB] 
Get:3 http://repo.varnish-cache.org/debian/ wheezy/varnish-3.0 varnish 3.0.4-1 (diff) [14.1 kB] 
Fetched 2,060 kB in 0s (11.4 MB/s) 
gpgv: keyblock resource `/home/lkarsten/.gnupg/trustedkeys.gpg': file open error 
gpgv: Signature made Fri 14 Jun 2013 11:56:48 CEST using RSA key ID 87218D9C 
gpgv: Can't check signature: public key not found 
dpkg-source: warning: failed to verify signature on ./varnish_3.0.4-1.dsc 
dpkg-source: info: extracting varnish in varnish-3.0.4 
dpkg-source: info: unpacking varnish_3.0.4.orig.tar.gz 
dpkg-source: info: applying varnish_3.0.4-1.diff.gz 
lkarsten@lb1:~$
lkarsten@lb1:~$ cd varnish-3.0.4
lkarsten@lb1:~/varnish-3.0.4$ ./autogen.sh
[..]
lkarsten@lb1:~/varnish-3.0.4$ ./configure --prefix=/usr
[..]
lkarsten@lb1:~/varnish-3.0.4$ make

If configure or make fails, you might need some additional packages. Run an apt-get build-dep varnish and work from there. (if editline fails on you, remember to rerun configure after installing it)

3. Build and install the vmod

lkarsten@lb1:~$ cd libvmod-cookie/
lkarsten@lb1:~/libvmod-cookie$ ./autogen.sh
+ aclocal -I m4
+ libtoolize --copy --force
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
+ autoheader
+ automake --add-missing --copy --foreign
configure.ac:8: installing `./config.guess'
configure.ac:8: installing `./config.sub'
configure.ac:11: installing `./install-sh'
configure.ac:11: installing `./missing'
src/Makefile.am: installing `./depcomp'
+ autoconf
lkarsten@lb1:~/libvmod-cookie$ 
lkarsten@lb1:~/libvmod-cookie$ ./configure VARNISHSRC=~/varnish-3.0.4/
[..]
# and finally
lkarsten@lb1:~/libvmod-cookie$ make
[..]
libtool: link: ( cd ".libs" && rm -f "libvmod_cookie.la" && ln -s "../libvmod_cookie.la" "libvmod_cookie.la" )
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie/src'
make[2]: Entering directory `/home/lkarsten/libvmod-cookie'
rst2man README.rst vmod_cookie.3
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'
lkarsten@lb1:~/libvmod-cookie$ 
lkarsten@lb1:~/libvmod-cookie$ sudo make install
[..]
/bin/mkdir -p '/usr/local/share/man/man3'
 /usr/bin/install -c -m 644 vmod_cookie.3 '/usr/local/share/man/man3'
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'
lkarsten@lb1:~/libvmod-cookie$

At this point you should have the two vmod files available for Varnish:

lkarsten@lb1:~/libvmod-cookie$ ls -l /usr/lib/varnish/vmods/
total 64
-rwxr-xr-x 1 root root 966 Jul 29 11:11 libvmod_cookie.la
-rwxr-xr-x 1 root root 41538 Jul 29 11:11 libvmod_cookie.so
-rw-r--r-- 1 root root 16128 Jun 17 13:38 libvmod_std.so
lkarsten@lb1:~/libvmod-cookie$

And you are done!

"import cookie" should now work without issue in your /etc/varnish/default.vcl.


July 23, 2013

Per BuerThe essential vmods all Varnish users should know about

Approximately two years ago we released Varnish Cache 3.0. In my view the most important feature was the support for modules. Thanks to the hello world vmod, which almost all of the modules out there are based on, it’s rather easy to write a trivial module. It is obviously more complicated to write more complex modules and since there is no sandboxing involved we see modules wrecking havoc in varnishd regularly. Memory leaks and segfaults are found in modules from time to time. However, there are modules that can add really wonderful functionality without destabilizing your varnishd. In my opinion these are some of the modules all Varnish users should know about.

July 22, 2013

Lasse KarstensenSetting client.ip in Varnish VCL with libvmod-ipcast

I’ve written a new Varnish 3.0 VMOD called ipcast.

It has a single function; ipcast.clientip(ipstring) which sets the internal Varnish variable client.ip to whatever IPv4/IPv6 address you give as the argument.

You need this if you want to do ACL checks on connections done through a load balancer or SSL terminator. In those cases client.ip would be 127.0.0.1 and you get the real client’s IP address in the X-Forwarded-For (or similar) header.

You can find it here:

https://github.com/lkarsten/libvmod-ipcast

Here is some example VCL to illustrate how it works. I think the regex needs some work, suggestions/pull requests are welcome.

import ipcast;
acl friendly_network {
    "192.0.2.0"/24;
}
sub vcl_recv {
    if (req.http.X-Forwarded-For !~ ",") {
        set req.http.xff = req.http.X-Forwarded-For;
    } else {
        set req.http.xff = regsub(req.http.X-Forwarded-For,
                "^[^,]+.?.?(.*)$", "\1");
    }

    if (ipcast.clientip(req.http.xff) != 0) {
        error 400 "Bad request";
    }

    if (client.ip !~ friendly_network) {
            error 403 "Forbidden";
    }
}

July 18, 2013

cd34Varnish and Node.js

While working with a client installation they wanted to run Varnish in front of their node.js powered site to eliminate having node serve the static assets. Socket.io uses HTTP/1.0 and cannot be cached. Minimally these few lines can be added to their respective functions and things will work. Obviously you’ll want to set expires on […]

July 15, 2013

Lasse Karstensenlibvmod-cookie; simpler cookie handling in Varnish VCL

I’ve written a Varnish module (VMOD) to simplify handling of HTTP cookies in Varnish VCL.

In essence it does the usual stuff you’d usually do with regular expressions, but with much simpler syntax and readability.

It works by parsing the req.http.Cookie into a small data structure. Afterwards you can add/delete/filter the list, and finally ask for a combined string back again.

Here is the list of functions:

  • cookie.parse(cookiestring) – parse the http.req.Cookie string.
  • cookie.get(cookiename) – get the value of a single parsed cookie.
  • cookie.set(cookiename, cookievalue) – set the value of a cookie.
  • cookie.delete(cookiename) – delete a single cookie.
  • cookie.filter_except(filterstring). Delete all cookies but the ones matching the names in the comma-separated filterstring (“cookie1,cookie2,cookie3″.) This should be quite useful.
  • cookie.get_string() – return the parsed string.

You can find it here:

https://github.com/lkarsten/libvmod-cookie


June 30, 2013

ops42Properly redirect to mobile pages

It is just amazing how much advice and examples one can find for how to redirect to a mobile equivalent of a given HTTP address. Oversimplified, wrong and harmful advice that is. And no, I’m not talking about that 301 vs 302 bullshit. For the love of God, stop listening to those overpaid, know-nothing SEO […]

June 17, 2013

Dr CarterVarnish 3.0.4 released

Summary of changes from 3.0.3 to 3.0.4

  • The ACL code had a bug which could lead to false negatives. This has been assigned CVE-2013-4090.
  • Varnish will now return an error if the client sends multiple Host headers.
  • If the backend sent invalid gzip while using ESI, Varnish would in some cases assert. It now works correctly.
  • TCP_NODELAY is now enabled, which should lead to performance improvements in some cases.

Download here.

Fuller list of changes: https://www.varnish-cache.org/trac/browser/doc/changes.rst

June 13, 2013

Per BuerSpeculative Lock Elision in Varnish Cache

This year we’ve seen quite an interesting announcement from Intel. Their Haswell processors are now on sale. Usually we haven’t cared much about what Intel releases as new features in CPUs don’t always impact the performance on server software that much. Haswell, on the other hand, now has a feature, Transactional memory, that is pretty cool and might actually influence Varnish's performance quite a bit. Here is why.

Per BuerAdvanced cache invalidation strategies

“There are only two hard things in Computer Science: cache invalidation and naming things.” -- Phil Karlton Today, I’ll write about the hardest one of these two; cache invalidation. This used to be rather straightforward. Most websites had a rather simple structure between their content repository and their URL structure. This made it rather simple to purge URLs when the content changed.

May 27, 2013

Per BuerQ&amp;A from the &quot;You can cache everything&quot; webinar

We lost the audio on the Q&A session of the "You can cache everything" webinar. Here are the questions we got along with their answers.