Planet Varnish

April 13, 2015

Stefan CaunterThe unconditional interest of Leafs fans

Leafs tickets are seen as an investment to hold, not a conditional payment on success.The Leaf team is an incredibly valuable sport property that is basically destroyed every year by the media that keep them incredibly valuable. How? The players are given exalted status based on next to nothing on an achievement scale. People get […]

April 09, 2015

Per BuerIntroducing the Varnish API Engine

Over the last couple of years we’ve seen an explosion in the use of HTTP-based APIs. We’ve seen them go from being a rather slow and useless but interesting technology fifteen years ago to today's current, high performance RESTful interfaces that powers much of the web and most of the app-space. Varnish Cache has been used for HTTP-based APIs since its inception. The combination of caching, high performance and the flexibility brought by VCL makes it an ideal proxy for APIs. We’ve seen people doing rather complex protocol negotiations in VCL to do interesting things like matching frontend and backend protocols.

March 05, 2015

Ingvar Hagelundvarnish-4.0.3 for Fedora and EPEL

varnish-4.0.3 was released recently. I have wrapped packages for Fedora and EPEL, and requested updates for epel7, f21 and f22. They will trickle down as stable updates within some days. I have also built packages for el6, and after som small patching, even for el5. These builds are based on the Fedora package, but should be only cosmetically different from the el6 and el7 packages available from http://varnish-cache.org/.

Also note that Red Hat finally caught up, and imported the necessary selinux-policy changes for Varnish from fedora into el7. With selinux-policy-3.13.1-23.el7, Varnish starts fine in enforcing mode. See RHBA-2015-0458.

My builds for el5 and el6 are available here: http://users.linpro.no/ingvar/varnish/4.0.3/. Note that they need other packages from EPEL to work.

Update 1: I also provide an selinux module for those running varnish-4.0 on el6. It should work for all versions of varnish-4.0, including mine and the ones from varnish-cache.org.

Update 2: Updated builds with a patch for bugzilla ticket 1200034 are pushed for testing in f21, f22 and epel7. el5 and el6 builds are available on link above.

Enjoy.

Ingvar

Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at redpill-linpro.com.

March 04, 2015

Hrafnhildur SmaradottirPoul Henning Kamp speaking at Copenhagen Summit

With the Varnish Summit in Copenhagen only three weeks away (March 26th) we are thrilled to announce that Poul-Henning Kamp, the lead architect and developer of Varnish Cache, will be joining us there.

February 26, 2015

Hrafnhildur SmaradottirVarnish Summit in London sold out!

We are looking forward to the Varnish Summit in London next week (March 5 in Tech City). And Varnish users and other enthusiasts are apparently equally excited because we are sold out and have a full waiting list!

February 16, 2015

Hrafnhildur SmaradottirGirl geeks meet-up at Varnish Software

Last week we had the pleasure to co-host a Girl Geek Dinner event at our Oslo offices in collaboration with the Norwegian chapter of the event series (#ggdo).

February 11, 2015

Hrafnhildur SmaradottirKeen on meeting up with us in Copenhagen?

The Varnish Software series 2015 is off to a great start and next stop is wonderful Copenhagen. To respond to last year's popularity of the series in Scandinavia we decided Copenhagen would be one of the first cities we'd visit.

January 28, 2015

Hrafnhildur SmaradottirLondon Tech City, here we come!

We are thrilled to announce the venue for this year's Varnish summit in London. We've selected a beautiful penthouse in East London Tech City.

January 19, 2015

Lasse KarstensenPROXY protocol in Varnish

Dag has been working implementing support for HAProxy’s PROXY protocol[1] in Varnish. This is a protocol adds a small header on each incoming TCP connection that describes who the real client is, added by (for example) an SSL terminating process. (since srcip is the terminating proxy)

We’re aiming for merging this into Varnish master (so perhaps in 4.1?) when it is ready.

The code is still somewhat unfinished, timeouts are lacking and some polishing needed, but it works and can be played with in a development setup.

Code can be found here: https://github.com/daghf/varnish-cache/tree/PROXY

I think Dag is using haproxy to test it with. I’ve run it with stunnel (some connection:close issues to figure out still), and I’d love if someone could test it with ELB, stud or other PROXY implementations.

1: http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt


January 08, 2015

Ingvar Hagelundrpm packages of vmod-ipcast

Still on varnish-3.0? Missing the ability to filter X-Forwarded-For through ACLs? Use vmod ipcast by Lasse Karstensen.

I cleaned up and rolled an rpm package of vmod-ipcast-1.2 for varnish-3.0.6 on el6. It’s available here: http://users.linpro.no/ingvar/varnish/vmod-ipcast/.

Note that the usage has changed a bit since the last version. You are now longer permitted to change client.ip (and that’s probably a good thing). Now it’s called like this, returning an IP address object:

ipcast.ip("string","fallback_ip");

If the string does not resemble an IP address, the fallback ip is returned. Note that if the fallback ip is an unvalid address, varnishd will crash!

So, if you want to filter X-Forwarded-For through an ACL, you would something like this:

import ipcast;
sub vcl_recv {
   # Add some code to sanitize X-Forwarded-For above here, so it resembles one single IP address
   if ( ipcast.ip(req.http.X-Forwarded-For, "198.51.100.255") ~ someacl ) {
     # Do something special
   }
}

And that’s all for today.

Varnish Cache is powerful and feature rich front side web cache. It is also very fast, that is, Fast as in on steroids, and powered by The Dark Side of the Force.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at redpill-linpro.com.

November 27, 2014

Per BuerLinux vm tunables

There are quite a few tunables in the Linux kernel. Reading the documentation it is clear that quite a few of them could have an impact on how Varnish performs. One that caught my attention is dirty_background_writeback tunable. It allows you to set a limit for how much of the page cache would be dirty, i.e. contain data not yet written to disk, before the kernel will start writing it out.

Per BuerIntroducing Varnish High Availability

There is a reason why people install Varnish Cache in their servers. It’s all about the performance. Delivering content from Varnish is often a thousand times faster than delivering it from your web server. As your website grows, and it usually grows significantly if your cache hit rates are high, you’ll rely more and more on Varnish to deliver the brunt of the requests that are continuously flowing into your infrastructure.

November 24, 2014

Per BuerIntroducing Varnish High Availability

There is a reason why people install Varnish Cache in their servers. It’s all about the performance. Delivering content from Varnish is often a thousand times faster than delivering it from your web server. As your website grows, and it usually grows significantly if your cache hit rates are high, you’ll rely more and more on Varnish to deliver the brunt of the requests that are continuously flowing into your infrastructure.

Per BuerIntroducing Varnish High Availability

There is a reason why people install Varnish Cache in their servers. It’s all about the performance. Delivering content from Varnish is often a thousand times faster than delivering it from your web server. As your website grows, and it usually grows significantly if your cache hit rates are high, you’ll rely more and more on Varnish to deliver the brunt of the requests that are continuously flowing into your infrastructure.

Per BuerIntroducing Varnish High Availability

There is a reason why people install Varnish Cache in their servers. It’s all about the performance. Delivering content from Varnish is often a thousand times faster than delivering it from your web server. As your website grows, and it usually grows significantly if your cache hit rates are high, you’ll rely more and more on Varnish to deliver the brunt of the requests that are continuously flowing into your infrastructure.

Per BuerIntroducing Varnish High Availability

There is a reason why people install Varnish Cache in their servers. It’s all about the performance. Delivering content from Varnish is often a thousand times faster than delivering it from your web server. As your website grows, and it usually grows significantly if your cache hit rates are high, you’ll rely more and more on Varnish to deliver the brunt of the requests that are continuously flowing into your infrastructure.

November 19, 2014

Per BuerIntroducing Varnish Massive Storage Engine

Varnish was initially made for web site acceleration. We started out using a memory mapped file to store objects in. It had some problems associated with it and was replaced with a storage engine that relied on malloc to store content. While it usually performed better than the memory mapped files performance suffered as the content grew past the limitations imposed by physical memory.

November 10, 2014

Per BuerScaling up the dmcache tests

Following up on the test of dmcache we decided to scale it up a bit, to get better proportions between RAM, SSD and HDD. So we took the dataset and made it 10 times bigger. It is now somewhere around 30GB with an average object size of 800Kbyte. In addition we made the backing store for Varnish ten times biggers as well, increasing it from 2GB to 20GB. This way we retain the cache hit rate of around 84% but we change the internal cache hit rates significantly. The results where pretty amazing and shows what a powerful addition dmcache can be for IO intensive workloads.

November 07, 2014

Per BuerAccelerating your HDD with dm-cache or bcache

As you might or might not know we’ve been working on this storage backend for a year now, built for handling large data volumes like the ones we see in online video and CDNs. The new storage backend is written with performance in mind leveraging some novel ideas we have to make things go a lot faster. If you want to know more you can get in touch with me or come to one of our summits where I’ll be presenting. Since the new storage engine relies much more on IO capacity we’re suddenly looking at things such as filesystems and IO performance. We expect most of the deployments of this software to take place on solid state drives. However, if you are going to cache up a petabyte sized video library, having a secondary cache level with some SATA-based disk cabinets might make a lot of sense. The problem with SATA based storage is that the IO capacity is abysmal compared to solid state drives. So, we want as much memory as possible to achieve a reasonably high hit rate for the Linux page cache. However, if you attach 20 terabyte of SATA storage 384GB of RAM is still a bit less than I would like to see in order to cache this. So, how do we cache 20TB in a reasonably cost-effective way.

October 13, 2014

Lasse KarstensenVarnish VMOD static code analysis

I recently went looking for something similar to pep8/pylint when writing Varnish VMODs, and ended up with OCLint.

I can’t really speak to how good it is, but it catches the basic stuff I was interested in.

The documentation is mostly for cmake, so I’ll give a small tutorial for automake:

  • (download+install oclint to somewhere in $PATH)
  • apt-get install bear
  • cd libvmod-xxx
  • ./autogen.sh; ./configure –prefix=/usr
  • bear make # “build ear” == bear. writes compile_commands.json
  • cd src
  • oclint libvmod-xxx.c # profit
  • Which will tell you about unused variables, useless parentheses, dead code and so on.


    October 03, 2014

    Lasse KarstensenAnnouncing libvmod-tcp: Adjust Varnish congestion control algorithm.

    I’ve uploaded my new TCP VMOD for Varnish 4 to github, you can find it here:
    http://github.com/lkarsten/libvmod-tcp.

    This VMOD allows you to get the estimated client socket round trip time, and then let you change the TCP connection’s congestion control algorithm if you’re so inclined.

    Research[tm][0] says that Hybla is better for long high latency links, so currently that is what it is used for.

    Here is a quick VCL example:

    if (tcp.get_estimated_rtt() > 300) {
    set req.http.x-tcp = tcp.congestion_algorithm("hybla");
    }

    One thing to note is that VCL handling is very early in the TCP connection lifetime. We’ve only just read and acked the HTTP request. The readings may be off, I’m analyzing this currently.
    (As I understand it the Linux kernel will keep per-ip statistics, so for subsequent requests this should get better and better..)

    References:
    0: Esterhuizen, A., and A. E. Krzesinski. “TCP Congestion Control Comparison.” (2012).


    October 02, 2014

    Per BuerMajor new product releases scheduled for the Varnish Summits

    Announcement: For the last year we’ve been doing quite a bit of development which we’re finally ready to present. We have three major product updates that we’re about to launch. In the upcoming Summits we’ll be announcing these three updates.

    September 30, 2014

    Lasse KarstensenFresh Varnish packages for Debian/Ubuntu and Redhat systems

    We use continuous integration when developing Varnish Cache. This means that we run our internal test suite (varnishtest) on all commits, so we catch our mistakes earlier.

    This pipeline of build jobs sometimes end up with binary packages of Varnish, which may be useful to people when they know they exist. They may not be the easiest to find, which this blog post tries to remedy.

    Development wise, Varnish Cache is developed with GIT with a master branch for development and a set of production branches, currently 3.0 and 4.0.

    Unreleased packages for Varnish master can be found here: https://jenkins.varnish-software.com/view/varnish-master/

    Unreleased packages of Varnish 4.0 can be found here: https://jenkins.varnish-software.com/view/varnish-4.0/

    (There is also a set of 3.0 jobs, but you should really go for 4.0 these days.)

    The latest commits in each of the production branches may contain fixes we’ve added after the last production release, but haven’t cut a formal release for yet. (For example there are some gzip fixes in the 3.0 branch awaiting a 3.0.6 release, which I really should get out soon.)

    Some jobs in the job listing just check that Varnish builds, without creating any output (or artifacts as Jenkins calls it.) This applies for any jobs with “-build-” in the name, for example varnish-4.0-build-el7-x86_64 and varnish-4.0-build-freebsd10-amd64.

    The Debian and Ubuntu packages are all built from one job currently, called varnish-VERSION-deb-debian-wheezy-amd64. Press “Expand all” under artifacts to get the full list.

    Redhat/RHEL packages are built in the different el5/el6/el7 jobs.

    The unreleased packages built for 3.0 and 4.0 are safe. This is the process used to build the officially released packages, just a step earlier in the process. The varnish-master packages are of course failing from time to time, but that is to be expected.

    The version numbers in the packages produced may be a bit strange, but that is what you get with unreleased software builds.

    I’m happy to improve this process and system if it can help you run never versions of Varnish, comments (either here or on IRC) are appreciated.


    September 25, 2014

    Ruben RomeroVarnish Developer News: VDD14Q3 @ VG.no/Schibsted in Oslo, Norway

    Once every quarter there is a VDD - Varnish Developer Day. We typically gather between 8-18 people that are interested in Varnish development and discuss Varnish development. unless you want to use your day in Varnish packaging, internals or VCL design, you are better off just reading this post and/or the notes from the meeting :-)The last VDD was held in Oslo, in the offices of VG.no and Schibsted at Akersgata. After that we went to a DevOps meetup and finished the evening going for some food and drinks.

    September 16, 2014

    Espen BraastadBlog for a Sysadmin - Monitoring Health in Varnish Cache

    At Varnish Software, we like to share tips and tricks and ensure our knowledge is being shared with our readers. In what I hope will become a series under the guise of 'Blog for a Sysadmin', I'd like to take you through the essentials of maintaining your Varnish Cache setup. First up—Monitoring your Varnish Cache setup.

    September 10, 2014

    Per BuerA message for North America from Per Buer, Founder and CTO

    Announcement: Now well known amongst the Varnish Cache and tech community, we’ve got eyes for North America, namely in hotbeds like Silicon Valley (surprise, surprise). So, what better way to show we’re serious than hosting a Varnish Summit in San Francisco to present and discuss all things Varnish. Learn from the masters on 4 December.

    September 01, 2014

    Ruben RomeroNext Varnish Core Developer meeting to happen in Oslo

    Just a reminder that on September 17th the developer team will be gathering in Oslo for their quarterly meeting.

    August 21, 2014

    Espen Braastad10 Varnish Cache mistakes and how to avoid them

    Varnish Cache is one of the world’s most popular HTTP caching solutions. Like any piece of critical web content delivery software, it pays to know some fundamentals and have a few tricks up your sleeve in order to navigate your optimum setup and ensure things are running smoothly when you need it most. We’ve compiled two handfuls of common mistakes Varnish Cache users can run into. If you have more you’d like to suggest, we’d love to write about them. Simply comment below or email blog@varnish-software.com.

    June 26, 2014

    Per BuerStill using the GET and HEAD commands? Meet httpie.

    I've been using libwww-perl's GET, POST and HEAD commands for the last ten years at least. Typing "GET -Used" is more or less muscle memory by now and my attempts and getting friendly with the curl command line has never bore fruit. libwww-perl is pretty ancient and it has some annoying snags and those of my friend who are using curl are not raving about their experiences either.  Thanks to a post on hacker news a couple of week back I found something new and it is has changed a very, very, very small part of my life. Somewhat.

    June 25, 2014

    Per BuerPartitioning your Varnish Cache

    If you have multiple virtual hosts being handled by the same server you some times want some sort of QoS to apply to your caching. You might want to reserve a certain amount of memory for each virtual hosts. Let me show you how it can be done.

    June 24, 2014

    Per BuerNotes from the last Varnish Developer Day

    Once every quarter there is a VDD - Varnish Developer Day. We typically gather between 8-16 people that are interested in Varnish development and discuss where we are and where we wanna go.  The last VDD was held in Stockholm, in the offices of Redpill Linpro in Solna. Roughly the outcome was something like this.

    June 23, 2014

    Per BuerAdding headers to gain insight into VCL

    Varnish logs a lot. Sometimes it is a bit too much and verifying that your VCL works the way it is supposed to can be a bit of a bother, especially on a busy server. The new logging in Varnish 4  help a lot, but a lot of the time its easier to just a add a header or two to indicate what is happening.

    June 20, 2014

    Per BuerGrace in Varnish 4 - Stale-while-revalidate semantics in Varnish

    Grace mode has since its inception in Varnish 2.1 been a key feature in Varnish. Initially a feature to mitigate thread pile-ups and the resulting thundering herd was soon adopted to other uses, mainly having Varnish to continue to serve requests when the backend got in trouble. As part of the rework done with the threading model in Varnish 4.0 things changed and grace got somewhat changed semantics. The main change being Varnish is now capable of delivering a stale object and issue an asynchronous refresh request, thereby removing the penalty the first user gets when hitting a stale object.

    June 10, 2014

    Espen BraastadIntroducing the Varnish Tuner

    Varnish Cache is a high performance web application accelerator that performs very good out of the box. The default parameters have evolved since the start of the project, and they fit fine for a small site or a developer installation. However in order to run Varnish at scale you might want to tune it a bit to make it perform and scale even better.

    June 03, 2014

    Lasse KarstensenWhat happened to ban.url in Varnish 4.0?

    tl;dr; when using Varnish 4 and bans via varnishadm, instead of “ban.url EXPRESSION”, use “ban req.url ~ EXPRESSION”.

    In Varnish 3.0 we had the ban.url command in the varnishadm CLI. This was a shortcut function expanding to the a bit cryptic (but powerful) ban command. In essence ban.url just took your expression, prefixed it with “req.url ~ ” and fed it to ban. No magic.

    We deprecated this in Varnish 4.0, and now everyone has to update their CMS’s plugin for cache  invalidation. Hence this blog post. Perhaps it will help. Perhaps not. :-)

    Some references:


    May 16, 2014

    Per BuerGetting virtual hosts right with Varnish Cache.

    When answering questions in the forums, the mailing lists or our support, one of the most common topics to come up is virtual host. Virtual hosts are tricky and with Varnish and Apache/Nginx it is common to misconfigure it. Here I’ll explain how they actually work, how to verify it is working and how to set it up.

    April 16, 2014

    Per BuerThread organization in Varnish Cache 4.0

    One of the two biggest changes in Varnish 4.0 is how the threads work. Varnish uses threads for doing all the heavy lifting and it seems to be working out quite well. In Varnish 3.0 one thread would service each client, doing whatever that client wanted it to do. Within reason, obviously. These are very decent threads. The thread would deliver from cache, fetch content from the backend, pipe, etc.

    April 14, 2014

    Yves HwangFrontend/Senior Software Developer wanted in our London or Oslo office

    Exciting open source company seeks Frontend/Senior Software Developer in our London or Oslo office.

    April 10, 2014

    Hrafnhildur SmaradottirVarnish Cache 4.0 is released!

    Varnish Cache 4.0 is out! The Varnish team is thrilled and has just celebrated with a sip of bubbly. We are happy to announce this release to our community and the world. It includes some awesome enhancements for all of you to enjoy!

    April 09, 2014

    Ruben RomeroVarnish 4.0 Q&A on Performance, VMODs, SSL, IMS, SWR and more...

    During our "What's coming in Varnish 4.0?" Hangout (see the video now) two weeks ago we got some questions. I am getting back to you with some answers.

    MacYvesBuilding vagent2 for Varnish Cache 4.0.0 beta 1 for OS X 10.9.2

    For those keen bunnies that wishes to jump in and help us test out varnish cache 4.0.0 Beta 1 with varnish-agent 2, here’s how you do it on OS X 10.9.2 Mavericks.

    Prerequisites

    Homebrew dependencies

    Install the following with Homebrew

    • automake 1.14.1
    • libtool 2.4.2
    • pkg-config 0.28
    • pcre 8.34
    • libmicrohttpd 0.9.34

    Build varnish cache 4.0.0 beta 1

    1. Download and extract varnish cache 4 https://repo.varnish-cache.org/source/varnish-4.0.0-beta1.tar.gz
    2. run ./autogen.sh
    3. run ./configure
    4. make

    Build varnish-agent2 for varnish cache 4.0.0 beta 1

    1. Clone varnish-agent from repo https://github.com/varnish/vagent2
    2. Checkout the varnish-4.0-experimental branch
    3. export VARNISHAPI_CFLAGS=-I/tmp/varnish/varnish-4.0.0-beta1/include
    4. export VARNISHAPI_LIBS="-L/tmp/varnish/varnish-4.0.0-beta1/lib/libvarnishapi/.libs -lvarnishapi"
    5. run ./autogen.sh
    6. run ./configure
    7. make

    Note that if you run make install for varnish cache 4 or varnish-agent, it would then install it for you respectively.


    Per BuerLogging in Varnish 4.0

    The logging in Varnish Cache is one of the unique features of Varnish that in my mind sets it apart from the rest of the software world. It combines both logging in great detail with both performance, manageability, sensible privacy defaults and access to great debugging help. The shared memory log, logs everything that happens without the need to adjust log levels or without significantly affecting performance, eliminating the need for turning on any debug switch. Many of you have surely been in a situation where you have an application that is misbehaving. Then you increase the log level to debug and the problem magically disappears - obviously caused by the application slowing down just enough that the race condition you encountered earlier is now gone. The weakness of logging everything is that so much information is available that the administrator can sometimes be overwhelmed by all the information presented. It’s a figurative firehose of information and drinking from it can be painfull.  Martin has reimplemented a new logging framework in Varnish Cache 4.0. Out of all the new stuff in Varnish Cache 4.0 this might be the most significant one. It’s also the most complex feature, requiring quite a bit of time to fully understand how it works.

    Per BuerLocking down Varnish 4.0 - high security installations of Varnish

    Varnish Cache 4.0 is just around the corner. We did a beta release a couple of weeks back and the feedback has been pretty good and I think the release will be out later this week. Anyways, there are a lot of canges in Varnish 4.0.  A couple of the changes in Varnish Cache 4.0 are security related ones, giving you the option to lock down a Varnish installation to make it ultra secure.

    April 01, 2014

    Per BuerVarnish Cache 4.0 beta 1 is out

    The beta is out. Varnish Cache 4.0 is just around the corner and we need a bit of help to cross the finishing line. The beta seems to be pretty solid, we've given it a fair beating in out testing facilities as well as some pretty rough production workload. The final production tests last week where done in cooperation with A-Media and Redpill Linpro and where quite encouraging. Varnish 4.0 ran stable on their workload happily chugging away 1300 requests a second. If you have the option of giving Varnish Cache 4.0 a spin on your website, or giving it parts of your traffic we would be grateful. Crashes are very unlikely at this point, but just to be safe, we would recommend having a load balancer in front of the beta so you can gradually increase the load. If you need any help with converting the VCL, please let us know.

    March 28, 2014

    Per BuerWhen communities feel betrayed

    This week the Internet has been all up in arms about Face acquiring Oculus Rift. Many people have been angry about Facebook buying this company, something we did not see when Facebook acquired WhatsApp or Instagram.

    March 25, 2014

    Yves HwangVCL change management and contiguous integration with your Varnish using VAC API and Git

    Managing multiple Varnish instances and their respective VCL is made significantly easier with Varnish Administration Console (VAC) through its API. This blog post aims to illustrate an example of VCL change management and continuous integration with multiple Varnish instances using VAC API and a little magic from Git.

    March 24, 2014

    Ruben RomeroJoin our Varnish 4.0 Hangout on Wednesday

    Two days to go for our Varnish 4.0 Hangout with the Varnish Core Developer team. Make sure you join us. RSVP now!

    March 19, 2014

    Yves HwangVAC 3

    Software releases are not won by any single effort, but they are by processes, persistence, quality, and commitment. Most of all, solid teamwork. A release constitutes a bundling of various Git commits, neatly cherry picked to form a release candidate. From code changes to test automation, to build pipelines, the piece of software is scrutinised closely at every step.

    March 17, 2014

    Ruben RomeroVarnish Core Devs tell you what is coming in Varnish 4.0 (w/Q&A)

    Hangout On Air with PHK, Per, Martin & Lasse. We are now eagerly preparing for the release of Varnish 4.0. The release is planned for Q2 and to give you a sneak preview of what is coming we have scheduled a Google Hangout-on-Air session with Varnish’s Chief Architect, Poul-Henning Kamp, Varnish Software’s Founder and CTO, Per Buer, as well as our key Varnish Developers, Martin Blix-Grydeland and Lasse Karstensen.

    March 13, 2014

    Yves HwangReal-time statistics API with VAC

    Varnish Administration Console (VAC) is broken up into two major components. Its Management UI, and a RESTful API, with the UI utilising the API. There are a diverse number of features supported by the API, for example, change management, statistics and etc. This blog post aims to showcase the Real-time Statistics API.

    March 07, 2014

    Hrafnhildur SmaradottirUsing Varnish Tools to visualize your most popular content

    We just love it when our customers use our products in innovative ways that really help them create business value. One of the most recent examples of such use is how our customer vg.no. (Norway’s most popular internet news provider) is using Varnish Tools in combination with D3 to visualize their most popular online content.

    February 21, 2014

    ZenikaCreating a Varnish 4 module

    Don't let the title fool you, Varnish 4 hasn't been released yet. However, the master branch (which holds Varnish 4 developments) in its current state should be close enough to what the first release will look like. I've been bitten by API changes several times, but that's the deal when writing code against work in progress[1].

    Note

    [1] Last tested against the revision d0c0ee9

    What's new in Varnish 4 ? I will simply copy/paste an answer from Varnish Software (blame my laziness on this one): The 4.0 release is underway and should be released after the summer, with a prerelease during the summer, hopefully. The main new features are: New logging framework (see this blog post... Read Creating a Varnish 4 module

    February 11, 2014

    Yves HwangAdding Varnish Cache related statistics to your APM

    This blog post aims to introduce AppDynamics Varnish Extension for gathering cache related statistics. At the core of AppDynamics Varnish Extensions is varnishstat data that are fed back to AppDynamics Pro via varnish-agent in order to provide key statistics for a complete performance overview of your web stack.

    February 04, 2014

    Yves HwangVarnish Hashninja explained

    Recently we have had a few questions about the Varnish Hashninja VMOD, and how this all ties into Varnish Plus. This blog post aims to demystify the ever-so-elusive Hashninja, and help the readers find the best fit for their cache invalidation solution.

    January 16, 2014

    Per BuerIntroducing Varnish Plus

    This month we’re launching Varnish Plus. The rationale behind Varnish Plus is to combine all our services and products in one offering, market it under one name. So, when you become our customer, you get access to everything you need in one place.

    January 13, 2014

    Per BuerAre you a C programmer in Oslo?

    We're looking for a C-programmer in Oslo to work on Varnish and related stuff.

    Hrafnhildur SmaradottirVarnish’s role in the NYTimes.com tech platform redesign

    The NYTimes.com recently did a redesign of their website. Or rather, they did a complete reinvention of the website’s entire digital platform. In a blog post on their Open blog, they introduce the technology they used in the redesign. The objective with the redesign was to have a faster running code base and to make future changes easier.

    December 19, 2013

    Lasse KarstensenConverting a Varnish 3.0 VMOD to 4.0

    So we’re getting closer to releasing the first proper 4.0 version of Varnish Cache. One of the things we need to fix is to get all the vmod writers to make sure their vmod works with the new version.

    Here are my notes from doing just that, in the hope to make it simpler for others.

    In 4.0, you don’t need the source tree of Varnish any more. The include files will be enough, and pkg-config will find them for you.

    Make sure that /usr/lib/pkgconfig/varnishapi.pc and /usr/share/aclocal/varnish.m4 exists. If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source. (I need multiple varnishd versions when debugging customer problems, so I let them live in /opt/varnishX.Y/ on my laptop)

    Pull/merge the new Makefile.am files from the master branch of libvmod-example.

    Header files: remove bin/varnishd/cache.h and add cache/cache.h.

    Vmod functions are now called with a vrt context as first argument. %s/struct sess \*sp/const struct vrt_ctx \*ctx/g

    The old sess struct has been split, some data is in vrt_ctx->req, and some is in vrt_vtx->req->sp. Look up what is where in cache/cache.h. 

    I’ve put up the 3.0->4.0 diff for vmod_policy.c as a gist: https://gist.github.com/lkarsten/8039861

    There was a bit trouble of finding varnishtest, as src/Makefile was missing the reference entirely. I just fixed it by hand for now. Another thing for the 4.0 todolist, then.

    And finally; 

    lkarsten@immer:~/work/libvmod-policy/src$ make check
    /opt/varnish/bin/varnishtest -Dvarnishd=/opt/varnish/sbin/varnishd -Dvmod_topbuild=/home/lkarsten/work/libvmod-policy tests/test01.vtc
    # top TEST tests/test01.vtc passed (1.574)

     

    I have a working Varnish 4.0 vmod. :-D


    December 11, 2013

    Lasse KarstensenDNS RBL test address for development

    If you are writing code that checks a DNS real-time blockhole list (RBL), it looks like 127.0.0.2 is the standard address that is always in the black/white -list.

    This is probably know for most sysadmins/security people and whatnot, but wasn’t entirely trivial to find using Google.

    lkarsten@immer:~$ dig 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
    ; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55083
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
    ;; QUESTION SECTION:
    ;2.0.0.127.dnsbl.sorbs.net. IN A
    ;; ANSWER SECTION:
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.10
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.5
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.7
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.2
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.3
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.9
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.14
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.4
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.6
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.8
    ;; Query time: 17 msec
    ;; SERVER: 8.8.8.8#53(8.8.8.8)
    ;; WHEN: Wed Dec 11 14:12:20 2013
    ;; MSG SIZE rcvd: 203
    lkarsten@immer:~$

    Good to be able to actually test your code for hits also.

    (this is for libvmod-policy, so you can deny/reject POST/PUT from spammers in Varnish)


    November 29, 2013

    Per Buer404 handling in Varnish Cache

    When you have lots and lots of data, it’s hard to update it in a timely manner. Sometimes all you need is a bit of time while the systems in place are updating. A typical situation is the one where you have two backends serving the same data. However, one of them might lag behind the other one. When a request comes in to the one that is lagging it will throw a 404 Not Found. The solution is simple to describe; when you get a 404 try the other server.