Planet Varnish

August 13, 2010

Kristian Lyngstøl — Stripping cookies with VCL

The easiest way to get Varnish to help you out of the box, is to strip cookies you don’t want. Other than that, the default settings are pretty good.

So to ease that whole mess, I wrote a bash script for the occasion, maintained as of now at http://varnish-cache.org/browser/trunk/varnish-tools/cookie-stripper. It can create everything you need for a VCL file to just remove cookies and define a web server. A copy of the current version is here:

#!/bin/sh
# Author: Kristian Lyngstol
# Date: August 2010
# More info: http://kristianlyng.wordpress.com
# License: Consider it public domain.
#

usage() {
	echo
	echo "Varnish Cookie-strip VCL generator"
	echo "Copyright (C) 2010 Kristian Lyngstol "
	echo
	echo "$0 [-b backend:port] [-i indentation] [-c cookie] [-c cookie] ..."
	echo "Generates VCL for Varnish to strip cookies."
	echo " -c \tStrip cookie named \`cookie'. Can be"
	echo "            \tspecified multiple times"
	echo " -b \tAlso generate the backend definition to use"
	echo "               \ta web server at host:port as the backend. Both"
	echo "               \thost AND port, separated by colon, must be specified"
	echo " -t \tUse \`indent' when indenting the code, instead of"
	echo "            \t4 spaces."
	echo " -i         \tCreate case-insensitive cookie matching"
	echo
	echo "Example: $0 -b localhost:8181 -c __utmz -c __utma > /etc/varnish/default.vcl"
	echo
	echo "The VCL generated is for Varnish 2.1 and beyond and with the"
	echo "-b option there is no need to add or modify the final VCL to"
	echo "make Varnish \"just work\""
}
ind() {
	echo -n "$indent"
}

TEMP=`getopt -o hb:c:it: -n $0 -- "$@"`

if [ $? != 0 ] ; then echo "$(usage)" >&2 ; exit 1 ; fi

eval set -- "$TEMP"

icase=""
indent="    "
backend=""
cookiestring=""

while true ; do
	case "$1" in
		-b) backend="$2"; shift 2;;
		-i) icase="(?i)"; shift 1;;
		-t) indent="$2"; shift 2;;
		-c) cookiestrip="$cookiestrip $2"; shift 2;;
		-h) usage; exit 0 ;;
		--) shift; break ;;
		*) echo "Internal error! See $0 -h" ; exit 1 ;;
	esac
done

host=`echo $backend | sed s/:.*//`
port=`echo $backend | sed s/.*://`

if [ ! -z "$backend" ]; then
	if [ -z "$host" ] || [ -z "$port" ]; then
		echo "Invalid backend \"$backend\"" 1>&2
		echo "$(usage)" 1>&2
		exit 1;
	fi

	echo "backend default {"
	ind
	echo ".host = \"$host\";"
	ind
	echo ".port = \"$port\";"
	echo "}"
	echo
fi

if [ -z "$cookiestrip" ] && [ -z "$backend" ]; then
	echo "No -c or -b option specified." 1>&2
	echo "$(usage)" 1>&2
	exit 2;
fi

echo "sub vcl_recv {"
for a in $cookiestrip; do
	ind
	echo "# Remove cookie $a"
	ind
	echo -n 'set req.http.cookie = regsub(req.http.cookie,';
	echo -n "\"${icase}$a=[^;]*;?( |$)\""
	echo ',"");'
done

echo
ind
echo "# Remove the cookie header if it's empty after cleanup"
ind
echo 'if (req.http.cookie ~ "^ *$") {'
ind
ind
echo 'remove req.http.cookie;'
ind
echo '}'
echo '}'

Since I don’t have any cookie-needing web sites or servers, I haven’t tested it properly. I figure someone else will test it if it’s a worthwhile script to have around.

Feedback welcome.


August 02, 2010

Kristian Lyngstøl — Varnish backend selection through DNS

A common challenge to using a cache is maintaining a mapping between public site names and actual web servers (backends). If you only have one type of web server (or maybe two?), and it’s fairly static, this isn’t a big deal. However, if your infrastructure spans tens of different types of web servers, then it starts getting iffy. Here’s an example of how this could look:

director sports round-robin {
    { .backend = { .host = "sports1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "sports2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "sports3.internal.example.net"; .port = "80"; } }
}
director shop round-robin {
    { .backend = { .host = "shop1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "shop2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "shop3.internal.example.net"; .port = "80"; } }
}
director economy round-robin {
    { .backend = { .host = "economy1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "economy2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "economy3.internal.example.net"; .port = "80"; } }
}
director main round-robin {
    { .backend = { .host = "main1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "main2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "main3.internal.example.net"; .port = "80"; } }
}
sub vcl_fetch {
    if (req.http.host ~ "sports.example.net$") {
        set req.backend = sports;
    } elsif (req.http.host ~ "shop.example.net$") {
        set req.backend = shop;
    } elsif (req.http.host ~ "economy.example.net$") {
        set req.backend = economy;
    } else {
        set req.backend = main;
    }
}

This is obviously a bit of a drag, and so far we only added four sites.

Enter the DNS director

The DNS director allows you to define a single director containing one or more backends, just like any other backend director, but uses DNS to decide which one to pick. Simply put, it does a DNS lookup on the Host header and sees if it has a backend that matches.

Notice that it does NOT automatically try whatever IP the Host header resolves to. It has to know about the backend in advance. This might sound like a bit of a major flaw, but I choose to look at it as a safety net.

The DNS director also allow you to add a postfix to the host-name before it is looked up, so www.example.com could become www.example.com.internal.example.net. It has rudimentary DNS round-robin support and caches the DNS lookups (both successful lookups and misses). Since there isn’t a practical way of obtaining the TTL of a DNS result except apparently hand-coding the resolver or possibly adding some obscure dependency, the life-time of the DNS cache is defined by a setting in the director, cleverly named .ttl.

As a last added bonus, I also added a really easy way to shoot yourself in the leg. With the DNS director, you can specify a range of backends using .list and a acl-like syntax. However, remember that adding 10.0.0.0/8 means Varnish will internally generate 16 million backends. That’s PROBABLY not a good idea. So do use some moderation. A /24 or two shouldn’t be a big deal, but I’d try to narrow it down as much as possible.

Here’s the above example, except using FOO.example.net.internal.example.net instead of FOO<N>.internal.example.net, and assuming that the web servers are all in the 192.168.0.0/24 range or 172.16.0.0/24

director mydir dns {
    .list = {
        .port = "80";
        .connection_timeout = 0.4;
        "192.168.0.0"/24;
        "172.16.0.0"/24;
    }
    .ttl = 5m;
    .suffix = "internal.example.net"
}
sub vcl_recv {
    set req.backend = mydir;
}

Specifying connection timeout and similar attributes is optional in .list, but has to be before the list of IPs. You do not have to use .list, you can also add backends the same way you would with the random or round-robin director.

The above examples caches the DNS results for 5 minutes. I’ve also added some counters (visible through varnishstat): Number of DNS lookups, DNS cache hits, failed DNS lookups and how often the DNS cache is full. You may still want to do some basic sanitizing of domain names so as to reduce DNS spam, but now you can probably just use one regsub to match a number of sites.

Availability

The DNS director was committed to Varnish development trunk yesterday (Sunday, August 1st 2010) and I expect it to be available in Varnish 2.1.4. It has already been used in production at a few customer sites, with good results. Like any non-trivial piece of code, there are certain aspects of it I want to improve, but I do not foresee that as a blocker for including it in a release. It does not affect the rest of Varnish at all if it not used (unless you count adding 4 counters to varnishstat).

If you want to test it, you’ll have to use Varnish trunk. Alternatively, you can check out my github repo which is currently sitting at Varnish 2.1.2 + DNS director + return(refresh). It does look like you’ll have to compile from source, though. Unless, of course, you’re a Varnish Software customer, then you just drop us a mail and you’ll get your rpms or .debs shortly. (My marketing hat is currently firmly planted on my head).

The development of this feature was sponsored by Globo and Mercado Libre and implemented by myself/Varnish Software.


July 30, 2010

Kristian Lyngstøl — Varnish 2.1.3 – Use it!

We just released Varnish 2.1.3. And it’s good.

I am a person who value stability, predictability and stability over pretty much everything else. I will gladly use 3-year old software over the latest version if it’s working well. I only upgrade my desktop if I see a really good reason to do it. I rather wait 2 years for a new package to enter my favorite distribution than grab it from a source package to get a new feature.

Why do I tell you this? Because I am now ready to truly whole-heartedly recommend Varnish 2.1.3. Varnish 2.0.6 was a great release. It was stable, it worked well, we know how to get the most of it. There were no un-knowns. When we released Varnish 2.1.0, we knew that it was going to take a release or two to get the 2.1 releases equally good. I finally believe we are there, and that using Varnish 2.1.3 is (almost) as safe as 2.0.6. This is the version I will be recommending to our customers.

Varnish 2.0.6 compared to 2.1

Varnish 2.1 represents two years of development. Roughly. The performance of Varnish 2.1.3 is roughly the same as that of Varnish 2.0.6, with a few exceptions.

First, we now use the “critbit” hashing algorithm instead of “classic” as default. This switch revealed a few weaknesses in the implementation that we gradually resolved between Varnish 2.1.0 and 2.1.3. The benefit of critbit is that it requires far less locking to deliver content. It scales better with large data sets and is generally a nice thing to have.

We have also re-factored much of the code that relates to directors, which allowed us to add multiple new directors. Including directors to pick a backend based on source-ip, URL hash etc.

With Varnish 2.1.3 we also added a “log” command to VCL, which allows you to add generic log messages to the SHM-log through VCL, a much-requested feature.

We have also added basic support for Range-headers. This is not the smartest version of it around, but it fits into the KISS-approach of Varnish. When Range-support is enabled, Varnish will fetch the entire object upon a Range request, but deliver only the range that the client requested. This allows Varnish to cache the entire object, but deliver it in smaller segments.

An other important change between Varnish 2.0 and 2.1 is the removal of the object workspace. The most immediate effect of this is that you will have to write “beresp” instead of “obj” in the vcl_fetch part of your VCL. The bigger consequence is that you no longer have a obj_workspace parameter and all work previously done in obj_workspace is now done in sess_workspace, then Varnish allocates exactly as much space as it needs for the object once it’s finished. This should save you some memory on large data sets.

Now, there are several other changes, but most of them are internal. This is partly to make way for persistent storage, and also for general house-keeping. An other important reason why the perceived difference between Varnish 2.0.6 and 2.1 is not that big is that many of the features that were written for Varnish 2.1 were ported to Varnish 2.0. This includes new purging mechanisms, saint mode, resets in vcl_error and numerous bug fixes.

What’s next?

We are still working on persistent storage. It is available in Varnish 2.1 as an experimental feature, but it is missing certain key aspects – like LRU support. You can compare this to how critbit evolved: Critbit was available in Varnish 2.0, but not stable. We used the 2.0 release to fine tune critbit, and we will use 2.1 to improve persistent storage.

For Varnish 2.1.4, I will be merging the DNS director, which has been ready for some time now. I wanted to investigate some reports of memory leaks before I merged it, and those seem to be debunked now.

I will also be merging my return(refresh) code, which is fairly simple stuff. All it does is “guarantee” a cache miss, even if there is valid content in the cache. The use-case for this is when you update content and want to control who does the initial waiting. The typical example is when your front page updates, you send a script to it with a magic header (X-Refresh: Yes, for instance), then you look for that in VCL and make sure the client is coming from an allowed IP (if (client.ip ~ purgers), for example) and issue return(refresh), which will (oddly enough) refresh the content. Your clients wont have to wait and the front-page is updated immediately.

In the longer run, we are also looking at proper support for gzip. For the uninitiated, it should be emphasized that for normal operation, Varnish doesn’t need to support gzip. Normally, Varnish will simply forward the Accept-Encoding header to the web server, which will compress the content as it sees fit and return it with a Vary header. That way, Varnish can deliver compressed content without having to compress it itself. This works fine, until you introduce Edge Side Includes (ESI) into the mix. With ESI, Varnish has to parse the content returned to check for ESI commands, and it can’t do that if the content is compressed. So today, Varnish only supports uncompressed ESI. We wish to solve that. Properly.

I am sure I have forgotten some key elements, but this should hopefully be enough to make this a worth-while read.


July 28, 2010

Dr Carter — Varnish 2.1.3 released

<script type="text/javascript"> </script> <script src="http://pagead2.googlesyndication.com/pagead/show_ads.js" type="text/javascript"> </script>

From varnish-announce@varnish-cache.org :

”
We are pleased to announce Varnish 2.1.3, a bug fix and feature release in the Varnish 2.1 series.
The most notable changes are:

  • The scalability of critbit, the default hashing method, has been improved
  • A bug in varnishd would in some cases confuse varnishncsa leading to lost or wrong log lines.
  • Some bugs in the handling of Range requests has been fixed. This only matters if you enable Range support.
  • Add «log» command to VCL which will log to the Varnish log.

This is a summary of the changes, please see the changelog for a fuller list.

The release can be downloaded from Sourceforge, as usual: http://sourceforge.net/projects/varnish/
“

Kristian Lyngstøl — Smart bans with Varnish

Banning – or what is commonly referred to as purging – means adding an expression to the ban list. I wrote an introduction to purging a while back: http://kristianlyng.wordpress.com/2010/02/02/varnish-purges/, and today I’ll introduce you to the ban lurker.

First, let me apologize for the name confusion. We’ve realized that calling it “purging” seems to indicate that the objects are removed from the cache. As they are not, it’s more reasonable to call it banning – and that’s what Varnish calls it internally. As far as VCL and CLI goes, it’s still called purging.

A quick reminder on how Varnish bans objects

Varnish bans objects by adding an expression to a list, and then when an object is “hit”, it is tested against all expressions on the list since the last time it was hit. So only when an object is hit can it be evaluated.

You can ban on both req.* and obj.* items. When you ban in VCL, there are THREE contexts involved:

1. The VCL context that puts the ban on the ban-list.

2. The context of the next request that hits the object

3. The context of the object

This means that a purge for:

purge ("req.url == " req.url " && obj.http.magic == " req.http.purge-magic)

all three purges are used. The “req.url” outside the quotations marks (the second one) is in the context of the VCL that triggered this purge() function. The “req.url” inside the quotation marks is in the context of the request that will hit the object next – whatever that may be. The “obj.http.magic” variable will be in the context of the actual object on the cache, while the last “req.http.purge-magic” is in the current vcl-context.

If that was confusing, it’s ok. If it wasn’t confusing, I’m impressed.

Enter the ban-lurker

So the problem with Varnish bans is that not all objects are hit constantly. In fact, some objects are rarely ever hit. And they will linger even if they are banned. To solve this, the ban-lurker was created. Put simply, the ban-lurker will check older objects against the current ban-list to see if they are worth keeping around. This serves two functions: 1. Banned objects can be discarded. 2. The size ban-list can be reduced.

Now, the problem with the ban-lurker is that it doesn’t stem from a request. In other words, it doesn’t HAVE a “req.” structure. It doesn’t have a URL. So if your bans are for (“req.url == ” req.url), the ban lurker can’t help you, because it doesn’t have a req.url to test against. It can ONLY test against obj.

As most bans are for req.url, this means that the ban-lurker isn’t very useful out-of-the-box. Unless you adapt your approach to bans a bit.

Writing ban-lurker-friendly bans

To utilize the ban-lurker, there are two things you need to do:

1. Turn it on with “param.set ban_lurker_sleep 0.1″ (for example. Any non-zero value enables it)

2. Do not use req.* in your bans.

Now, how to avoid using req? Well, the simples method is probably to store req.url on the object, and ban on that. Example:

sub vcl_fetch {
    set obj.http.x-url = req.url;
}
sub vcl_deliver {
    unset resp.http.x-url; # Optional
}
sub vcl_recv {
    if (req.request == "PURGE") {
          if (client.ip !~ purgers) {
              error 401 "Not allowed";
           }
          purge("obj.http.x-url ~ " req.url); # Assumes req.url is a regex. This might be a bit too simple
    }
}

The above code-snippet simply stores the URL on the object as a HTTP header called x-url, then uses that instead of req.url to ban. It also removes the x-url header before it’s returned to the clients. It may not look as intuitive (… because “req.url == ” req.url is clearly intuitive), but it allows you to take advantage of the ban-lurker.

Conclusion

We (Varnish Software and the general Varnish community) don’t know how useful the ban-lurker will prove to be, nor do we know what the best grace period for the ban_lurker_sleep parameter is. This is all fairly new, and not used much. We’re confident in it’s stability, but not the benefit.

However, it’s clear that if you have a possibly large data-set and you do frequent purges, using the ban-lurker can help keep the size of the ban-list within manageable limits, as well as help you throw banned content out faster. We can’t really see the downside of it.

However, using the ban-lurker without adjusting your approach to actual bans is pointless. It wont be able to do anything except wake up every second and discover it can’t do anything.

I hope this helped. Happing banning!


July 26, 2010

Dr Carter — Another way to link Varnish and MaxMind GeoIP

In the tutorial http://varnish-cache.org/wiki/GeoipUsingInlineC, you can see one way to do.
But there is another way, with no need to compile a geoip_plugin.so.

Follow these steps:

  • Go on http://www.maxmind.com/app/c
  • Download http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
  • Download the last database http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
  • Install GeoIp:
    • ./configure
    • make
    • make check
    • make install
  • Update your database: GeoIp.dat (/usr/local/share/GeoIP/GeoIP.dat)
  • Edit your /etc/sysconfig/varnish :
    • add : 
      -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -L/usr/local/lib -lGeoIP -lmemcached -o %o %s' \
    • or if you don’t need memcached: 
      -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/local/lib -lGeoIP -o %o %s' \
  • Add the following lines in your vcl 
    C{

 #include <GeoIP.h>

  static GeoIP *gi = NULL;

  const char*
  get_country_code(const char* ip)
  {
        const char* country = NULL;

        if (gi == NULL)
                gi = GeoIP_new(GEOIP_STANDARD);

        if (ip != NULL)
                country = GeoIP_country_code_by_addr(gi, ip);

        return country ? country : "Unknown";
  }

}C

    C{
  VRT_SetHdr(sp, HDR_REQ, "\017X-Country-Code:", (*get_country_code)( VRT_IP_string(sp, VRT_r_server_ip(sp))), vrt_magic_string_end);
}C

Now you have the country code in a new header: X-Country-Code

July 22, 2010

Dr Carter — How to obtain Time Taken with varnishncsa ?



There is a way to add the time taken to serve the request in the varnishncsa logs.

You must use the version 2.1.2 of Varnish.
Then apply the following patch http://varnish-cache.org/ticket/712 , you can download here.


In order to use the new option, launch the varnishncsa with the command line:

varnishncsa -L "%h %l %u %t \"%{VarnishR}i\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D"

%D refer to “The time taken to serve the request, in microseconds”.


Here a list of options:

  • %H, Protocol version
  • %{Host}i
  • %{Referer}i
  • %U, URL path
  • %q, the query string
  • %U%q, URL path and query string
  • %{User-agent}i
  • %{X-Forwarded-For}i
  • %b, Bytes
  • %h (host name / IP adress)
  • %m, Request method
  • %s, Status
  • %t, Date and time
  • %u, Remote user
  • %D, The time taken to serve the request, in microseconds
  • %{X-Cache-Age}i Age of served object

I don’t know if the patch is official.
Be careful on production, i’ve some segfault:

kernel: varnishncsa[29466]: segfault at 0000000000000000 rip 000000368a2797e0 rsp 00007fff0348dbf8 error 4

July 15, 2010

Kristian Lyngstøl — A Varnish Crash Course for aspiring sysadmins

Ok, so you’ve just started working in a Big Sysadmin Department, and they use Varnish. You may have to deal with it, but are not going to be working on it on a daily basis. This is some of what you should know about varnish in case you have to extinguish some fires.

This post is heavily inspired by my visit to Paris last week, and as such, I’d like to say hello to Ludo and everyone else (this is mostly written on my plane ride back home).

1. What Varnish is, and how it fits into the web

Varnish is a reverse proxy. It operates on layer 7, unlike your typical load balancer (though some of them have layer 7 functionality, like basic health checks, they can hardly compare to Varnish). For all intents and purposes, Varnish is the first “web server” that a browser hits.

Varnish has a simple, yet powerful configuration language – the Varnish Configuration Language (VCL) – which allows it to make intelligent decisions on what to cache, and how. It can rewrite urls, it can perform redirects and it can direct traffic at different web servers based on both the specific request it is dealing with (for example, if the request is for “http://www.example.com/sports”, it might go to a dedicated sport-server) and on the state of the web server.

By default, Varnish should Just Work. It will only cache content it is certain can be safely cached. That means that it will only cache GET requests, and only content where no cookies are involved. This behavior can – and most likely will – be overridden by your VCL.

In regards to the RFC2616 (The HTTP RFC), Varnish is not really what they are talking about when they refer to proxies and caches. Varnish belongs on the origin-server side of the equation, as the same people who control the web server can also control the cache. But not entirely, so you we have done some approximations using common sense to make Varnish fit into the RFC.

2. Configuration, stopping and starting.

Varnish has two separate types of configuration types. VCL defines policy, and only policy. It can be used to implement features that are otherwise not present (for example the ability to purge content simply by adding “?purge=yes” to the url, or any other scheme like that).

The second type of configuration is program arguments – or parameters. Parameters relate to how Varnish behaves on the machine it is running it. You can define the size of the cache, the user and group to run Varnish under, what ports are used by Varnish, where varnish will find its VCL, and so forth.

On Debian-based systems (including Ubuntu), you will typically find the parameters defined in /etc/default/varnish. Similarly, you will find them in /etc/sysconfig/varnish on Red Hat.

The VCL is often stored in /etc/varnish/, and typically /etc/varnish/default.vcl – but that is defined by a startup argument.

Additionally, you can change MOST of the parameters while Varnish is running. There is only requirement for this: Varnish needs to be started with a -T argument, to enable the management interface.

The best way to use the management interface is by telnetting to it. This gives you a nice and simple interface. With a “help” command. You can run the same commands with the “varnishadm” tool, but that will not give you as extensive error messages. The rule of thumb that I use is “telnet for interactive commands, varnishadm for scripting purposes”.

Parameters are changed by using “param.set” after reviewing them with “param.show”. Changes are applied immediately, but it might take some time for them to be visible. A good example of this delay is if you change the default ttl, as that applies to new objects in the cache from that point on.

You can also load and use new VCL while Varnish is running. This does not introduce any (known) glitches or slowdowns or delays in Varnish, as it is a matter of switching 4 pointers around – and varnish will still keep the old configurations around while they are still used (for instance if a backend is still in use).

Loading VCL is done with the “vcl.load ” command. That will only _load_ the configuration, not actually use it. If your VCL has syntax errors, this is when they will show up. After it has loaded, you can switch to the new config with “vcl.use “. Alternatively use my varnish_reload script. Be very careful with “reload” arguments to init scripts, as they may be implemented using a restart.

Starting Varnish is left as an exercise to the reader.

3. Quickly getting an overview of Varnish

varnishstat.

Run it, look at it, and understand it. This previous blog post gives a quick introduction to varnishstat:  http://kristianlyng.wordpress.com/2009/12/08/varnishstat-for-dummies/.

One thing you should always check is the uptime of varnish – in varnishstat. Because Varnish will “never” crash permanently. It can – however – crash repeatedly. This is because Varnish has two processes: A management process and a “everything else” process. If you have a bug, it is likely to take down the “everything else” process, and the management process will notice this and restart it immediately. This is logged to syslog in detail.

On Debian systems (including Ubuntu), /var/log/syslog is more extensive than plain /var/log/messages.

You also want to look out for disk activity. This is the number one killer of Varnish-performance. If you are seeing extensive disk activity, it might make sense to reduce the size of the cache so it fits in memory. And tune the caching scheme to make sure there are no duplicate objects (for instance a www.example.com/foo object and a separate example.com/foo object). Or buy more memory. Or make sure the disk is an SSD. (Hint: vmstat 5).

One trick that is commonly applied to avoid disk activity is to put the shmlog on a tmpfs. This is generally not required, but doesn’t hurt. The shmlog and the compiled VCL is typically stored at /usr/var/varnish/(hostname)/ or similar.

Do not let the virtual memory usage of Varnish scare you. But do be afraid of the resident memory usage. It is not uncommon to see Varnish use 80G of virtual memory but only 28G resident on a machine with 32G.

Lastly you have varnishlog. This will tell you exactly what varnish is doing with each request, but is extremely extensive. Keep in mind that varnishlog can filter. Here are some freebies, and I’ll leave it to the reader to expand upon them (or a future blog post):

varnishlog -o RxUrl /some/url
List requests with “/some/url” in the url. Including /some/url, /some/url/blah and /blah/some/url/bla.

varnishlog -b -o
Only list backend traffic.

varnishlog -b -o -i TxURL
List URLS going to a backend.

varnishlog -c -o RxHeader FireFox
List requests from clients with “FireFox” present in any of the HTTP headers supplied.

varnishlog -o TxStatus 500
List all requests sent back to a client with status code 500.

4. Clear part of the cache

Let’s say your web server misbehaved and your VCL wasn’t smart enough to spot it, and you’ve cached malformed data. The easiest way to clean that up is using the “purge” function through the management interface. It will not free up memory, but it will make sure the content is refreshed the next time someone asks for it.
This supports regular expressions similar to what you find in VCL. I wrote a detailed post about purging: http://kristianlyng.wordpress.com/2010/02/02/varnish-purges/

Some highlights:

purge req.url == /some/specific/url && req.http.host == “www.example.com”
purge req.url ~ “^/some/generic/prefix” && req.http.host ~ “^(www\.)?example\.(com|org|net)”
purge obj.http.cache-control ~ “max-age=2222222222222″

5. Final advice

Don’t use 32 bit.

Read my post on best practices: http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/

Don’t over do it. KISS is king. Standard OS packages, as simple a VCL as you can, as little tuning of parameters as you can. Etc.

Remember: If Varnish is told not to cache based on response from a web server, it will cache the decision not to cache. This means that if your front page temporarily fails and varnish is told not to cache it, make sure the TTL is low, otherwise all traffic to the front page will go to a web server until the hitpass object has expired.

Ask for advice early. It’s much harder to fix a system that is deployed and in production than help you avoid problems. You can get commercial help from us at Varnish Software or the Varnish community.


July 09, 2010

Kristian Lyngstøl — Announcing varnish_gather

Varnish_gather is a small script I’ve written (just now) to gather various information regarding varnish to make it easier to send possibly relevant information to the people who can help you with varnish.

The official location is: http://bohemians.org/~kristian/varnish/varnish_gather . The usage is simple: Download it, make it executable, then run it. It does not require root privileges, and supplying the admin port is optional.

./varnish_gather > varnish_info.log

./varnish_gather localhost:6082 > varnish_info2.log

Only intended for Debian/Ubuntu and Red Hat variants of GNU/Linux for now. Patches are welcome.

That’s it.

For reference, heres a copy of the version available when I post this:

#!/bin/sh
# varnish_gather - Gather debug information for varnish issues
# Copyright (C) 2010 Kristian Lyngstol
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
#
#
# Mostly harmless.

if [ -z "$1" ]; then
	echo 1>&2 "WARNING: Without a hostname:port for the admin interface, this script is less useful"
	echo 1>&2 "Typical usage: ./varnish_gather localhost:6082"
	sleep 5;
fi

ITEM=0

banner()
{
	echo "--------------------------------"
	echo "Item $ITEM: $*"
	echo
	ITEM=$(( $ITEM + 1 ))
}

mycat()
{
	if [ -r $1 ]; then
		banner "File: $1"
		cat $1
	fi
}

banner dmesg
dmesg

banner logs
mycat /var/log/dmesg

for a in /var/log/messages /var/log/syslog; do
	if [ -r $a ]; then
		banner "Filtered log: $a"
		grep varnish $a
	fi
done

mycat /proc/cpuinfo
mycat /proc/version

banner Memory usage
free -m

banner Vmstat 5x5
vmstat 5 5

banner Process scan
ps aux | egrep '(varnish|apache|mysql|nginx)'

banner varnishstat
varnishstat -1

if [ ! -z "$2" ]; then
	banner "varnishstat for $2"
	varnishstat -1 -n $2;
fi

banner mount
mount

banner netstat
netstat -nlpt

banner netstat2
netstat -np

mycat /etc/default/varnish
mycat /etc/sysconfig/varnish

banner "Local files"
find /usr/local -name varnish

if [ ! -z $1 ]; then
	banner "Varnishadm tinkering"
	varnishadm -T $1 vcl.list
	varnishadm -T $1 vcl.show boot
	varnishadm -T $1 param.show
	varnishadm -T $1 purge.list
else
	banner "NO ADMINPORT SUPPLIED"
fi

banner "End"


Dr Carter — Updating Anti-Scraping Captcha



Today, we spotted some IP located in the same area that have won the captcha challenge within a very short time.

So we decided to change the system of captcha.

We stop with the phpcaptcha system, now we use the reCaptcha of google.
The system seems to be stronger.

July 06, 2010

Dr Carter — Varnish VCL to patch IIS6 Compression



In IIS6, static compression is happening on a separate thread. So on receiving a request, first response is uncompressed and IIS used to start a separate thread to compress the file and keep it in compressed files cache.

Problem, Varnish keeps the uncompressed version of the file.

So I’ve added few line to my vcl_fetch:

#patch compression IIS6 (first demand is not compressed)
 if (req.http.Accept-Encoding) {
    if (!(obj.http.Content-Encoding)) {
       set obj.ttl = 0s;
    }
 }

Be careful to remove Accept-Encoding for binaries files, otherwise images can’t be cached:

if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

June 15, 2010

Per Buer — Going to Velocity 2010?

Tollef Fog Heen and myself will be visiting Velocity next week. If you would like to meet and discuss Varnish in any way, please get in touch with us. So, if you are looking for help scaling your web site, a partnership or you're just plain curious, please send me an email to perbu [at] varnish-software [dot] com.

June 04, 2010

Per Buer — Profit models, part 5 - Proprietary on certain platforms

There seems to a lot of ways to monetize open source. The last post covered open core - where you have some add on component that is more or less proprietary. Today In our quest to cover them all it's time to turn to the somewhat exotic. I actually haven't seen any projects doing this. If you know of any, please leave a comment.

Varnish is tied to Unix and POSIX. Some of the design decisions made makes porting Varnish to Windows really hard. There are, however, other considerations in addition to the purely technical ones. 

The Varnish Community

We get a lot of help from our community. I believe this is in part due to the open source culture. You're supposed to be a good citizen and help out. Our users are good at debugging and they operate in an environment where debugging tools are easily available. So, requests for core dumps, stack traces, system call traces and log excerpts are fullfilled more often then not.

So, what is the situation on Windows? For starters the tools to produce decent debugging information is

  • not part of the operating system
  • not easily available, on FreeBSD and Linux you can install a debugger with a single command
  • not something the average system administrator is trained in using

So, I'm speculating that Windows users won't contribute in the same way as FreeBSD and Linux users do. If we decide to port Varnish to Windows, it not at all certain we'd get a the same support from the community.

Case in point; MySQL on Windows and Linux

Lets try to verify this in some way. I did a search in the MySQL bugzilla for bugs in the InnoDB storage engine. One search on Linux and one on Windows. Guess what, there are twice as many bugs reported on Linux as there is on Windows. This even the majority of MySQL users (50%) use MySQL on Windows. It might be because MySQL runs much smoother on Windows, but I doubt that.

I think it is because Linux sysadmins are better community players then Windows sysadmins. It's in their blood. The same goes for FreeBSD, of course.

So, if this is true it undermines the rationale behind porting Varnish to Windows. The port will in itself be expensive and the maintenance cost will be higher. So what should we do? Well, we could make the port proprietary. Varnish could have a proprietary cousin, Lacquer, that only runs on Windows. It would contain the same code base, but with a giant windows patch maintained out of tree. In order to get access to it you would have to pay and you would be supported in much the same way that any other proprietary product, except we would of course have better service. :-)

I can certainly see Lacquer being popular. It would make give a lot of Sharepoint administrators an viable option accelerating Sharepoint in their Windows only environment. Right now pure Windows environments that want web accelerators within their firewalls are left with the old, expensive and slow alternatives. 

Let me underline one thing. This is not something we have planned to do. This is pure speculation on my side. It's just published here to help others starting up a business on open source a few more options. Then again, if there is demand.... Please, tell me what you think in the comments.

The picture is (c) 2006 Darwin Bell and can be found here.

May 26, 2010

Ingvar Hagelund — Accelerating the Internet (or actually, Squid) with Varnish

Squid is an old, working-horse, caching proxy server that can be configurated to act as a reverse proxy. Varnish is the opposite, it’s an extremely fast http accellerator that’s configurated to be, well, just that. So I thought, just for the fun of it, what about configurating Varnish to cache the Internet for me, that is, use it as a general forwarding caching proxy server.

Obviously, we can’t define varnish backends for the entire world. But Squid can do that. So I used our corporate Squid proxy, and put a local varnish cache in front of it. The vcl is very simple:

backend default {
  # This is squidbox
  .host = "11.22.33.44";
  .port = "3128";
}

That’s it, actually. Start up varnish, and use that varnish instance’s address and http port as proxy in your web browser.

Then, using an ugly little perl script “proxytest” for testing, we found these quite interesting results:

$ for i in "" squidbox:3128 varnishbox:6081; do ./proxytest $i http://www.slashdot.org/ 10; done
1.0836112s
0.3773585s
0.0352446s

Lesson learned: Varnish is some 10 times faster than Squid, when caching the Internet!

With thanks to eric for playing with settings.

May 24, 2010

Per Buer — Profit models, part 4 - Spring Source

When up Varnish Software I went through a number of open source companies looking at how their business. This has resultet in a series of blog posts describing different business models that work well with open source. My last post described Red Hat and how they have managed to build their company on pure open source.

Today I would like to write about Spring Source. I think Spring Source is an very exciting company. They started out with something I haven't seen that much in the open source world. The Spring Framework, which is some sort of library full of useful stuff for writing lightweight Java applications. The strange part is to sell support agreements. I just cant see anyone actually buy support for an API. I can see how one could build a business on an proprietary API, or even how one on a dual licenced product like Trolltech did with Qt. But who would really like to buy support for a freely available library?

 

So I guess Spring didn't sell too many support subscriptions for its library, but Spring Framework did wonders for the Spring brand. So, they had this brand and a lot of good people. What do you do? They did several thing. They acquired Hyperic, which I think was a brilliant investment but somewhat off topic to this post.

Most users who would want to deploy an application based on Spring Framework would want to deploy it on Tomcat. So, Spring made TC Server. TC Server is a version of the Apache Tomcat Servlet engine with some proprietary manageability added on top. From my understanding TC Server contains all the Tomcat code and then attached some extras. It's a 100% compatible with Tomcat. The extras makes it easier to manage and debug Tomcat. If you want to deploy Tomcat in an enterprise environment, what you really want is TC Server. You also get support and indemnification with TC Server.

So, Spring does write most of the code that is going into the Tomcat repository. This gives them the knowledge necessary and the creditability to deliver support. In addition the proprietary add-ons actually add value without limiting the value of Tomcat.  You can, and most people do, run stock Tomcat and it will run just as fast and just as stable as TC Server. But TC Server will reduce the hassle out of running it in a big scale operation. I love the way Spring has balanced this. The difficulty, as with all open source, is to give value to your paying customers without taking something away from your community.

This model is described as "open core". Other companies such as Sugarcrm are doing the same thing, although I think Sugar has gone somewhat far in limiting their community version. When we wanted CRM software for Varnish Software we couldn't use Sugarcrm CE - we had to use the Pro version. The CE version lacks a lot of what we thought was essential business functionality. I guess this limits Sugarcrms adoption - but on the other hand it probably increases their conversion ratio.

I think this model makes a lot of sense for Varnish Software. We would be able to distribute Varnish Cache under a very liberal licence. The Varnish Administration Console, our tool for helping you manage a complex Varnish installation will add value to those that have it without taking value from the D.I.Y. or command line crowd.

What do you think of this? Would you rather see the Enterprise/Community paradigm win over the open core?

May 19, 2010

Dr Carter — Something weird with varnishncsa


Our blocking srapers system have reported us a IP which browse continuously one of our website.

So I have done a grep on the varnishncsa log, but i don’t see it in realtime.
I’ve done a grep on the all day varnishncsa log, i’ve seen:

91.88.187.67 - - [19/May/2010:00:58:14 +0200] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
91.88.187.67 - - [19/May/2010:00:39:13 +0200] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
91.88.187.67 - - [19/May/2010:01:25:15 +0200] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
91.88.187.67 - - [19/May/2010:04:02:24 +0200] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
91.88.187.67 - - [19/May/2010:04:28:14 +0200] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"

When i do varnishncsa in realtime with a varnishncsa | grep “/0_2″, i can see the backend traffic:

server1 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server2 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server1 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server2 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server1 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server1 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server2 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
server3 - - [00/Jan/1900:00:00:00 +0000] "POST http://www.ouest-france.fr/0_2 HTTP/1.1" (null) - "http://www.ouest-france.fr/0_2?page_ref=/0_4" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"
...
...

Varnishlog contains:

115 SessionClose c pipe
115 ReqStart     c 91.88.187.67 4436 1089132167
115 RxRequest    c POST
115 RxURL        c /0_2
115 RxProtocol   c HTTP/1.1
115 RxHeader     c x-requested-with: XMLHttpRequest
115 RxHeader     c Accept-Language: fr
115 RxHeader     c Referer: http://www.ouest-france.fr/0_2?page_ref=/0_4
115 RxHeader     c Accept: application/xml, text/xml, */*
115 RxHeader     c Content-Type: application/x-www-form-urlencoded
115 RxHeader     c x-requested-handler: ajax
115 RxHeader     c Accept-Encoding: gzip, deflate
115 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; ficeLiveConnector.1.3; OfficeLivePatch.0.0)
115 RxHeader     c Host: www.ouest-france.fr
115 RxHeader     c Content-Length: 13
115 RxHeader     c Connection: Keep-Alive
115 RxHeader     c Cache-Control: no-cache
115 RxHeader     c Cookie: crm_cookieEnabled=1; __utmc=88432901; xtvrn=$61164$; __utma=88432901.1379090940.1266326763.1274179082.1274180897.682; __utmz=88432901.1274169829.676.57.utmcsr=ofmnewsletter|utmccn=une|utmcmd=lettredinformation
115 VCL_call     c recv
115 VCL_return   c pipe
115 VCL_call     c pipe
115 VCL_return   c pipe
209 BackendOpen  b server1 192.120.1.182 35846 192.120.1.68 80
115 Backend      c 209 director_of server1
209 TxRequest    b POST
209 TxURL        b /0_2
209 TxProtocol   b HTTP/1.1
209 TxHeader     b x-requested-with: XMLHttpRequest
209 TxHeader     b Accept-Language: fr
209 TxHeader     b Referer: http://www.ouest-france.fr/0_2?page_ref=/0_4
209 TxHeader     b Accept: application/xml, text/xml, */*
209 TxHeader     b Content-Type: application/x-www-form-urlencoded
209 TxHeader     b x-requested-handler: ajax
209 TxHeader     b Accept-Encoding: gzip, deflate
209 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.4; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)
209 TxHeader     b Host: www.ouest-france.fr
209 TxHeader     b Content-Length: 13
209 TxHeader     b Connection: Keep-Alive
209 TxHeader     b Cache-Control: no-cache
209 TxHeader     b Cookie: crm_cookieEnabled=1; __utmc=88432901; xtvrn=$61164$; __utma=88432901.1379090940.1266326763.1274179082.1274180897.682; __utmz=88432901.1274169829.676.57.utmcsr=ofmnewsletter|utmccn=une|utmcmd=lettredinformation
209 TxHeader     b X-Forwarded-For: 91.88.187.67
209 TxHeader     b X-Varnish: 1089132167
209 TxHeader     b X-Forwarded-For: 91.88.187.67
209 TxHeader     b whitelisted: 0
209 TxHeader     b X-Scraping: 1
209 TxHeader     b X-Hostname: 67.187.88-91.rev.gaoland.net
209 BackendClose b server1
115 ReqEnd       c 1089132167 1274251253.994797468 1274251254.165796518 0.000020742 0.001982212 0.169016838
115 StatSess     c 91.88.187.67 4436 0 1 1 1 0 0 948 0

I’ve searched on google about something like that. What I found is a problem on a box from a french ISP which loop on an url.

But i don’t understand why i can’t see all the client traffic with varnishncsa. Is it due to the “pipe” command ?
I don’t understand why there isn’t http status in the varnishncsa for the few lines i’ve captured.

May 18, 2010

Dr Carter — Production test of blocking scraping system


Today is a big day ! This is the first test of our blocking scraping system on production.
We have put it on two of our three varnish.

For a first test we don’t want to block scrapers but just log them.

So we wrote a light version of the script.

# Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA
# All rights reserved.
#
# Author: Vincent ROBERT <v.robert@of2m.fr>
# Thanks to: Tony FOUCHARD <t.fouchard@of2m.fr>
# Thanks to: Cyrille MAHIEUX <c.mahieux@of2m.fr>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

C{
  #define HOSTNAME_MAX_SIZE 200
  #include <string.h>
  #include <libmemcached/memcached.h>
  #include <syslog.h>
  #include <netdb.h>
  #include <sys/types.h>
  #include <sys/socket.h>
}C

sub check_ip {
  set bereq.http.whitelisted = "0";
  if (bereq.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css|swf|xml|flv)(\?(.)*)?$") {
     set bereq.http.whitelisted = "1";
  }
  else {
     if (bereq.http.host == "internalwebSite") {
       set bereq.http.whitelisted = "1";
     }
     elsif ((bereq.http.X-Forwarded-For == "IPaddress1") || (bereq.http.X-Forwarded-For == "IPaddress2")) {
       set bereq.http.whitelisted = "1";
     }
  }

  if (bereq.http.whitelisted=="0") {
    C{

      void ip_to_hostname(char *ipaddr, char *hostname)
      {
        struct addrinfo * result;
        int successfull=0;
        int error;

        error = getaddrinfo(ipaddr, NULL, NULL, &result);
        if (error==0) {
           error = getnameinfo(result->ai_addr, result->ai_addrlen, hostname, HOSTNAME_MAX_SIZE, NULL, 0, 0);
           if (error==0) {
             successfull=1;
           }
        }

        freeaddrinfo(result);

        if (!successfull){
          memcpy(hostname, ipaddr, HOSTNAME_MAX_SIZE*sizeof(char));
          hostname[HOSTNAME_MAX_SIZE-1]=0;
        }
      }

      memcached_server_st *servers = NULL;
      memcached_st *memc;
      memcached_return rc;
      char hostname[HOSTNAME_MAX_SIZE]="";

      char key[500]= "varnish_";
      strcat(key, VRT_GetHdr(sp, HDR_REQ, "\005host:"));
      strcat(key, "_");
      strcat(key, VRT_IP_string(sp, VRT_r_client_ip(sp)));

      memc= memcached_create(NULL);
      servers= memcached_server_list_append(servers, "memcacheServer", 11211, &rc);
      rc= memcached_server_push(memc, servers);

      if (rc == MEMCACHED_SUCCESS) {

        uint64_t intval;
        rc= memcached_increment(memc, key, strlen(key), (uint64_t)1, &intval);

        if (rc != MEMCACHED_SUCCESS)
          rc= memcached_set(memc, key, strlen(key), "1", 1, (time_t)60, (uint32_t)0);
        else
          if (intval>30) {
            ip_to_hostname(VRT_IP_string(sp, VRT_r_client_ip(sp)), hostname);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Scraping:", "1", vrt_magic_string_end);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Hostname:", hostname, vrt_magic_string_end);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-HostOrig:", VRT_GetHdr(sp, HDR_REQ, "\005host:"), vrt_magic_string_end);
            if (intval<300)
              rc= memcached_set(memc, key, strlen(key), "500", 3, (time_t)600, (uint32_t)0);
          }
      }
      memcached_free(memc);
      memcached_server_list_free(servers);
    }C
    if (bereq.http.X-Scraping=="1") {
      if (!(bereq.http.X-Hostname ~ "(\.(yahoo\.(net|com))|(exabot\.com)|(googlebot\.com)|(search\.msn\.com))$")) {
        C{
          syslog(LOG_INFO, "Scraping detected from %s %s on %s with %s",VRT_IP_string(sp, VRT_r_client_ip(sp)), VRT_GetHdr(sp, HDR_BEREQ, "\013X-Hostname:"), VRT_GetHdr(sp, HDR_REQ, "\005host:"), VRT_GetHdr(sp, HDR_REQ, "\013User-Agent:"));
        }C
      }
    }
  }
}

sub vcl_miss {
  call check_ip;
  fetch;
}

sub vcl_pipe {
  call check_ip;
  pipe;
}

sub vcl_pass {
  call check_ip;
  pass;
}

The load is about 600 requests/s on each Varnish server.
We put just one memcached server.

The result are:

  • No CPU overhead on Varnish or too small to be viewed
  • No memory leak spotted.
  • Memcached server is almost idle: about 60 requests/s and 80 KBytes used !
  • Turn off the memcached server don’t affect Varnish and user experience
  • Good news, the system show us a lot of scrappers and many spiders
  • Bad news, we have detected one or two proxy as scrappers, but they are not.

Here, a piece of our logs :

Jun  9 10:26:40 ouest-proxy03 varnishd[4015]: Scraping detected from 94.23.240.216 ns209194.ovh.net on www.ouest-france.fr with posh
Jun  9 10:29:28 ouest-proxy03 varnishd[4015]: Scraping detected from 94.23.198.123 ns207043.ovh.net on www.ouest-france.fr with Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Jun  9 10:29:55 ouest-proxy03 varnishd[4015]: Scraping detected from 69.191.249.201 injr-spdrproxy1.bloomberg.com on www.ouest-france.fr with BLP_bbot/0.1
Jun  9 10:29:59 ouest-proxy03 varnishd[4015]: Scraping detected from 89.122.29.79 adsl89-122-29-79.romtelecom.net on www.ouestfrance-emploi.com with Java/1.6.0_04
Using phpMemCacheAdmin, you can see memcache server is fine:




Here the mk-query-digest profiling of memcache server TCPdump:

 ./mk-query-digest --type memcached --group-by res --limit 10 dump_zym.log

# 69.3s user time, 440ms system time, 83.70M rss, 171.79M vsz
# Current date: Tue May 18 12:17:09 2010
# Files: dump_zym.log
# Overall: 70.68k total, 3 unique, 67.20 QPS, 0.01x concurrency __________
#                    total     min     max     avg     95%  stddev  median
# Exec time             9s    20us    22ms   122us   224us   233us    98us
# Time range        2010-05-18 11:45:40.061323 to 2010-05-18 12:03:11.908228
# bytes             17.63k       0       3    0.26    0.99    0.43       0
#  74% (53k)  Memc_incr
#  25% (18k)  Memc_miss
#  25% (18k)  Memc_set

# Profile
# Rank Query ID Response time    Calls R/Call   Item
# ==== ======== ================ ===== ======== =========
#    1 0x           4.9508 57.3% 34673   0.0001
#    2 0x           2.4583 28.5% 17995   0.0001 NOT_FOUND
#    3 0x           1.2310 14.2% 18015   0.0001 STORED
Here the top of one of the Varnish:

top - 16:38:02 up 57 days,  5:09,  3 users,  load average: 0.36, 0.54, 0.54
Tasks: 104 total,   1 running, 103 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.0%us,  1.1%sy,  0.1%ni, 93.2%id,  0.0%wa,  0.3%hi,  0.3%si,  0.0%st
Mem:  33030932k total, 29536176k used,  3494756k free,   781740k buffers
Swap: 11727408k total,     3792k used, 11723616k free,   727416k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2016 varnish   20   0 35.1g  26g  81m S   38 84.2 122:50.26 varnishd
24769 root      20   0 98.8m  80m  80m S    1  0.3   0:03.26 varnishncsa
Here the varnishstat of one of our Varnish:

0+07:03:59                                                                                                                                                                      ouest-proxy02.sdv.fr
Hitrate ratio:       10      100     1000
Hitrate avg:     0.7814   0.7972   0.8006

     3693144       103.00       145.18 Client connections accepted
    13991870       556.00       550.02 Client requests received
    11372520       469.00       447.05 Cache hits
       14376         1.00         0.57 Cache hits for pass
     2546655        85.00       100.11 Cache misses
     2622997        87.00       103.11 Backend conn. success
          41         0.00         0.00 Backend conn. not attempted
         507         0.00         0.02 Backend conn. failures
          96         0.00         0.00 Backend conn. reuses
         996         0.00         0.04 Backend conn. was closed
        1093         0.00         0.04 Backend conn. recycles
          36         0.00         0.00 Fetch head
     2338109        71.00        91.91 Fetch with Length
      226850        15.00         8.92 Fetch chunked
         932         0.00         0.04 Fetch wanted close
        1540          .            .   N struct sess_mem
         691          .            .   N struct sess
     2151242          .            .   N struct object
      877521          .            .   N struct objecthead
          56          .            .   N struct vbe_conn
         341          .            .   N struct bereq
         144          .            .   N worker threads
        1474         0.00         0.06 N worker threads created
           0         0.00         0.00 N queued work requests
       19144         0.00         0.75 N overflowed work requests
          29          .            .   N backends
         549          .            .   N expired objects
      395693          .            .   N LRU nuked objects
      345902          .            .   N LRU moved objects
          23         0.00         0.00 HTTP header overflows
    10527830       435.00       413.85 Objects sent with write
          12         0.00         0.00 Objects overflowing workspace
     3693079       113.00       145.17 Total Sessions
    13991789       557.00       550.01 Total Requests
       57104         2.00         2.24 Total pipe
       19567         0.00         0.77 Total pass
     2565891        90.00       100.86 Total fetch
  5115095581    210266.25    201072.98 Total header bytes
 71922504943   2823420.83   2827253.62 Total body bytes
      779458        22.00        30.64 Session Closed
         407         0.00         0.02 Session Pipeline
         317         0.00         0.01 Session Read Ahead
    13466511       535.00       529.36 Session Linger
    12094595       508.00       475.44 Session herd
   680468640     26550.16     26749.03 SHM records
    40799330      1602.01      1603.81 SHM writes
       52523         2.00         2.06 SHM flushes due to overflow
       64353         1.00         2.53 SHM MTX contention
         292         0.00         0.01 SHM cycles through buffer
     5476474       264.00       215.28 SMA allocator requests
     4256875          .            .   SMA outstanding allocations
 32212244810          .            .   SMA outstanding bytes
 63522967274          .            .   SMA bytes allocated
 31310722464          .            .   SMA bytes free
        1494         0.00         0.06 SMS allocator requests
      776444          .            .   SMS bytes allocated
      776444          .            .   SMS bytes freed
     2566011        86.00       100.87 Backend requests made
           4         0.00         0.00 N vcl total
           4         0.00         0.00 N vcl available
           1          .            .   N total active purges
           1         0.00         0.00 N new purges added
Next step: deploy on our last server, and really blocking !

May 14, 2010

Per Buer — Profit models, part 3 - Red Hat

When we started up Varnish Software, we had a look at a few profit models for Open Source. We've are presenting these in a series of blog posts. This is the third post in the series, covering Red Hat and how they generate revenue.

The Company

The company we're looking at Red Hat. Red Hat is the biggest pure open source company today, with revenues nearing a billion dollars. A small fish compared to Microsoft and Oracle, but still a good sized company.

Red Hat was the first company to create a profitable business on Open Source. It did so in a time where everybody understood that Open Source was important but had no idea on how to generate revenue.

Back in 2002 Red Hat had "Red Hat Linux", a Linux distribution which was free. If I recall you could buy a boxed set for $50 or download it for free. The box came with 30 day support. Not a very successful company.

Red Hat Enterprise Linux

So, Red Hat thought long and hard and came out with their Enterprise software. They had some issues with naming in the beginning, but settled on Red Hat Enterprise Linux after a while. As the name indicates. they also focused solely on the Enterprise. All they cared about was the data center. They did one thing and by all standards they have done a very good job.

So, how does RHEL relate to open source? The source code for RHEL is freely available. But the source code isn't really very useful. Building RHEL from source would probably take months, even if you know what you are doing. Binaries aren't made available to the public, you have to be a paying customer to get to the binaries. 

Limiting distribution and creating scarcity

All of the software is licensed under free licences, but Red Hat put trademarked material in with the software, inhibiting distribution and creating the scarcity needed to charge for their core product. For all practical purposes, Red Hat Enterprise Linux, is a proprietary system. It's built on open source, but as you can't distribute it, it's hardly free software.

Quality

When you limit the distribution your product won't receive the same amount of testing it would get if it was free. When MySQL tried to duplicate Red Hats model the quality of their software dropped considerably. The reason was obvious - the user base dropped to about 1% of what it was earlier. Bummer.

Now, if you can compensate for this with proper Q/A and rigid testing the problem goes away. Red Hat has managed to do this but there is a cost. If you're a small company, like Varnish Software is, the relative cost might be to high.

Confusion

Developers like free software. Easy to download and install. A lot of developers that develop for Red Hats platforms use the free versions of the software. These are, however, not always compatible with the Enterprise versions of the software. This hit JBoss particularly hard, where both the community version and the enterprise version are under the same brand, more or less. A lot of developers where put in considerable pain when having to migrate their newly developed application from the community edition of JBoss to the enterprise edition - which lags behind by a couple of years.

A clear message

Red Hats message is clear. They offer the best server operating system out there. And it costs money. 

May 07, 2010

Per Buer — Profit models, part 2 - Wordpress

Before starting up Varnish Software I went through a number of open source companies looking at how they are able to generate revenue. I thought I'd share what I've found. My last post described Mozilla Firefox and how they generate revenue through Google.

Wordpress Logo

This next one is also my favorite. Wordpress surfaced in 2003. At the time the blogosphere was dominated by Movable Type, blog software from Six Apart. Movable Type was free for personal use. After some success they started restricting their users more and more, pushing their paid version of the software. At the same time Wordpress was free software and its use exploded, leaving Movable Type in the dust.

Six Apart responded to the success of Wordpress by relicencing Moveable Type under a free licence. But it was a bit too late. As Google Trends shows, Wordpress overtook Moveable Type in 2004 and is approximately ten times bigger today. Furthermore, I have never seen a proprietary project switch to a free project and actually have success, but that is content for another post.

So how does Wordpress generate its revenue? If you were to start a blog, would you go through all the fuzz with installing Linux, MySQL, Apache and the blog software and messing with backup, hosting and all the other stuff? Well, if you a practicing or a recovering sysadmin (like me) you might, otherwise you'll just head over to wordpress.com and register an account. You might even sign up to their premium service, giving you extra support and features. If you don't, Wordpress will show a few ads on your blog, generating the few pennies necessary the run the service.

This is it. Wordpress gives away the software running their platform. Users and developers download it, test it and improve it. They also blog, write and talk about it, helping build the Wordpress brand and thereby paving the way for the wordpress.com service. This last part is really the key. The marketing power of thousands of enthusiastic users is worth humongous amounts of money.

How is this relevant for Varnish Software? Would you consider buying Varnish Cache from Varnish Software as a hosted service? I think I would. Would you? 

May 05, 2010

Ingvar Hagelund — RPM packages of varnish-2.1.2

Varnish is a high performance web accelerator. For more information about varnish, see the project homepage.

I had just added varnish-2.1.1 to rawhide when the upstream team released a bugfix update varnish-2.1.2.

Rawhide is updated. I also built packages for RHEL5. You can download them at http://users.linpro.no/ingvar/varnish/2.1.2/, or from SourceForge.

Per Buer — Profit models for open source, part 1 - Firefox

In my quest to find a profit model for Varnish Software we took a look at how other projects manage to generate revenue. My favorite was Firefox. Firefox is one of the most successful open source projects ever. It's has had a huge success, both in terms of marketshare and revenue generated. It has managed to get huge penetration in the main stream. An impressive feat.

firefox logo

Firefox is a reasonably new project, with its 1.0 release about six years ago. Due to almost perfect timing and really good quality it's adoption grew in a huge way. 

Today Mozilla Foundation has revenues of about 100 million USD. So how does Firefox generate revenue? First and foremost its the little box in the upper right corner. It sends traffic to Google, then Google shows a few ads, sometimes people click the ads and then Mozilla Corporation gets a share of the revenue. Brilliant, isn't it?

Some of the most success open source projects make money with open source - not from open source.

Let me know if you have any suggestions. Input on how Varnish Software could generate revenue with its product is very welcome.

Dr Carter — Varnish 2.1.2 released

From varnish-announce@varnish-cache.org :

”

Hot on the heels of 2.1.1, Varnish 2.1.2 is now released.


It fixes a regression introduced with Range support which was added in 2.1.1 whereby large objects would get junk appended to them. This would not show up in browser due to the Content-Length header, but confused at least some load balancers leading to very strange errors.


The release can be downloaded from Sourceforge, as usual: http://sourceforge.net/projects/varnish/

“

May 04, 2010

Dr Carter — HEADS-UP: Bug in Varnish 2.1.1 may append junk to objects

From varnish-announce@varnish-cache.org :



”

Thanks to Audun Ytterdal, I just fixed a really boneheaded bug I introduced in Varnish 2.1.1


This will warrant a 2.1.2 release, but because that typically takes a week or so, I am sending this heads-up to the varnish-announce list, so those of you who are affected can take evasive action in the meantime.


If you backend sends chunked encoding (typically only CGI processing) and delivers objects larger than 128k, you may hit this bug.


To fix this for good, you can apply the one-line patch attached to the commit message below.


As a workaround, you can increase the “fetch_chunksize” parameter to be bigger than any object your backend may deliver with chunked encoding.  Be aware that this will affect storage usage negatively.


Telling your backend to not use chunked encoding (if possible) is also a water-tight workaround.


My apologies,


Poul-Henning

“

May 02, 2010

Per Buer — Choosing a profit model for open source

Choosing a profit model for an open source business is hard. For a proprietary software model its quite easy - you create scarcity by limiting the distribution of your software and then you sell licenses.

For open source its a bit different. First and foremost, there is no scarcity, so you can't really charge for the software itself. So, you have to find some other way of making the money to pay your developers. I'll try to list a few things you need to consider when choosing a profit model for an open source project.

There is a community. It's not yours. The community is there. They have their own needs and their own agenda. If you bring them an elegant way to organize themselves they can realize their own agenda.

 

Make sure your profit model does not annoy the community. http://www.flickr.com/photos/teban32/547351984/

 

Allow others to innovate. By relinquishing control over your project or at least provide a platform for innovation within the project (plug-ins, etc) others will lift the project to higher levels. 100 people can usually innovate better than 2.

Realize you can't make money of everything you do. Make sure your whole organization understands this. I once had the displeasure of meeting the regional sales director of a major open source business. I believe his words where "You have to induce fear into the community". His sales organization went around and scared people with the viral effects of the GPL. They forced a proprietary licensed version of the software onto their customers. His destructive effect on users, developers, partners and potential clients was really bad.

Over the next few posts I'll present the different profit models we considered when starting Varnish Software. I would very much like your input in the comments below.

 

The image can be found here and is licened under a Creative Commons licence. (c) 2007 _Teb

April 26, 2010

cd34 — DDOS attack mitigation

Today we had a DDOS attack on one of our clients. They were running prefork with mod_php5 with a rather busy application. While we initially started filtering IP addresses using iptables and a few crudely hacked rules, we knew something had to be done that was a little more permanent. Moving to MPM-Worker with PHP served with FastCGI seemed reasonable, but, looking at the history of the attacks on this machine, I believe Apache still would have been vulnerable since we cannot filter the requests early enough in Apache’s request handler.

Apache does have the ability to fight some DDOS attacks using mod_security and mod_evasive, but, this particular attack was designed to affect apache prior to the place where these modules hook into the request. This also precludes using fail2ban. We could run mod_forensic or mod_logio to assist fail2ban, but, it is still a stopgap measure.

We could have used some Snort rules and tied those to iptables, but, that is a rather bad solution to the problem.

While we could have used Varnish, their application would have had some issues. mod_rpaf can help by adjusting the REMOTE_ADDR to take the value from X-Forwarded-For that Varnish sets. mod_geoip actually inserts itself before mod_rpaf, so, we would have needed to make a modification to mod_geoip and recompiled it. I’m not sure how Varnish would have handled Slowloris and we had to fix this now.

Putting them behind a Layer 7 load balancer would have isolated the origin server and handled the brunt of the attack on the load balancer, but, again we would have needed mod_rpaf and some modifications to their code.

In the end, Lighttpd and Nginx appeared to be the only documented solution. After the conversion, we did find documentation that said Varnish and Squid were immune to Slowloris. With Nginx or Lighttpd, we didn’t have IP address issues to contend with, it would be easy enough to modify the fastcgi config to pass the GEOIP information in the same request variable that their application expected. We knew we had to run PHP under FastCGI, so, we might as well pick a tool where we can block the attack in the webserver without having to worry about firewall rules. We did put a few firewall rules in place to block the larger offenders.

in the http { } section of our nginx config, we added:

    client_body_timeout 10;
    client_header_timeout 10;
    keepalive_timeout 10;
    send_timeout 10;
    limit_zone limit_per_ip $binary_remote_addr 16m;

and in the server { } section, we added:

    limit_conn limit_per_ip 5;

Since each server was expecting to handle one or two requests from each IP, this gave us a little headroom while solving the problem in the right place.

I believe Varnish would have held the connection and wouldn’t have sent a request to the backend which makes it fairly versatile as a tool to deal with DDOS attacks. While I do like the ability to block certain requests in VCL, the methods listed to fight this type of attack appeared to favor a non-threaded webserver. Varnish in front of Apache would have worked, but, we already knew we needed to move from Apache at some point with this client and this gave us an opportunity to shift them while under the gun.

Wouldn’t have had it any other way.

April 24, 2010

Kristian Lyngstøl — Using an OpenPGP Smartcard on Ubuntu 10.04

We recently bought a few OpenPGP SmartCard version 2.0 at Varnish Software, with card readers to match, and I’ve been using mine for a month or so now. However, there are a few challenges involved, specially with the PCMCIA-based Omnikey CM4040 card reader. Until today, mine has been flaky at best.

The use case for this is to store your ssh key on a smart card, along with your encryption and signature key.

In addition to hardware, the version 2.0 of the OpenPGP card requires gnupg 1.4.10 or gnupg 2.0.10 or newer, which is not necessarily easily available. It is available on Ubuntu 10.04, however.

I’ve tried to describe the necessary steps, though I might have forgotten some elements of it. That being said, this isn’t meant to be a recipe for people who don’t know what they are doing, it’s intended for those of you who understand what’s actually being done. Let me know if anything crucial is missing.

Setting up the software

You will need gnupg2, gnupg-agent, pcscd and libpcsclite1 to begin with. If you are using th Omnikey CM4040, you will most likely also need pcsc-omnikey. The software stack works by having gpg2 or ssh talk to the gpg-agent (which will also act as an ssh-agent), this in turn will spawn scdaemon which talks to pcscd which talks to the actual card, if my understanding is correct.

apt-get install gnupg2 gnupg-agent pcscd libpcsclite1

To be able to use the ssh bit, you have to remove the ssh-agent from /etc/X11/Xsession.d/, simply copy it out of the way: cp /etc/X11/Xsession.d/90×11-common_ssh-agent /etc/X11/ . Now edit /etc/X11/Xsession.d/90gpg-agent and find the STARTUP=”$GPGAGENT –daemon …” line and add –enable-ssh-support. After that, you’ll be using gpg as your ssh agent when you next log in.

Setting up the Omnikey CM4040

To get the 4040 up and working, you need to install the pcsc-omnikey package if you didn’t already do that. Unfortunately, it doesn’t drop the required config in /etc/reader.conf.d/ as it’s supposed to, so grab it from the example doc, then regenerate /etc/reader.conf (which is just a collection of /etc/reader.conf.d/) and restart pcscd:


# cp /usr/share/doc/pcsc-omnikey/examples/reader.conf /etc/reader.conf.d/
# update-reader.conf
# service pcscd restart

If you had already tried gpg2 –card-status or similar, you will most likely have to kill scdaemon, and it’s fairly stubborn, so it typically takes 3-4 SIGTERMs before it dies.

After this, though, you should be all set up.

Trying the card

gpg2 --card-status

A couple of things can happen:

  1. You get card information and it’s all nice and dandy
  2. You get most of the card information, but things like vendor ID is blank or incorrect – upgrade gnupg.
  3. It hangs – kill scdaemon.
  4. You get a “no card found” error: remove and reinsert the card, restart pcscd and kill scdaemon, then pound your head into the wall if it still doesn’t change.
  5. You get a more generic error: see point 4.

To make sure the card isn’t caching, try removing it then typing gpg2 –card-status. You should obviously get an error, and if you don’t, it’s likely scdaemon that needs a kill. This typically happens if you remove the card reader or restart pcscd.

Assuming the card is showing up, it’s time to use it: gpg2 –card-edit

Setting up most of this should be easy, but there’s at least one point that needs an explanation: Your public key is _NOT_ stored on the card! After you’ve created a new key (or imported one), you will have to send it to a key server the normal way and set the url on the card to match. The reason you want to set the url is because it makes it far easier when you use your card on a different machine: All you do is run –edit-card and type ‘fetch’ and it will download your public keys.

The other thing you must remember, which you probably already know from reading the notes that came with your card: If you type the admin pin code wrong more than 3 times in a row, your card is essentially bust. Luckily, you rarely need the admin pin.

If you haven’t managed to set your pin yet, type help and read. To generate a new set of keys, simply type generate. You can also move the keys using normal gpg2 commands, but as I haven’t done that myself, I can’t rule out pit falls.

Once your keys are generated, that’s really all there is. You can add uids later like you would with any other gpg key (ie: gpg2 –edit-key Your-fpr-here).

Using ssh

This should already Just Work, but to verify that ssh can see your key, use: ssh-add -l. You can also add your old ssh keys to gpg that way. You’ll be prompted for pass phrase two times: The original pass phrase and the new one. See gpg-agents man pages for details.

Exporting the public key can either be done by ssh-copy-id,  or ssh-add -L.

Using GPG

You probably want to make sure you set up key trust for your key (gpg2 –edit-key ..;   trust) when you’re on a new computer, but besides that, it’ll be just like normal GPG. I’d post a pretty picture of the PIN-dialog, but it seems it grabs most of the screen, so that’s a bit difficult.

Hooking it up to PAM

I’ve had moderate success with libpam-poldi. I’m saying moderate because only one application can hold your card at any given time, which is fine as long as everything goes through gpg2-agent and scdaemon, but if you try to use your smartcard  for sudo, you’ll likely find it doesn’t do you much good. The poldi-package worked pretty much out of the box for me, after gluing it in to pam. It has fairly good instructions, so I’ll leave it as an exercise to the reader to actually set it up.

One interesting thing, though, is that if you hook it up to gnome-screen-saver, it’ll work flawlessly, and even take pin-cache into consideration. Login also works fine, as gpg-agent hasn’t started yet.


April 21, 2010

Dr Carter — Improvement of blocking web scraping solution with Varnish Memcached & inline C

In the precedent version of code there was no possibility to whitelist a client hostname.

For example, when googlebot crawl the website, it can use a lot of different Ip address. It’s too difficult to maintain a whitelist of ip address, we need to do a reverse dns to obtain the client hostname.

So I’ve added a reverse dns operation in case of detecting scraping. Once we’ve the client hostname, we can test it with a regular expression to decide if it’s in the whitelist or not.

Please note that the code is normaly ipv6 compatible but only tested with ipv4.

At the top of vcl:

# Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA
# All rights reserved.
#
# Author: Vincent ROBERT <v.robert@of2m.fr>
# Thanks to: Tony FOUCHARD <t.fouchard@of2m.fr>
# Thanks to: Cyrille MAHIEUX <c.mahieux@of2m.fr>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

C{
  #define HOSTNAME_MAX_SIZE 200
  #include <string.h>
  #include <libmemcached/memcached.h>
  #include <syslog.h>
  #include <netdb.h>
  #include <sys/types.h>
  #include <sys/socket.h>
}C

Here the new check_ip function:

sub check_ip {
  set bereq.http.whitelisted = "0";
  if (bereq.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css|swf|xml|flv)(\?(.)*)?$") {
     set bereq.http.whitelisted = "1";
  }
  else {
     if (bereq.http.host == "whitelisted website") {
       set bereq.http.whitelisted = "1";
     }
  }
  if (bereq.http.whitelisted=="0") {
    C{

      void ip_to_hostname(char *ipaddr, char *hostname)
      {
        struct addrinfo * result;
        int successfull=0;
        int error;

        error = getaddrinfo(ipaddr, NULL, NULL, &result);
        if (error==0) {
           error = getnameinfo(result->ai_addr, result->ai_addrlen, hostname, HOSTNAME_MAX_SIZE, NULL, 0, 0);
           if (error==0) {
             successfull=1;
           }
        }

        freeaddrinfo(result);

        if (!successfull){
          memcpy(hostname, ipaddr, HOSTNAME_MAX_SIZE*sizeof(char));
          hostname[HOSTNAME_MAX_SIZE-1]=0;
        }
      }

      memcached_server_st *servers = NULL;
      memcached_st *memc;
      memcached_return rc;
      char hostname[HOSTNAME_MAX_SIZE]="";

      char key[50]= "varnish_";
      strcat(key, VRT_IP_string(sp, VRT_r_client_ip(sp)));

      memc= memcached_create(NULL);
      servers= memcached_server_list_append(servers, "memcached server", 11211, &rc);
      rc= memcached_server_push(memc, servers);

      if (rc == MEMCACHED_SUCCESS) {

        uint64_t intval;
        rc= memcached_increment(memc, key, strlen(key), (uint64_t)1, &intval);

        if (rc != MEMCACHED_SUCCESS)
          rc= memcached_set(memc, key, strlen(key), "1", 1, (time_t)60, (uint32_t)0);
        else
          if (intval>30) {
            ip_to_hostname(VRT_IP_string(sp, VRT_r_client_ip(sp)), hostname);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Scraping:", "1", vrt_magic_string_end);
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Hostname:", hostname, vrt_magic_string_end);
            if (intval<300)
              rc= memcached_set(memc, key, strlen(key), "500", 3, (time_t)600, (uint32_t)0);
          }
      }
      memcached_free(memc);
      memcached_server_list_free(servers);
    }C
    if (bereq.http.X-Scraping=="1") {
      if (!(bereq.http.X-Hostname ~ "\.googlebot\.com$")) {
        C{
          syslog(LOG_INFO, "Scraping detected from %s %s",VRT_IP_string(sp, VRT_r_client_ip(sp)), VRT_GetHdr(sp, HDR_BEREQ, "\013X-Hostname:"));
        }C
        if ((bereq.url ~ "/securimage/securimage_show.php")) {
          set bereq.url = regsub(bereq.url, "^(.*)/securimage/securimage_show.php?(.*)", "/securimage/securimage_show.php?\2");
        }
        else {
          set bereq.http.referer = "http://" bereq.http.host bereq.url;
          set bereq.url = "/";
        }
        set bereq.http.host = "captcha websites";
        set req.backend = captcha server;
      }
    }
  }
}

sub vcl_miss {
  call check_ip;
  fetch;
}

sub vcl_pipe {
  call check_ip;
  pipe;
}

sub vcl_pass {
  call check_ip;
  pass;
}

April 19, 2010

Kristian Lyngstøl — Caching a Debian repository with Varnish

In the past I’ve both run my own debian mirror and used apt-cacher to reduce the amount of duplicate package downloads I do at home when I upgrade multiple computers (virtual or otherwise). Mirroring sort of defeated the purpose as I used far more bandwidth than I needed, and apt-cacher was not horribly robust. The reason I want this is twofold: 1. I’m a geek. 2. I often have 5-10 debian-based hosts (virtual+physical) at home.

So now I use Varnish instead.

Debian archives (in this context, debian and ubuntu are the same) are twofold: information about packages and the packages themself. Ie: “apt-get update” versus “apt-get install”. I cache “^/debian/.*\.deb$” for 21 days (random number) and everything else in “^/debian” for 12 hours.

I’ve set up repo.kristian.int to point to the web host, which means that if I for some reason don’t want to use my local Varnish cache, I can just point it to a real debian mirror and the clients wouldn’t notice the difference.

Just for the heck of it, Varnish is set up to use 50GB of -smalloc memory. It’s fun to have disk space.

Pros:

  • No maintenance needed – it’s just a HTTP cache.
  • Reduced bandwidth usage.
  • Faster local upgrades.

Cons:

  • No streaming delivery yet, so adds a delay if it’s cache miss. Since the final delivery is gbit, this is hardly a real issue. And since streaming delivery is on the todo-list for Varnish….
  • Restarting Varnish flushes the cache. I will be using persistence to solve this.

Vcl:

backend lo {
        .host = "127.0.0.1";
        .port = "8080";
}

backend debian {
        .host = "ftp.no.debian.org";
        .port = "80";
}

sub vcl_recv {
        if (req.url == "/purgeall") {
                purge("req.url ~ .*");
                error 200 "Purged all";
        }
        if (req.url ~ "^/debian/.*") {
                set req.backend = debian;
        } else {
                set req.backend = lo;
        }
}

sub vcl_fetch {
        if (req.url ~ "^/debian") {
                if (req.url ~ ".deb$") {
                        set beresp.ttl = 21d;
                } else {
                        set beresp.ttl = 12h;
                }
        } elsif (req.url ~ "^/slaughter") {
                set beresp.ttl = 1s;
        } elsif (req.url ~ "^/munin/") {
                set beresp.ttl = 30s;
        } else {
                set beresp.ttl = 10s;
                set beresp.cacheable = false;
        }
}


Notes on the vcl

The VCL is an unedited copy/paste of the actual VCL I use, and it’s running on an internal Varnish server. I’m not protecting things like /purgeall, which you should if you copy this.

Also note that I consistently fall through to the default VCL instead of trying to out-smart it. That’s how I recommend you write a VCL file, as the default VCL handles cookies, authorization headers and strange HTTP requests (ie: TRACE) in a sensible way, in addition to adding X-Forwarded-For logic.

Adding support for Ubuntu would just mean adding an ubuntu mirror and copying the logic for ^/debian to ^/ubuntu.


Kristian Lyngstøl — My blog just moved – again

New employer (varnish software), thus new blog. This time, I’ve gone for wordpress in the hopes that I wont have to move in two years. My first blog was at dev.beryl-project.org/…, then moved to dev.compiz-fusion.org/… then to kristian.blog.linpro.no…. No more moving around.

I’ve also imported the posts from both the compiz-blog and the linpro-one, and retagged some of the older untagged posts (ie: beryl/compiz-posts). Hopefully, it’s not too much of a mess.


April 16, 2010

Dr Carter — Blocking web scraping: cpu consumption improvement

In order to reduce cpu overhead, I deported memcache function from vcl_recv to vcl_miss, vcl_pipe and vcl_pass.
Indeed i don’t care about robots when they hit the cache. But I’m very embarrassed when backend have too many load.

Take a look on the scripts:

# Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA
# All rights reserved.
#
# Author: Vincent ROBERT <v.robert@of2m.fr>
# Thanks to: Tony FOUCHARD <t.fouchard@of2m.fr>
# Thanks to: Cyrille MAHIEUX <c.mahieux@of2m.fr>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

sub check_ip {
  set bereq.http.whitelisted = "0";
  if (bereq.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css|swf|xml|flv)(\?(.)*)?$") {
     set bereq.http.whitelisted = "1";
  }
  else {
     if (bereq.http.host == "whitelisted website") {
       set bereq.http.whitelisted = "1";
     }
  }
  if (bereq.http.whitelisted=="0") {
    C{
      memcached_server_st *servers = NULL;
      memcached_st *memc;
      memcached_return rc;

      char key[50]= "varnish_";
      strcat(key, VRT_IP_string(sp, VRT_r_client_ip(sp)));

      memc= memcached_create(NULL);
      servers= memcached_server_list_append(servers, "memcacheServer", 11211, &rc);
      rc= memcached_server_push(memc, servers);

      if (rc == MEMCACHED_SUCCESS) {

        uint64_t intval;
        rc= memcached_increment(memc, key, strlen(key), (uint64_t)1, &intval);

        if (rc != MEMCACHED_SUCCESS)
          rc= memcached_set(memc, key, strlen(key), "1", 1, (time_t)60, (uint32_t)0);
        else
          if (intval>30) {
            VRT_SetHdr(sp, HDR_BEREQ, "\013X-Scraping:", "1", vrt_magic_string_end);
            syslog(LOG_INFO, "Scraping detected from %s",VRT_IP_string(sp, VRT_r_client_ip(sp)));
            if (intval<300)
              rc= memcached_set(memc, key, strlen(key), "500", 3, (time_t)600, (uint32_t)0);
          }
      }
      memcached_free(memc);
      memcached_server_list_free(servers);
    }C
    if (bereq.http.X-Scraping=="1") {
      if ((bereq.url ~ "/securimage/securimage_show.php")) {
        set bereq.url = regsub(bereq.url, "^(.*)/securimage/securimage_show.php?(.*)", "/securimage/securimage_show.php?\2");
      }
      else {
        set bereq.http.referer = "http://" bereq.http.host bereq.url;
        set bereq.url = "/";
      }
      set bereq.http.host = "captcha.site";
      set req.backend = captchaServer;
    }
  }
}

sub vcl_miss {
  call check_ip;
  fetch;
}

sub vcl_pipe {
  call check_ip;
  pipe;
}

sub vcl_pass {
  call check_ip;
  pass;
}

April 15, 2010

Dr Carter — How to block web scraping using Varnish VCL, inline C & MemCached ?

Websites are not only visited by human. Spiders, bots, good or bad come everyday. Some bots don’t care about the pressure they put on servers.
So i decided to search for a solution to detect and stop them.

Our infrastruture is based on 3 Varnish servers on the top and many other backend of any kind (apache, IIS).

The principle is to modify the VCL by incorporating inline C code to store the number of requests for each visitor.
If an IP address exceeds the number of requests allowed per minute, the system sends a captcha. The surf can only resume if the captcha is found.

At the top of vcl:

# Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA
# All rights reserved.
#
# Author: Vincent ROBERT <v.robert@of2m.fr>
# Thanks to: Tony FOUCHARD <t.fouchard@of2m.fr>
# Thanks to: Cyrille MAHIEUX <c.mahieux@of2m.fr>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

C{
  #include <string.h>
  #include <libmemcached/memcached.h>
  #include <syslog.h>
}C

in the vcl_recv:

remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;

set req.http.whitelisted = "0";

if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css|swf|xml|flv)(\?(.)*)?$") {
  set req.http.whitelisted = "1";
}
else {
  if (req.http.host == "whitelisted website") {
    set req.http.whitelisted = "1";
  }
}

if (req.http.whitelisted=="0") {
  C{
    void testIP () {
      memcached_server_st *servers = NULL;
      memcached_st *memc;
      memcached_return rc;

      char key[50]= "varnish_";
      strcat(key, VRT_IP_string(sp, VRT_r_client_ip(sp)));

      memc= memcached_create(NULL);
      servers= memcached_server_list_append(servers, "memcacheServer", 11211, &rc);
      rc= memcached_server_push(memc, servers);

      if (rc == MEMCACHED_SUCCESS) {
        uint64_t intval;
        rc= memcached_increment(memc, key, strlen(key), (uint64_t)1, &intval);

        if (rc != MEMCACHED_SUCCESS)
          rc= memcached_set(memc, key, strlen(key), "1", 1, (time_t)60, (uint32_t)0);
        else
          if (intval>30) {
            VRT_SetHdr(sp, HDR_REQ, "\013X-Scraping:", "1", vrt_magic_string_end);
            syslog(LOG_INFO, "Scraping detected from %s",VRT_IP_string(sp, VRT_r_client_ip(sp)));
            if (intval<300)
              rc= memcached_set(memc, key, strlen(key), "500", 3, (time_t)3600, (uint32_t)0);
          }
      }
      memcached_free(memc);
      memcached_server_list_free(servers);
    }
    testIP();
  }C
  if (req.http.X-Scraping=="1") {
    if ((req.url ~ "/securimage/securimage_show.php")) {
      set req.url = regsub(req.url, "^(.*)/securimage/securimage_show.php?(.*)",
            "/securimage/securimage_show.php?\2");
    }
    else {
      set req.http.referer = "http://" req.http.host req.url;
      set req.url = "/";
    }
    set req.http.host = "captcha.site";
    set req.backend = captchaServer;

    if (req.request == "POST") {
      pass;
    }
    lookup;
  }
}

in the captcha file:

<?php
    header("Expires: Sun, 1 Jan 2000 12:00:00 GMT");
    header("Last-Modified: " . gmdate("D, d M Y H:i:s") . "GMT");
    header("Cache-Control: no-store, no-cache, must-revalidate");
    header("Cache-Control: post-check=0, pre-check=0", false);
    header("Pragma: no-cache");

    $showCaptcha=true;

    if(isset($_POST['captcha_code'])) {
        require_once('./securimage/securimage.php');
        $img = new Securimage();

        $showCaptcha = !$img->check($_POST['captcha_code']);

        if(!$showCaptcha) {
            //code is correct
            if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
                $memCache=new Memcache;
                $memCache->connect('memcacheServer', 11211);

                $tIp=explode (",",$_SERVER['HTTP_X_FORWARDED_FOR']);
                $ip=$tIp[0];

                $memCache->delete("varnish_".$ip);
                $memCache->close();
            }
            //redirect to original location
            header('Location: '.$_SERVER['HTTP_REFERER']);
            echo ("<a href='".$_SERVER['HTTP_REFERER']."'>retour site<a>");
            die();
        }
    }

    if($showCaptcha){
        echo '<h1>Are You Human ?</h1>';
        echo '<form method="post" action="'.htmlentities($_SERVER['HTTP_REFERER']).'" />';
        echo '<fieldset>';
        echo '<img src="./securimage/securimage_show.php?sid='.md5(uniqid(time())).'" id="image" />';
        echo '<br />';
        echo 'What code is in the image? :<br />';
        echo '<input type="text" name="captcha_code" />';
        echo '<input type="submit" value="Submit" />';
        echo '</fieldset>';
        echo '</form>';
    }
?>

daemon options needed to start Varnish:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/production.vcl \
             -u varnish -g varnish \
             -s malloc,4G \
             -p lru_interval=3600 \
             -p thread_pool_max=2000 \
             -p listen_depth=2048 \
             -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/local/include/libmemcached/memcached.h -lmemcached -o %o %s' \
             -h classic,500009 \
             -t 0"

other commands needed to permit memcache and Varnish working together:

  • yum install memcached.x86_64
  • yum install libmemcached-devel.x86_64
  • /usr/sbin/varnishd -C -f /etc/varnish/production.vcl -p ‘cc_command=exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ -lmemcache -o %o %s’

Using Varnish 2.0.6, memcached 1.4.5, libmemcached 0.39, our test in development reveal a increase of CPU under heavy load.
memcached_increment and memcached_set are the cause of increasing.

At this time, we have not put this configuration on production.


Ressources:

March 19, 2010

Kristian Lyngstøl — A sandboxed /home directory

A while back I hacked together some aufs-based scripts for sandboxing. There are two different scenarios at hand:

1. Sandboxing my /home-directory, so changes to applications don’t get saved and my home directory doesn’t fill up with lard (well, if it does, it’s gone after a reboot).

2. Making sure my firewall doesn’t write to “disk” as that’s a flash-card. And in the process ending up with a sandbox for /.

The two setups are very distinct, and the /home-directory is definitely easier to deal with, since things like dist-upgrades is a non-issue. People have asked me to write about it, so here it is.

The little home directory that could

The rationale behind this was: 1. Play around. 2. Keep a clean home directory.

The first thing I had to do was figure out a way to handle persistent storage and a base system. This was reasonably simple. I moved ~/Documents ~/Source and ~/tmp to my raid: /media/snop/kristian/home-base/, and made two new directories: /media/snop/kristian/home-base/sandbox and /media/snop/kristian/home-base/base. base/ was going to be … the actual base system. So I copied my entire home directory into base/, created a git repo from it, and started deleting whatever I didn’t need or want. This would include things like browser cache.

Next, I set up a tmpfs on home-base/sandbox/. Now the real magic: I stack-mounted home-base/sandbox/ on top of home-base/base/ with /home/kristian as the mountpoint. In other words: Upon initial login, /home/kristian would contain just what home-base/base/ contained, but any further changes to /home/kristian/ would now be stored in home-base/sandbox/ – a tmpfs. So how do I update it? Well, I have a git-repo in my home-directory now – since home-base/base/ was a git-repo. So I can simply make a patch and apply it to the base system – if I really want to. I’ve been running this for several months now, and I must say it works rather well.

I still had to handle persistence, though. But this was easy: After I mount everything, I sym-link ~/Documents, ~/Source and ~/tmp to their respective original directories now stored on /media/snop/kristian/home-base/. Just for the record: ~/Documents is a git repo itself, ~/tmp is for anything I need for a while but isn’t important enough to store in a “proper” location, and ~/Source is essentially a handful of git-repos I work on.

So how was all this possible?

Simple: AUFS.

Script:

#!/bin/sh -e

set -x
BASE=/media/snop/kristian/home-base
MYHOME=/home/kristian
MYUID=kristian
MYGID=kristian
TMPSIZE=500M
LINKS="Documents Source tmp"

mount tmpfs -t tmpfs -o mode=700,size=$TMPSIZE,uid=$MYUID,gid=$MYGID $BASE/sandbox
mount -t aufs aufs $MYHOME -o relatime,dirs=$BASE/sandbox=rw:$BASE/base=ro

for a in $LINKS; do
        if [ -f $MYHOME/$a ]; then
                echo "Warning: $a defined as link, but present in base files."
                continue
        fi
        ln -s $BASE/$a $MYHOME/$a
done

And this is how it looks from df, while running:

kristian@nihilus:~$ df -h /media/snop/kristian/home-base/{sandbox,base} /home/kristian/
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 500M  456M   45M  92% /media/snop/kristian/home-base/sandbox
/dev/mapper/vg_storage-snop
                      2.8T  2.2T  527G  82% /media/snop
aufs                  500M  456M   45M  92% /home/kristian
kristian@nihilus:~$ uptime
 14:37:35 up 55 days, 19:54,  3 users,  load average: 0.19, 0.27, 0.16
kristian@nihilus:~$ ls
Documents  Downloads  f  Source  tmp  ukurock.mp3

I included the uptime to point out that it took 55 days to use up the 500M (it went full yesterday). The ls-output demonstrates one of the reasons I like this setup: it’s really really clean. (also: Hi Petter ;) ). I’m planning to upgrade my system so I can increase the tmpfs-size. Right now, the browser cache tends to use up most of it. But I can tell you this: Building stuff on a tmpfs is …. fun.

So far I’ve only had ONE misshap with data loss, which involved moving things around rather creatively. So keep that in mind if you want to try this out.

But yeah, if you’re into experimenting, this is something you may want to check out. It’s simple, clean and safe, given reasonable precautions. It helps that ~/Documents is already a git repo which is reasonably clean. All in all, this has been an enjoyable experience. Next step: Hooking it into pam_mount and setting it up on my laptop:

kristian@kjeks:~$ ls -a1 | wc -l
318

It’s friday!


March 04, 2010

cd34 — WordPress Cache Plugin Benchmarks

A lot of time and effort goes into keeping a WordPress site alive when it starts to accumulate traffic. While not every site has the same goals, keeping a site responsive and online is the number one priority. When a surfer requests the page, it should load quickly and be responsive. Each addon handles caching a little differently and should be used in different cases.

For many sites, page caching will provide decent performance. Once your sites starts receiving comments, or people log in, many cache solutions cache too heavily or not enough. As many solutions as there are, it is obvious that WordPress underperforms in higher traffic situations.

The list of caching addons that we’re testing:

* DB Cache (version 0.6)
* DB Cache Reloaded (version 2.0.2)
* W3 Total Cache (version 0.8.5.1)
* WP Cache (version 2.1.2)
* WP Super Cache (version 0.9.9)
* WP Widget Cache (version 0.25.2)
* WP File Cache(version 1.2.5)
* WP Varnish (in beta)
* WP Varnish ESI Widget (in beta)

What are we testing?

* Frontpage hits
* httpload through a series of urls

We take two measurements. The cold start measurement is taken after any plugin cache has been cleared and Apache2 and MySQL have been restarted. A 30 second pause is inserted prior to starting the tests. We perform a frontpage hit 1000 times with 10 parallel connections. We then repeat that test after Apache2 and the caching solution have had time to cache that page. Afterwards, http_load requests a series of 30 URLs to simulate people surfing other pages. Between those two measurements, we should have a pretty good indicator of how well a site is going to perform in real life.

What does the Test Environment look like?

* Debian 3.1/Squeeze VPS
* Linux Kernel 2.6.33
* Single core of a Xen Virtualized Xeon X3220 (2.40ghz)
* 2gb RAM
* CoW file is written on a Raid-10 System using 4x1tb 7200RPM Drives
* Apache 2.2.14 mpm-prefork
* PHP 5.3.1
* WordPress Theme Test Data
* Tests are performed from a Quadcore Xeon machine connected via 1000 Base T on the same switch and /24 as the VPS machine

This setup is designed to replicate what most people might choose to host a reasonably popular wordpress site.

tl;dr Results

If you aren’t using Varnish in front of your web site, the clear winner is W3 Total Cache using Page Caching – Disk (Enhanced), Minify Caching – Alternative PHP Cache (APC), Database Caching – Alternative PHP Cache (APC).

If you can use Varnish, WP Varnish would be a very simple way to gain quite a bit of performance while maintaining interactivity. WP Varnish purges the cache when posts are made, allowing the site to be more dynamic and not suffer from the long cache delay before a page is updated.

W3 Total Cache has a number of options and sometimes settings can be quite detrimental to site performance. If you can’t use APC caching or Memcached for caching Database queries or Minification, turn both off. W3 Total Cache’s interface is overwhelming but the plugin author has indicated that he’ll be making a new ‘Wizard’ configuration menu in the next version along with Fragment Caching.

WP Super Cache isn’t too far behind and is also a reasonable alternative.

Either way, if you want your site to survive, you need to use a cache addon. Going from 2.5 requests per second to 800+ requests per second makes a considerable difference in the usability of your site for visitors. Logged in users and search engine bots still see uncached/live results, so, you don’t need to worry that your site won’t be indexed properly.

Results

<style> .tborder { border:1px solid #000; } .th { background-color: #ccc; } .teven { background-color: #ddd; } .tdnum { text-align: right; } .trecommend { background-color: #cfc; } .thonmen { background-color: #ffc; } .tsmall { font-size: 8pt; } </style>

Sorted in Ascending order in terms of higher overall performance

Addon Apachebench Cold Start
Warm Start		 http_load Cold Start
Warm Start
Req/Second Time/Request 50% within x ms Fetches/Second Min First Response Avg First Response
Baseline 4.97 201.006 2004 15.1021 335.708 583.363
5.00 200.089 2000 15.1712 304.446 583.684
DB Cache 4.80 208.436 2087 15.1021 335.708 583.363
Cached all SQL queries 4.81 207.776 2091 15.1712 304.446 583.684
DB Cache 4.87 205.250 2035 14.1992 302.335 621.092
Out of Box config 4.94 202.624 2026 14.432 114.983 618.434
WP File Cache 4.95 201.890 2009 15.8869 158.597 549.176
4.99 200.211 2004 16.1758 99.728 544.107
DB Cache Reloaded 5.02 199.387 1983 15.0167 187.343 589.196
All SQL Queries Cached 5.03 200.089 1985 14.9233 150.145 586.443
DB Cache Reloaded 5.06 197.636 1968 14.9697 174.857 589.161
Out of Box config 5.08 196.980 1968 15.181 257.533 587.737
Widgetcache 6.667 149.903 1492 15.0264 245.332 602.039
6.72 148.734 1487 15.1887 299.65 598.017
W3 Total Cache 153.45 65.167 60 133.1898 8.916 85.7177
DB Cache off, Page Caching with Memcached 169.46 59.011 57 188.4 9.107 50.142
W3 Total Cache 173.49 57.639 52 108.898 7.668 86.4077
DB Cache off, Minify Cache with Memcached 189.76 52.698 48 203.522 8.122 43.8795
W3 Total Cache 171.34 58.364 50 203.718 8.097 44.1234
DB Cache using Memcached 190.01 52.269 48 206.187 8.186 42.4438
W3 Total Cache 175.29 57.048 48 87.423 7.515 107.973
Out of Box config 191.15 52.314 47 204.387 8.288 43.217
W3 Total Cache 175.29 57.047 51 204.557 8.199 42.9365
Database Cache using APC 191.19 52.304 48 200.612 8.11 44.6691
W3 Total Cache 114.02 87.703 49 114.393 8.206 82.0678
Database Cache Disabled 191.76 52.150 49 203.781 8.095 42.558
W3 Total Cache 175.80 56.884 51 107.842 7.281 87.2761
Database Cache Disabled, Minify Cache using APC 192.01 52.082 50 205.66 8.244 43.1231
W3 Total Cache 104.90 95.325 51 123.041 7.868 74.5887
Database Cache Disabled, Page Caching using APC 197.55 50.620 46 210.445 7.907 41.4102
WP Super Cache 336.88 2.968 16 15.1021 335.708 583.363
Out of Box config, Half On 391.59 2.554 16 15.1712 304.446 583.684
WP Cache 161.63 6.187 12 15.1021 335.708 583.363
482.29 20.735 11 15.1712 304.446 583.684
WP Super Cache 919.11 1.088 3 190.117 1.473 47.9367
Full on, Lockdown mode 965.69 1.036 3 975.979 1.455 9.67185
WP Super Cache 928.45 1.077 3 210.106 1.468 43.8167
Full on 970.45 1.030 3 969.256 1.488 9.78753
W3 Total Cache 1143.94 8.742 2 165.547 0.958 56.7702
Page Cache using Disk Enhanced 1222.16 8.182 3 1290.43 0.961 7.15632
W3 Total Cache 1153.50 8.669 3 165.725 0.916 56.5004
Page Caching – Disk Enhanced, Minify/Database using APC 1211.22 8.256 2 1305.94 0.948 6.97114
Varnish ESI 2304.18 0.434 4 349.351 0.221 28.1079
2243.33 0.44689 4 4312.78 0.152 2.09931
WP Varnish 1683.89 0.594 3 369.543 0.155 26.8906
3028.41 0.330 3 4318.48 0.148 2.15063

Test Script

#!/bin/sh

FETCHES=1000
PARALLEL=10

/usr/sbin/apache2ctl stop
/etc/init.d/mysql restart
apache2ctl start
echo Sleeping
sleep 30
time ( \
echo First Run; \
ab -n $FETCHES -c $PARALLEL http://example.com/; \
echo Second Run; \
ab -n $FETCHES -c $PARALLEL http://example.com/; \
\
echo First Run; \
./http_load -parallel $PARALLEL -fetches $FETCHES wordpresstest; \
echo Second Run; \
./http_load -parallel $PARALLEL -fetches $FETCHES wordpresstest; \
)

URL File for http_load

http://example.com/

http://example.com/2010/03/hello-world/

http://example.com/2008/09/layout-test/

http://example.com/2008/04/simple-gallery-test/

http://example.com/2007/12/category-name-clash/

http://example.com/2007/12/test-with-enclosures/

http://example.com/2007/11/block-quotes/

http://example.com/2007/11/many-categories/

http://example.com/2007/11/many-tags/

http://example.com/2007/11/tags-a-and-c/

http://example.com/2007/11/tags-b-and-c/

http://example.com/2007/11/tags-a-and-b/

http://example.com/2007/11/tag-c/

http://example.com/2007/11/tag-b/

http://example.com/2007/11/tag-a/

http://example.com/2007/09/tags-a-b-c/

http://example.com/2007/09/raw-html-code/

http://example.com/2007/09/simple-markup-test/

http://example.com/2007/09/embedded-video/

http://example.com/2007/09/contributor-post-approved/

http://example.com/2007/09/one-comment/

http://example.com/2007/09/no-comments/

http://example.com/2007/09/many-trackbacks/

http://example.com/2007/09/one-trackback/

http://example.com/2007/09/comment-test/

http://example.com/2007/09/a-post-with-multiple-pages/

http://example.com/2007/09/lorem-ipsum/

http://example.com/2007/09/cat-c/

http://example.com/2007/09/cat-b/

http://example.com/2007/09/cat-a/

http://example.com/2007/09/cats-a-and-c/

February 26, 2010

cd34 — Using Varnish to assist with AB Testing

While working with a recent client project, they mentioned AB Testing a few designs. While I enjoy statistics, we looked at Google’s Website Optimizer to track trials and conversions. After some internal testing, we opted to use Funnels and Goals rather than the AB or Multivariate test. I had little control over the origin server, but I did have control over the front-end cache.

Our situation reminded me of a situation I encountered years ago. A client had an inhouse web designer and a subcontracted web designer. I felt the subcontracted web designer’s design would convert better. The client wasn’t completely convinced, but agreed to running two designs head to head. However, their implementation of the test biased the results.

What went wrong?

Each design was run for a week, in series. While this provided ample time for gathering data, the inhouse designer’s design ran during a national holiday with a three day weekend, and the subcontractor’s design ran the following week. Internet traffic patterns, the holiday weekend, weather, sporting events, TV/Movie premieres, etc. added so many variables which should have invalidated the results.

Since Google’s AB Testing has session persistence and splits traffic between the AB tests, we need to emulate this behavior. When people run AB tests in series rather than parallel, or, switch pages with a cron job or some other automated method, I cringe. A test at 5pm EST and 6pm EST will yield different results. At 5pm EST, your target audience could be driving home from work. At 6pm EST they could be sitting down for dinner.

How can Varnish help?

If we allow Varnish to select the landing page/offer page outside the origin server’s control, we can run both tests run at the same time. An internet logjam in Seattle, WA would affect both tests evenly. Likewise, a national or worldwide event would affect both tests equally. Now that we know how to make sure the AB Test is fairly balanced, we have to implement it.

Redirection sometimes plays havoc on browsers and spiders, so, we’ll rewrite the URL within Varnish using some Inline C and VCL. Google uses javascript and a document.location call to send some visitors to the B/alternate page. Users that have javascript disabled, will only see the Primary page.

Our Varnish config file contains the following:

sub vcl_recv {
  if (req.url == "/") {
    C{
      char buff[5];
      sprintf(buff,"%d",rand()%2 + 1);
      VRT_SetHdr(sp, HDR_REQ, "\011X-ABtest:", buff);
    }C
    set req.url = "/" req.http.X-ABtest "/" req.url;
  }
}

We’ve placed our landing pages in /1/ and /2/ directories on our origin server. The only page Varnish intercepts is the index page at the root of the site. Varnish randomly chooses to serve the index.html page from /1/ or /2/, internally rewrites our URL and serves it from the cache or the origin server. Since the URL rewriting is done within vcl_recv, subsequent requests for the page don’t hit the origin. The same method can be used to test landing pages that aren’t at the root of your site by modifying the if (req.url == “”) { condition.

You can test multipage offers by placing additional pages within the /1/ and /2/ directories on your origin along with the signup form. Unlike Google’s AB Test, Varnish does not support session persistence. Reloading the root page will result in the surfer alternating between both test pages. Subsequent pages need to be loaded from /1/ or /2/ based on which landing page was selected.

When doing any AB Test, change as few variables as possible, document the changes, and analyze the difference between the results. Running at least 1000 views of each is an absolute minimum. While Google’s Multivariate test provides a lot more options, a simple AB test between two pages or site tours can give some insight into what works rather easily.

If you cannot use Google’s AB Test or the Multivariate Test, using their Funnels and Goals tool will still allow you to do AB Testing.

February 18, 2010

cd34 — Varnish VCL, Inline C and a random image

While working with the prototype of a site, I wanted to have a particular panel image randomly chosen when the page was viewed. While this could be done on the server side, I wanted to move this to Varnish so that Varnish’s cache would be used rather than piping the request through each time to the origin server.

At the top of /etc/varnish/default.vcl

C{
  #include <stdlib.h>
  #include <stdio.h>
}C

and our vcl_recv function gets the following:

  if (req.url ~ "^/panel/") {
    C{
      char buff[5];
      sprintf(buff,"%d",rand()%4);
      VRT_SetHdr(sp, HDR_REQ, "\010X-Panel:", buff);
    }C
    set req.url = regsub(req.url, "^/panel/(.*)\.(.*)$", "/panel/\1.ZZZZ.\2");
    set req.url = regsub(req.url, "ZZZZ", req.http.X-Panel);
  }

The above code allows for us to specify the source code in the html document as:

<img src="/panel/random.jpg" width="300" height="300" alt="Panel Image"/>

Since we have modified the request uri in vcl_recv before the object is cached, subsequent requests for the same modified URI will be served from Varnish’s cache, without requiring another fetch from the origin server. Based on the other VCL and preferences, you can specify a long expire time, remove cookies, or do ESI processing. Since the regexp passes the extension through, we could also randomly choose .html, .css, .jpg or any other extension you desire.

In the directory panel, you would need to have 

/panel/random.0.jpg
/panel/random.1.jpg
/panel/random.2.jpg
/panel/random.3.jpg

which would be served by Varnish when the url /panel/random.jpg is requested.

Moving that process to Varnish should cut down on the load from the origin server while making your site look active and dynamic.

February 16, 2010

Kristian Lyngstøl — Wikish – Mediawiki-editing on command line

wiki.sh (or wikish) is a shell-script I wrote to work with our internal mediawiki-wikis. It’s functional for both the new varnish-software wiki and the old redpill-linpro wiki I use.

Basically it lets you edit wiki-content with vi, vim, gvim or even nvi. Very simple, and a result of mvs, wikipediafs and mediawiki-fuse being seemingly horrible messes. It also makes it possible for me to easily script copy/pasting from one wiki to an other without much hassel. All in all it’s a little over 200 lines of sh-code using lwp-request/curl and awk. It works.

Get it from: http://github.com/KristianLyng/wikish

Currently doesn’t support “normal” mediawiki-login, but rpl and varnish-software use basic auth over ssl and work fine. I’ll gladly add normal login if someone gives me an account on a wiki they own and operate where I can test it (not wikipedia please, unless you are wikimedia).

Usage example: wikiedit “Main Page”. (that simple, yes)

PS: Sorry for hijacking the name.


February 02, 2010

Kristian Lyngstøl — Varnish purges

Varnish purges can be tricky. Both the model of purging that Varnish use and the syntax you need to use to take advantage of them can be difficult to grasp. It took me about five Varnish Administration Courses until I was happy with how I explained it to the participants, specially because the syntax is the most confusing syntax we have in VCL. However, it’s not very hard to work with once you understand the magic at work.

0. Separating purges and forced expiry

There are two ways to throw something out of the cache before the TTL is due in Varnish. You can either find the object you want gone and set the TTL to 0 forcing it to expire, or use Varnish’ purge mechanism. Setting ttl to 0 has it’s advantages, since you evict the object immediately, but it also means you have to evict one object at a time. This is fairly easy and usually done by having Varnish look for a “PURGE” request and handle it. This is not what I’ll talk about today, though. Read http://varnish-cache.org/wiki/VCLExamplePurging for information on forcibly expiring an object.

1. The challenges of purging a cache

The main reason people purge their cache, is to make room for updated content. Some journalist updated an article and you want the old one – possibly cached for days – gone. In addition, you may not know exactly what to cache, or it might be broader than just one item. En example would be a template used to generate multiple php files. Or all sports articles.

All in all, you do not purge to conserve memory. Because you expect that the cache will be filled soon.

If you are to purge all your php pages and you have 150 000 objects, you may not want to go looking for them either. This the reason some competing cache products are slow at large purging. By looking for all these objects, you might have to hit the disk to fetch cold objects.

In varnish, we also leave it up to VCL what’s unique to an object. That is to say: You can override the cache hash. By default it’s the host name or server IP combined with the “URL”. This is usually what people want, but sometimes you may want to add a cookie into the mix, for instance. The point is, we don’t know exactly what people cache on.

2. How Varnish attacks the problem

In Varnish, you purge by adding a purge to a list. This list can grow large if you add several very specific purges, but we try to reduce the overlap as much as possible. The purge in question can be pretty much anything you can match in VCL, including regular expressions on URLs, host names and user-agents for that matter. You can see the list by typing “purge.list” in the command line interface (CLI, or telnet).

Each object in your cache points to the last purge it was tested against. When you hit an object, it checks if there are any new purges in the list, test the object against them, then either evict the object and fetch a new one, or update the “last tested against”-pointer.

Because of this, the ‘req’-structure you are evaluating is actually that of the client to access the object next, not the client who pulled the object from the backend. It also means that every single object in your cache that is hit will be tested against all purges to see if it matches. But it’s spread out over time. It might sound wasteful, but it means you can add purges at constant time, and not really think about the cost of evaluating them.

It also means the object stays in the cache until it expires if it is not hit. So you don’t free up memory.

3. Adding purges “by hand”

Want to purge a http://example.com/somedirectory/ and everything beneath that path?

purge req.http.host == example.com && req.url ~ ^/somedirectory/.*$

or

purge req.url ~ ^/somedirectory/ && req.http.host == example.com

Want to purge all objects with a “Cache-Control: max-age=” set to 3600 ?

purge obj.http.Cache-Control ~ max-age=3600

or to take white space into account and no trailing numbers:

purge obj.http.Cache-Control ~ max-age ?= ?3600[^0-9]

Notice that all of the variables are in the same “VCL-context” as the client to hit the object next, so if you purge on req.http.user-agent, it’s fairly random if the object is really purged, because you (probably) can’t predict what user-agent the next person to visit a specific object is using. If you wish to purge based on a parameter sent from the “original” client, you will have to store that parameter in obj.http somewhere and remove it in vcl_deliver if you don’t want to expose it.

4. Adding purges in VCL

This is where it gets tricky. The normal example of why, is this: purge(“req.url == ” req.url);

Normal programming-thinking would tell you that this would match everything, since the url is always equal to itself. This is where VCL string concatenation comes into the picture. In reality, you are writing: “add this to the purge list: The string containing “req.url == ” and the value of the variable req.url”.

In other words, if the client access http://example.com/foobar and hit the code above, this would say: “Add the string containing “req.url == ” and “/foobar” to the purge list.” The quotation marks are essential!

I find it easier to think of it as preparing a string for the purge-command on cli. Varnish concatenates two strings without any special sign.

In the end, this is the rule of thumb: Put everything you expect to see literally when you type “purge.list” inside quotation marks, and put things you wish to replace with the variable of the calling session outside.

So you actually have three different VCL contexts to worry about:

  1. The context that originally pulled the object in from a backend (not much you can do here unless you hide things in obj.http)
  2. The context that will hit the object and thereby test the object against the purge. Any variable in this context has to be inside quotation marks.
  3. The context that triggered the purge, variables from this context should be outside quotation marks, so they are replaced with their string values before being added to the purge list.

The reason you do not need quotation marks if you enter the purge command on the command line interface is because you don’t have the third context. There is no req.url in telnet, since you are not going through VCL at all.

Some examples, note that when I say “supplied by the client” I mean the client initiating the purge, typically some smart system you’ve set up:

Purge object on the current host and URLs matching the regex stored in the X-Purge-Regex header supplied by the client:

purge("req.http.host == " req.http.host " && req.url ~ " req.http.X-Purge-Regex);

Purge all php for any example.com-domain:

purge("req.http.host ~ example.com$ && req.url ~ ^/.*\.php");

Same, but for the host provided in the X-Purge-HostPHP:

purge("req.http.host ~ " req.http.X-Purge-HostPHP " && req.url ~ ^/.*\.php");

Purge objects with X-Cache-Channel set to “sport”:

purge("obj.http.X-Cache-Channel ~ sport");

Same, but purge the cache-channel set in the header ‘X-Purge-CC’:

purge("obj.http.X-Cache-Channel ~ " X-Purge-CC);

Purge in vcl_fetch if the backend sent a X-Purge-URL header (weird thing to do, but fun example):

sub vcl_fetch {
(....)
if (obj.http.X-Purge-URL) {
purge("req.url ~ " obj.http.X-Purge-URL);
}
(...)
}

(PS: I have not actually tested all these examples, but they look correct)


January 26, 2010

Kristian Lyngstøl — Varnish best practices

A while ago I wrote about common Varnish issues, and I think it’s time for an updated version. This time, I’ve decided to include a few somewhat uncommon issues that, if set, can be difficult to spot or track down. A sort of pitfall-avoidance, if you will. I’ll add a little summary with parameters and such at the end.

1. Run Varnish on a 64 bit operating system

Varnish works on 32-bit, but was designed for 64bit. It’s all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you’re somewhat on your own. However, try to fit it within 2GB. I wouldn’t recommend a cache larger than 1GB, and no more than a few hundred threads… (Why are you on 32bit again?)

2. Watch /var/log/syslog

Varnish is flexible, and has a relatively robust architecture. If a Varnish worker thread was to do something Bad and Varnish noticed, an assert would be triggered, Varnish would shut down and the management process would start it up again almost instantly. This is logged. If it wasn’t, there’s a decent chance you wouldn’t notice, since the downtime is often sub-second. However, your cache is emptied. We’ve had several customers contact us about performance-issues, only to realize they’re essentially restarting Varnish several times per minute.

This might make it sound like Varnish is unstable: It’s not. But there are bugs, and I happen to see a lot of them, since that’s my job.

An extra note: On Debian-based systems, /var/log/messages and /var/log/syslog is not the same. Varnish will log the restart in /var/log/messages but the actual assert error is only found in /var/log/syslog, so make sure you look there too.

The best way to deal with assert errors is to search our bug tracker for the relevant function-name.

3. Threads

The default values for threads is based on a philosophy I’ve since come to realize isn’t optimal. The idea was to minimize the memory footprint of Varnish. So by default, Varnish uses 5 threads per thread pool. By default, that’s 10 threads minimum. The maximum is far higher, but in reality, threads are fairly cheap. If you expect to handle 500 concurrent requests, tune Varnish for that.

A little clarification on the thread-parameters: thread_pool_min is the minimum number of threads for each thread pool. thread_pool_max is the maximum total number of threads. That means the values are not on the same scale. The thread_pools parameter can safely be ignored (tests have indicated that it doesn’t matter as much as we thought), but ideally having one thread_pool for each cpu core is the rule of thumb, if you want to modify it.

You also do not want more than 5000 as the thread_pool_max. It’s dangerous, though fixed in trunk. It’s also more often than not an indication that something else is wrong. If you find yourself using 5000 threads, the solution is to find out why it’s happening, not to increase the number of threads.

To reduce the startup time, you also want to reduce the thread_pool_add_delay parameter. ’2′ is a good value (as opposed to 20 which makes for a slow start).

4. Tune based on necessity

I often look at sites where someone has tried to tune Varnish to get the most out of it, but taken it a bit too far. After working with Varnish I’ve realized that you do not really need to tune Varnish much: The defaults are tuned. The only real exception I’ve found to this is number of threads and possibly work spaces.

Varnish is – by default – tuned for high performance on the vast majority of real-life production sites. And it scales well, in most directions. By default. Do yourself a favor and don’t fix a problem which isn’t there. Of all the issues I’ve dealt with on Varnish, the vast majority have been related to finding out the real problem and either using Varnish to work around it, or fix it on the related system. Off the top of my head, I can really only remember one or two cases where Varnish itself has been the problem with regards to performance.

To be more specific:

  • Do not modify lru_interval. I often see the value “3600″. Which is a 180 000% (one hundred and eighty thousand percent) increase from the default. This is downright dangerous if you suddenly need the lru-list, and so far my tests haven’t been able to prove any noticeable performance improvement.
  • Setting sess_timeout to a higher value increase your filedescriptor consumption. There’s little to gain by doing it too. You risk running out of file descriptors. At least until we can get the fix into a released version.

So the rule of thumb is: Adjust your threads, then leave the rest until you see a reason to change it.

5. Pay attention to work spaces

To avoid locking, Varnish allocates a chump of memory to each thread, session and object. While keeping the object workspace small is a good thing to reduce the memory footprint (this has been improved vastly in trunk), sometimes the session workspace is a bit too small, specially when ESI is in use. The default sess_workspace is 16kB, but I know we have customers running with 5MB sess_workspace without trouble. We’re obviously looking to fix this, but so far it seems that having some extra sess_workspace isn’t that bad. The way to tell is by asserts (unfortunately), typically something related to “(p != NULL) Condition not true” (though there can obviously be other reasons for that). Look for it in our bug report, then try to increase the session workspace.

6. Keep your VCL simple

Most of your VCL-work should be focused around vcl_recv and vcl_fetch. That’s where you define the majority of your caching policies. If that’s where you do your work, you’re fairly safe.

If you want to add extra headers, do it in vcl_deliver. Adding a header in vcl_hit is not safe. You can use the “obj.hits” variable in vcl_deliver to determine if it was a cache hit or not.

You should also review the default vcl, and if you can, let Varnish fall through to it. When you define your VCL, Varnish appends the default VCL, but if you terminate a function, the default is never run. This is an important detail in vcl_recv, where requests with cookies or Authroization-headers are passed if present. That’s far safer than forcing a lookup. The default vcl_recv code also ensures that only GET and HEAD-requests go through the cache.

Focus on caching policy and remember that the default VCL is appended to your own VCL – and use it.

7. Choosing storage backend (malloc or file?)

If you can contain your cache in memory, use malloc. If you have 32GB of physical memory, using -smalloc,30G is a good choice. The size you specify is for the cache, and does not include session workspace and such, that’s why you don’t want to specify -smalloc,32G on a 32GB-system.

If you can not contain your cache in memory, first consider if you really need that big of a cache. Then consider buying more memory. Then sleep on it. Then, if you still think you need to use disk, use -sfile. On Linux, -sfile performs far better than -smalloc once you start hitting disk. We’re talking pie-chart-material. You should also make sure the filesystem is mounted with noatime, though it shouldn’t be necessary. On Linux, my cold-hit tests (a cold hit being a cache hit that has to be read from disk, as opposed to a hot hit which is read from memory) take about 6000 seconds to run on -smalloc, while it takes 4000 seconds on -sfile with the same hardware.  Consistently. However, your milage may vary with things such as kernel version, so test both anyway. My tests are easy enough: Run httperf through x-thousand urls in order. Then do it again in the same order.

Some of the most challenging setups we work with are disk-intensive setups, so try to avoid it. SSD is a relatively cheap way to buy yourself out of disk-issues though.

8. Use packages and supplied scripts

While it may seem easier to just write your own script and/or install from source, it rarely pays off in the long run. Varnish usually run on machines where downtime has to be planned, and you don’t want a surprise when you upgrade it. Nor do you want to risk missing that little bug we realized was a problem on your distro but not others. If you do insist on running home-brew, make sure you at least get the ulimit-commands from the startup scripts.

This is really something you want regardless of what sort of software you run, though.

9. Firewall and sysctl-tuning

Do not set “tw_reuse” to 1 (sysctl). It will work perfectly fine for everyone. Except thousands of people behind various NAT-based firewalls. And it’s a pain to track down. Unfortunately, this has been an advice in the past.

Avoid connection-tracking on the Varnish server too. If you need it, you’ll need to tune it for high performance, but the best approach is simply to not do connection-tracking on a server with potentially thousands of new connections per second.

10. Service agreements

(Service agreements are partly responsible for my salary, so with that “conflict of interest” in mind….)

You do not need a service agreement to run Varnish. It’s free software.

However, if you intend to run Varnish and your site is business critical, it’s sound financial advice to invest some money in it. We are the best at finding potential problems with your Varnish-setup before they occur, and solving them fast when they do occur.

We typically start out by doing a quick sanity-test of your configuration. This is something we can do fast, both with regards to parameters, VCL and system configuration. Some of our customers only contact us when there’s something horribly wrong, others more frequently to sanity-check their plans or check up on how to use varnisncsa for their particular logging tool and so on. It’s all up to you.

We also have a public bug tracker anyone can access and submit to. We do not have a private bug tracker, though there are bugs that never hit the public bug tracker – but that’s because we fix them immediately. Just like any other free software project, really. We have several public mailing lists, and we answer them to the best of our ability, but there is no guarantee and our time is far more limited. If you run into a bug, my work on other bugs will be postponed until your problems are solved. Better yet: if you run into something you don’t know is a bug, we can track it down.

A service agreement gives you saftey. And your needs will get priority when we decide where we want to take Varnish in the future.

We also offer training on Varnish, if you prefer not to rely on outside competence.

Oh, and I get to eat. Yum.

Summary

Keep it simple and clean. Do not use connection tracking or tw_reuse. Try to fit your cache into memory on a 64-bit system.

Watch your logs.

Parameters:

thread_pool_add_delay=2
thread_pools = <Number of cpu cores>
thread_pool_min = <800/number of cpu cores>
thread_pool_max = 4000
session_linger = 50
sess_workspace = <16k to 5m>

So if you have a dual quad core CPU, you would have 8 cpu cores. This would make sense: thread_pools=8, thread_pool_min=100, thread_pool_max=4000. The number 800 is semi random: it seems to cover most use-cases. I addedd session_linger into the mix because it’s a default in Varnish 2.0.5 and 2.0.6 but not in prior versions, and it makes good sense.


January 21, 2010

Ingvar Hagelund — The usage of varnish revisited

Varnish is a high-performance HTTP accelerator. Working with Varnish is part of my day job. Among other things, I maintain the packages for Fedora and EPEL.

This is more or less a repost, with updated numbers.

Some months have passed, and it is time to run my poking scripts again, looking for sites that run Varnish. There is no deep magic here. I just parse the available top lists that I know of, and peek at the HTML headers of the sites that are listed. If there are subsites linked from the front page of the site, I scan them too. This means that twitter.com shows up, though Twitter only runs Varnish on its search site. Subsites with a Varnish match are shown in parenthesis in the results.

For the Nordic countries, I have found quite good lists, that is, upload result lists from the probably most visited media sites in the respective countries. Remember of course, that these are generally pay-to-be-included lists, and there may exist sites with far more hits than the ones listed.

For a global overview, I have used Alexa.

Now for the results. Varnish is sponsored by large Norwegian sites, so it is no big surprise that there are a lot of hits in Norway. Of the TNS Gallup top list, Varnish runs at 36 of the top 100 sites. That’s 3 up since my last probe.

For Denmark, I use FDIM’s list. From May, we are up from 3 to 11 sites in the top 100. For Finland, I use TNS’ numbers again. No changes there. For Sweden, I use the KIA Index list. I now probe the 200 top sites, so there are several more varnish sites on the list. In the top 100, we are up from 8 to 9.

For the Alexa’s World top 500 list, I have tweaked my filters a bit, and the list is up from 7 to 17 sites since my last probe in May. World Domination, here we come!

The whole World: Alexa's global top 500 list
Place  12 Varnish running on twitter.com (http://integratedsearch.twitter.com/search.html)
Place  47 Varnish running on photobucket.com (http://blog.photobucket.com/)
Place  90 Varnish running on mixi.jp (http://img.mixi.jp/static/css/basic/logout_quirks) (and others)
Place 101 Varnish running on weather.com (http://www.weather.com/) (weather.com)
Place 107 Varnish running on globo.com (globo.com) (http://www.globo.com/)
Place 111 Varnish running on ifeng.com (http://big5.ifeng.com/gate/big5/www)
Place 138 Varnish running on answers.com (http://www.answers.com/) (answers.com)
Place 170 Varnish running on orange.fr (http://r.orange.fr/r/Eorangepublicite)
Place 179 Varnish running on hulu.com (hulu.com)
Place 199 Varnish running on wikia.com (wikia.com)
Place 213 Varnish running on xinhuanet.com (http://big5.xinhuanet.com/gate/big5/www)
Place 290 Varnish running on people.com.cn (http://bbs1.people.com.cn/boardList) (and others)
Place 344 Varnish running on tuenti.com (http://estaticos1.tuenti.com/layout/web2/images/favicon)
Place 428 Varnish running on chinanews.com.cn (http://www.chinanews.com.cn/test/index_back_d.html) (and others)
Place 433 Varnish running on mercadolivre.com.br (http://veiculos.mercadolivre.com.br/) (and others)
Place 447 Varnish running on mercadolibre.com.mx (http://tendencias.mercadolibre.com.mx/) (and others)
Place 456 Varnish running on break.com (break.com)

In my last probe, I poked sites all over Europe. With a few exceptions, that was a bit less interesting. Global .com and .net sites tend to cover most of the top 100 entries, as I had only toolbar lists, and it’s not that spectacular that for example people in Serbia and Monte Negro are browsing Twitter, like the rest of the World does. So I have skipped other countries. If you know of any good top list for you country that is not toolbar based, please let me know.

All the gory details are available here.

Other more or less worth mentioned sites that is reported to use Varnish but does not show up in my lists, may be Slashdot, WAT TV, The Pirate Bay, JDownloader, e.Republik, WOWwiki, Globo.com, PCWelt.de, BlackPlanet, funnyordie.com, n-tv.de and 20minutos.es to name a few.

Do you know of other famous sites running Varnish? Use the comments.

January 19, 2010

Kristian Lyngstøl — Real-time and hit-and-run based statistics for VSTS

A while ago I wrote VSTS – the Varnish Stress Testing Suite. It’s not nearly as fancy as the name makes it sound: It’s a simple set of shell scripts that runs on a set of test servers and pounds Varnish under a couple of different scenarios. The idea is simple: Detect possible performance issues during the development cycle of Varnish by periodically testing the current code against the same well-established test.

It does the job, but could be far better. Specially when it comes to statistics.

The rrdtool-backed Python script is simply insufficient. As rrdtool likes to put data entry into a time-slot, and I don’t really operate in time slots, the data is both slow and imprecise. This is fine if you want data for every 5 minute throughout a year, but not if you want data collected anwhere from one time to ten times a day. Simply put: I want one data entry to represent one unit on the X-scale.

I also want to add “real-time” statistics. The current data is collected after each test, but I want data to be gathered throughout the relevant tests. This should be seperate from the other statistics. Hopefully, the system I end up with is flexible enough to allow me to plot all the datasets on the same graph, which should make any variations easy to spot.

Up until now I’ve only ever dealt with rrdtool when it comes to statistics (unless you count pre-rrdtool mrtg and the like), so I was hoping for some pointers before I start digging through what I suspect is a jungle of similar but different solutions. I’m comfortable working with most languages, and the primary concerns I’m looking at is implementation complexity and flexibility.

When I’m done with this batch of refactoring, I’ll do a proper writeup. However, what I’m most excited about is adding OpenSolaris into the target-platforms (should help clean out a few bugs that have been plaguing the Solaris-users for a while), adding better support for customized tests and improved robustness of VSTS.


January 13, 2010

Kristian Lyngstøl — Pushing Varnish even further

A while ago I did a little writeup on high-end Varnish tuning, where I noted that I made our single core 2.2GHz Opteron reach 27k requests/second. This begged the questions as to how well Varnish scale with hardware. So I went ahead and tried to overload our quad-core Xeon at 2.4GHz. It would obviously take some extra fire power. At the very least, four times as much as the last batch of tests.

Hardware involved

Our main set of test servers for Varnish are called varnish1, varnish2, varnish3, varnish4, varnish6 and varnish7. These have mostly different software and hardware – which is done intentionally so we can perform tests under different circuimstances. We routinely run tests against Varnish2 and Varnish4, which run CentOS and FreeBSD, respectively. For my last test, I used Varnish2 as the server and the remaining servers as test nodes. By any normal math, I would need  about 4 times more fire power to overload a 2.4GHz Quad core, compared to a single core Opteron at 2.2GHz.

To sum it up as far as this round of tests go:

  • Varnish1 – Single core Opteron
  • Varnish2 – Single core Opteron at 2.2GHz (used in the last round of tests)
  • Varnish3 – Single core Xeon (if I’m not much mistaken). It’s also the nginx server used as backend, but that just means 1 request every X minutes.
  • Varnish4 – Single core Opteron (FreeBSD)
  • Varnish6 – Dual core Xeon of some kind
  • Varnish7 – Quad-core Xeon at 2.4GHz

So I needed more power. As it happens, we do alot of training and we have three classrooms full of computers for students, and I borrowed two of these class rooms, adding the following to the mix:

  • 10 x single core Pentium Celerons at 2.9x GHz
  • 10 x Core 2 Duos at 2.4ish GHz

As you might notice – a large part of the challenge when you want to test Varnish is getting your test systems to keep up.

Basic test procedures

Same as last time, more or less: 1 byte pages and httperf. I’ve tried ab, siege and curl… And they simply do not offer the raw power of httperf combined with the control – if anyone cares to enlighten me on how to get the most out of them, then I’m more than willing to listen.

Ideally I wanted to test with 10 requests for each connection, and with mixed data set size. As it turns out, I ended up using 100 requests / second and bursting all of the requests, which is far from realistic. More on this later.

I have an intricate script system for the nightly tests, but that’s a story for an other time. For these tests I simply used clusterssh to replicate my input on 37ish shells. This has allowed me to instantly test identical setups on all the nodes, and to quickly review what their status is. I probably ran a thousand or more different variants of the same test this time around.

I’ve used varnishstat to monitor the request rate and other relevant stats, and top to monitor general load.

The backend I use is hosted on varnish3, which runs nginx and a simple rewrite to ‘current.txt’, which for this occasion was linked to a 1byte file.

Results

Varnish uses alot of threads, and as such, when it does finally saturate the CPU, the load average will skyrocket. On the last test, Varnish2 had a load of 600-700. During this load, Varnish2 would use 10-15 seconds to start ‘top’.

During this round of tests I had roughly 87GHz worth of clients, spread over 25 physical computers. All of the tests systems were running at full load. Varnish7 had a load average around 45. Logging in and starting top was close to instant. And Varnish was serving 143k requests per second.

Based on the load and general snappiness, I think it is safe to conclude that while Varnish was close to the breaking point, it hadn’t actually reached it. To put it simply: My clients were not fast enough. Before I told httperf to burst 100 requests for each connection, Varnish was serving 110-120k requests per second with a load less than 1.0, and the clients were still using all their fire power. I ended up stress testing my clients. Dammit.

However, as I came fairly close to the breaking point, I still believe there are a few interesting things to look at.

The scaling nature of Varnish

It’s very rare that you can see an application scale so well just by throwing cpu power and cpu cores at it. Varnish essentially didn’t get affected at all by the extra work needed to synchronize work on 4 cpu cores. In fact, if you look at the math, the raw performance on 4 cpu cores was actually BETTER than on one cpu core, when you look at it on a cycle-by-cycle.

I think it’s reasonably safe to say that when it comes to raw performance, we’ve nailed it with Varnish.

In fact, scaling Varnish is far more difficult when you increase your active data set beyond physical memory. Or when you introduce latency. Or when when you have a low cache hit rate. Or any other number of corner cases. There will always be bottlenecks.

What you can learn from this is actually simple: Do not focus on the CPU when you want to scale your Varnish setup. I know it’s tempting to buy the biggest baddest server around for a high-traffic site, but if your active data set can fit within physical memory and you have a 64-bit CPU, Varnish will thrive. And for the record: All CPU-usage graphs I’ve seen from Varnish installations confirm this. Most of the time, those sexy CPUs are just sitting idle regardless of traffic.

Myths and further research

Since I didn’t reach the breaking point, there’s not much I can say conclusively. However, I can repeat a few points.

Adjusting the lru_interval had little impact regardless of data set and access patterns. If I repeat this often enough, perhaps I’ll stop seeing new installations with an lru_interval of 3600: DO NOT SET lru_interval TO 3600. There. I didn’t even add the usual “unless you know what you are doing” part. I might’ve explained it before, but the problem is that it leaves you with a really badly sorted lru-listed that will cause Bad Things once you need to lru-nuke something. Possibly really really bad things. Like throwing out the 200 most popular objects on your site at the same time.

And the size of your VCL has little impact on the performance. I have not tested this extensively, but I’ve never registered a difference, and since your cpu will be idle most of the time anyway, you should NOT worry about CPU cycles in VCL.

An otherimportant detail is that your shmlog shouldn’t trigger disk activity. On my setup, it didn’t sync to disk to begin with, but you may want to stick it on a tmpfs just to be sure. I suspect this has improved throughout the 2.0-series of Varnish, but it’s an easy insurance. Typically the shmlog is found in /usr/var/varnish, /usr/local/var/varnish or similar (“ls /proc/*/fd | grep _.vsl” is the lazy way to find it).

I tried several different settings of thread pools, acceptors, listen depth, shmlog parameters, rush exponent and such, but none of it revealed much – most likely because I never pressured Varnish enough. This will be what I want to investigate further. But it should tell you something about how far you have to go before these obscure settings start to matter.

Feedback wanted

I figure this must be some sort of record, but I’m interested in what sort of numbers others have seen or are seeing. Have anyone even come close to the numbers above – synthetic or otherwise – from a single server? Regardless of software or hardware? This is not meant as a challenge or boast, but I’m genuinely curious on what sort of traffic people are able to push. I’m interested in more “normal” requests rates too – I’m a sucker for numbers. What are you seeing on your site? Have you had scaling issues?


January 08, 2010

Kristian Lyngstøl — Hitpass objects and Varnish

Yesterday presented an interesting case of “what the hey is going on!?” for a customer of ours. The problem presented itself as a front page that sporadically needed 40 to 50 seconds to purge. After a brief hunt I discovered a somewhat interesting set of coincidences.

Before I go into detail, I need to tell you what a hitpass object is and why it’s needed.

Varnish can pass objects – that is to say, deliver an object without storing it in the cache – at two different stages in the request handling. The first and simplest stage you can pass at is before you have contacted the web server. This can be because you know that every page under a specific url should not be cached, or that a cookie is present and needed. This is done in vcl_recv. The slightly more complex stage to  purge at is after Varnish has gotten a reply from the backend, typically because it sent a “Set-Cookie” header or otherwise had headers that indicate that the page isn’t cacheable. This is done in vcl_fetch.

Normally Varnish tries to only request a specific object once, since it will look the same to all users there is no reason to make the web server generate it multiple times, and it’s definitely not faster either. When you are passing an object, every client request has to go to the web server, so serializing the requests for the same url is not wanted.

If you pass in recv, it’s easy for Varnish to avoid this serialization . It already knows that the next request for the same URL will have to go to the backend. Varnish makes an “anonymous” object that is never entered into the cache. A very simple and effective design.

But in vcl_fetch, things are not so easy. Varnish can’t predict a pass in vcl_fetch ahead of time, so it has to serialize these requests as if it was being cached. This is obviously not a desirable situation. Enter the hitpass object.

To avoid serialized requests after a pass in vcl_fetch, Varnish makes a hitpass object which is entered into the cache for ‘obj.ttl’ period of time, which simply says that this url will be passed for the next obj.ttl period of time. This looks almost just like a normal object in the cache, but it has no content, just a flag telling Varnish to fetch it from a backend. And thus Varnish can perform parallel requests again, because it knows the request will not be cached before it contacts the web server. The moment Varnish sees a hitpass object, it will dereference (and thus unlock) the hitpass object and make an anonymous one, entering the same code-path as if it was a pass in vcl_recv.

Now, back to my original story and purging. What’s it got to do with hitpass objects? Well, the purges on this particular site is done in vcl_hit/vcl_miss by resetting ttl to 0, effectively expiring the object. The problem was that at some point, a header on the relevant front page had told Varnish not to cache. So a hitpass object was made, and it had a ttl of roughly 4 days. So the VCL never hit vcl_hit or vcl_miss – where the purging was being done – but instead went to vcl_pass and directly to the backend, which kindly generated the page even though it just got a PURGE request, which is a bit strange, but not all that strange. What’s worse, of course, is that the front page wasn’t cached at all for this period of time. As it turns out, Varnish also has a minor issue which I also discovered yesterday, in that it doesn’t actually purge hitpass objects correctly, at least not in this particular version of Varnish.

And if you are thinking “why not set the ttl to 0 in vcl_pass if you see a PURGE?”, even if we allowed you to modify obj.ttl in vcl_pass, it wouldn’t be the ttl of the hitpass object, but the anonymous object. This is also why you can’t simply decide to start caching it again – when you reach vcl_fetch after hitting a hitpass object, the object you are working on is anonymous, and never entered into the cache.

My solution? Add an ‘x’ to the cache hash for the particular url/host combination for an immediate “fix”. It was either that or restarting Varnish at peak hour. Oh, and add some sanity to the VCL to reduce the effect of this in the future.

Recommendations

Understand hitpass objects. If you are doing ‘pass’ in vcl_fetch, remember that the TTL matters. You may want to switch ‘pass’ in vcl_fetch with your own pass subroutine which just sets the ttl to 1 minute for instance, and use that function consistently. That way, you avoid long-lived hitpass objects, but still get the benefit of them when you want them.

Also: If you are using the artificial http “PURGE”-method to set ttl to zero in vcl_hit, don’t forget to check for it in vcl_pass. You probably want to throw a 503-error in vcl_pass if you see a PURGE request.


December 18, 2009

cd34 — Django CMS to support Varnish and Akamai ESI

Many years ago I ran into a situation with a client where the amount of traffic they were receiving was crushing their dynamically created site. Computation is always the enemy of a quick pageload, so, it is very important to do as little computation as possible when delivering a page.

While there are many ways to put together a CMS, high traffic CMS sites usually involve caching or lots of hardware. Some write static files which are much less strenuous, but, you lose some of the dynamic capabilities. Fragment caching becomes a method to make things a bit more dynamic as MasonHQ does with their page and block structure. Django-blocks was surely influenced by this or reinvented this method.

In order to get the highest performance out of a CMS with a page and block method, I had considered writing a filesystem or inode linklist that would allow the webserver to assemble the page by following the inodes on the disk to build the page. Obviously there are some issues here, but, if a block was updated by a process, it would automatically be reassembled. This emulates a write-through cache and would have provisions for dynamic content to be mixed in with the static content on disk. Assembly of the page still takes more compute cycles than a static file but is significantly less than dynamically creating the page from multiple queries.

That design seriously limits the ability to deploy the system widely. While I can control the hosting environment for personal projects, the CMS couldn’t gain wide acceptance. While Varnish is a rather simple piece of software to install, it does limit deploy-ability, but, provides a significant piece of the puzzle due to Edge Side Includes (ESI). If the CMS gets used beyond personal and small deployments, Akamai supports Edge Side Includes as well.

Rather than explain ESI, ESI Explained Simply contains about the best writeup I’ve seen to date to explain how ESI can be used.

The distinction here is using fragment caching controlled by ESI to represent different zones on the page. As a simple example, lets consider our page template contains an article and a block with the top five articles on the site. When a new post is added, we can expire the block that contains the top five articles so that it is requested on the next page fetch. Since the existing article didn’t change, the interior ESI included block doesn’t need to be purged. This allows the page to be constructed on the Edge rather than on the Origin server.

As I have worked with a number of PHP frameworks, none really met my needs so I started using Python frameworks roughly two years ago. For this CMS, I debated using Pylons or Django and ended up choosing Django. Since both can be run behind WSGI compliant servers, we’ve opened ourselves up to a number of potential solutions. Since we are running Varnish in front of our Origin server, we can run Apache2 with mod_wsgi, but, we’re not limited to that configuration. At this point, we have a relatively generic configuration the CMS can run on, but, there are many other places we can adapt the configuration for our preferences.

Some of the potential caveats:
* With Varnish or Akamai as a frontend, we need to pay closer attention to X-Forwarded-For:
* Web logs won’t exist because Varnish is serving and assembling the pages (There is a trick using ESI that could be employed if logging was critical)
* ESI processed pages with Varnish are not compressed. This is on their wishlist.

Features:
* Content can exist in multiple categories or tags
* Flexible URL mapping
* Plugin architecture for Blocks and Elements
* Content will maintain revisions and by default allow comments and threaded comments

Terms:
* Template – the graphical layout of the page with minimal CMS markup
* Element – the graphical template that is used to render a Block
* Block – a module that generates the data rendered by an Element
* Page – a Page determined by a Title, Slug and elements
* Content – The actual data that rendered by a block

Goals:
* Flexible enough to handle something as simple as a personal blog, but, also capable of powering a highly trafficed site.
* Data storage of common elements to handle publishing of content and comments with the ability to store information to allow threaded comments. This would allow the CMS to handle a blog application, a CMS, or, a forum.
* A method to store ancillary data in a model so that upgrades to the existing database model will not affect developed plugins.
* Block system to allow prepackaged css/templating while allowing local replacement without affecting the default package.
* Upgrades through pypy or easy_install.
* Ability to add CDN/ESI without needing to modify templates. The system will run without needing to be behind Varnish, but, its full power won’t be realized without Varnish or Akamai in front of the origin server.
* Seamless integration of affiliate referral tracking and conversion statistics

At this point, the question in my mind was whether or not to start with an existing project and adapt it or start from scratch. At this point, the closest Django CMS I could find was Django-Blocks and I do intend to look it over fairly closely, but, a cursory look showed the authors were taking it in a slightly different direction than I anticipated. I’ll certainly look through the code again, but, the way I’ve envisioned this, I think there are some fundamental points that clash.

As I already have much of the database model written for an older PHP CMS that I wrote, I’m addressing some of the shortcomings I ran across with that design and modifying the models to be a little more generic. While I am sure there are proprietary products that currently utilize ESI, I believe my approach is unique and flexible enough to power everything from a blog to a site or forums or even a classified ads site.

December 02, 2009

cd34 — No ESI processing, first char not ‘

After installing Varnish 2.0.5 on a machine, ESI Includes didn’t work. When using varnishlog, the first error that occurred when debugging was:

No ESI processing, first char not ‘< '

   12 SessionClose – timeout
   12 StatSess     – 124.177.181.149 50662 4 0 0 0 0 0 0 0
   12 SessionOpen  c 68.212.183.136 60087 66.244.147.44:80
   12 ReqStart     c 68.212.183.136 60087 409391565
   12 RxRequest    c GET
   12 RxURL        c /esi.html
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: cd34.colocdn.com
   12 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2b4) Gecko/20091124 Firefox/3.6b4
   12 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   12 RxHeader     c Accept-Language: en-us,en;q=0.5
   12 RxHeader     c Accept-Encoding: gzip,deflate
   12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   12 RxHeader     c Keep-Alive: 115
   12 RxHeader     c Connection: keep-alive
   12 RxHeader     c X-lori-time-1: 1259718658980
   12 RxHeader     c Cache-Control: max-age=0
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 VCL_call     c miss
   12 VCL_return   c fetch
   12 Backend      c 14 cd34_com cd34_com
   12 ObjProtocol  c HTTP/1.1
   12 ObjStatus    c 200
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Wed, 02 Dec 2009 01:50:59 GMT
   12 ObjHeader    c Server: Apache
   12 ObjHeader    c Vary: Accept-Encoding
   12 ObjHeader    c Content-Encoding: gzip
   12 ObjHeader    c Content-Type: text/html
   12 TTL          c 409391565 RFC 120 1259718659 0 0 0 0
   12 VCL_call     c fetch
   12 TTL          c 409391565 VCL 43200 1259718659
   12 ESI_xmlerror c No ESI processing, first char not ‘< '
   12 TTL          c 409391565 VCL 0 1259718659
   12 VCL_info     c XID 409391565: obj.prefetch (-30) less than ttl (-1), ignored.
   12 VCL_return   c deliver
   12 Length       c 68
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Type: text/html
   12 TxHeader     c Content-Length: 68
   12 TxHeader     c Date: Wed, 02 Dec 2009 01:50:59 GMT
   12 TxHeader     c X-Varnish: 409391565
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 TxHeader     c X-Cache: MISS
   12 ReqEnd       c 409391565 1259718659.088263512 1259718659.127703667 0.000059366 0.039401770 0.000038385
   12 Debug        c "herding"

ESI received significant performance enhancements in 2.0.4 and 2.0.5 so, it seemed something was incompatible. Downgrading to 2.0.3 and using the VCL from another machine still resulted in ESI not working.

In this case, mod_deflate was running on the backend which was causing the issue. However, in reading the source code, it appears that message could also occur if your ESI include wasn’t handing back properly formed XML/HTML content. If your include doesn’t contain valid content and is only returning a small snippet, you might consider passing:

-p esi_syntax=0x1

on the command line that starts Varnish.

The changes in Varnish address the issue of ESI being enabled on binary content. Since the first character isn’t an < in almost all binary files (jpg, mpg, gif) and isn't the start of most .css/.js files, varnish doesn't need to spend extra time checking those files for includes. While you can and should selectively enable esi processing, this is just an added safeguard and a performance boost to compensate for vcl that might have an esi directive on static/binary content.

Since Varnish 2.0.3 now worked properly with the new machine, we upgraded to Varnish 2.0.5 which introduced a very odd issue:

[Tue Dec 01 20:58:11 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.htmlt
[Tue Dec 01 20:58:13 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html7
[Tue Dec 01 20:58:24 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xfa
[Tue Dec 01 20:59:01 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xb5
[Tue Dec 01 20:59:06 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xe7
[Tue Dec 01 20:59:07 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xd4
[Tue Dec 01 20:59:08 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\x1c

This generated 404s on the piece of the page that contained the ESI include. Downgrading to 2.0.4 fixed the issue and the issue appears to already be fixed in Trunk. Varnish Ticket #585

Varnish 2.0.4 and mod_deflate disabled addressed the two issues that prevented ESI from working correctly on this new installation.

November 10, 2009

Ingvar Hagelund — varnish-2.0.5 released, rpm packages updated

Varnish is a state of the art high performance HTTP accelerator. varnish-2.0.5 was released yesterday.

I have submitted varnish-2.0.5 for Fedora and Fedora EPEL, and updates to the stable releases will be requested, so they will trickle down to the stable repos in a few weeks.

For RHEL and derivates, updates for both el4 and el5 are in the EPEL testing repo. For those who are too impatient to wait for stable, or want to participate in testing, you can update the package with yum:

rhel5# yum --enablerepo=epel-testing update varnish

… or download the package from RedHat:

http://download.fedora.redhat.com/pub/epel/testing/

Fedora packages are still pending for testing, but will be visible in a
few days, I guess. If you need packages for Fedora now, try
http://kojipkgs.fedoraproject.org/packages/varnish/.

Bugs in the package can be reported in Red Hat’s Bugzilla:
http://bugzilla.redhat.com/ or to varnish-dist(a)projects.linpro.no.

October 11, 2009

cd34 — Converting to a Varnish CDN with WordPress

While working with Varnish I decided to try an experiment. I knew that Varnish could assist sites, but, it has never been easy to run Varnish on a shared virtual or clustered virtual host. VPS or Dedicated servers are no problem because you can do some configuration. However, in this case, I wanted to see if we could use Varnish to emulate a CDN, and if so, how difficult would it be for wordpress.

As it turns out, WordPress has a particular capability built in that handles media uploads. In the admin, under Settings, Miscellaneous, there are two values. One that asks where uploads should be stored. That path is a relative path under your blog’s home directory. The second is the URL that points to that path. In most cases you need to leave this blank, but, we can use that to point the URL for images to use the CDN.

Settings, Miscellaneous

Store uploads in this folder: wp-content/uploads
Full URL path to files: http://cd34.colocdn.com/blog/wp-content/uploads

Second, all of the images that have been already posted need to have their URLs modified. Since I am a command line guy, I executed the following command in MySQL.

update wp_posts set post_content=replace(post_content,'http://cd34.com/blog/wp-content/uploads/','http://cd34.colocdn.com/blog/wp-content/uploads/');

According to the Yahoo YSlow plugin, my blog went from a 72 to a 98 out of 100 with this and a few other modifications. The site does appear to be much snappier as well.

July 27, 2009

cd34 — ESI Widget Issues in the Varnish, ESI, WordPress experiment

The administration interface is quite simple. When the widget is installed, drag it to the Sidebar, then, drag any widgets that you want displayed to the ESI Widget Sidebar.

esi-widget

Current issues:
* When a user is logged in and comments on a post, their ‘login’ information is left on the page if they are the first person to hit the page when Varnish caches the page. If someone is logged in and visits a post page and the page hasn’t been previously cached, the html that shows their login status is cached, though, new visitors see the information, but lack the credentials.

Addons that don’t work properly:
* Any poll application (possible solution to wrap widget in an ESI block)
* Any stat application (unless they convert to a webbug tracker, this probably cannot be fixed easily)
* Any advertisement/banner rotator that runs internal. OpenX will work, as will most non-plugin
* Any postcount/postviews addon
* CommentLuv?
* ExecPHP (will cache the output, but does work)
* Manageable

Any plugin that does something at the time of the post or comment phase, that isn’t dependent on the logged in data should work without a problem. If it requires a login, or uses the IP address to determine whether a visitor has performed an action, will have a problem due to the excessive caching. For sites where the content is needed to be served quickly and there aren’t many comments, ESI Widget would work well.

Because of the way Varnish works, you wouldn’t necessarily have to run Varnish on the server running WordPress. Point the DNS at the Varnish server and set the backend for the host to your WordPress server’s IP address and you can have a Varnish server across the country caching your blog.

July 22, 2009

cd34 — WordPress, Varnish and Edge Side Includes

While talking about WordPress and it’s abysmal performance in high traffic situations to a client, we started looking back at Varnish and other solutions to keep their machine responsive. Since most of the caching solutions generate a page, serve it and cache it, posts and comments tend to lag behind the cache. db-cache does work around this by caching the query objects so that the pages can be generated more quickly and does expire the cache when tables are updated, but, its performance is still lacking. Using APC’s opcode cache or memcached just seemed to add complexity to the overall solution.

Sites like perezhilton.com appear to run behind multiple servers running Varnish, use wp-cache, move the images off to a CDN which results in a 3 request per second site with an 18 second pageload. Varnish’s cache always shows an age of 0 meaning Varnish is acting more as a load balancer than a front-end cache.

Caching isn’t without its downside. Your weblogs will not represent the true traffic. Since Varnish intercepts and serves requests before they get to the backend, those hits never hit the log. Forget pageview/postview stats (even with addons) because the addon won’t get loaded except during caching. Certain Widgets that rely on cookies or IP addresses will need to be modified. A workaround is to use a Text Box Widget and do an ESI include of the widget. For this client, we needed only some of the basic widgets. The hits in the apache logs will come from an IP of 127.0.0.1. Adjust your apache configuration to show the X-Forwarded-For IP address in the logs. If you truly need statistics, you’ll need to use something like Google Analytics. Put their code outside your page elements so that waiting for that javascript to load doesn’t slow down the rendering in the browser.

The test site, http://varnish.cd34.com/ is running Varnish 2.0.4, Apache2-mpm-prefork 2.2.11, Debian/Testing, WordPress 2.8.2. I’ve loaded the default .xml import for testing templates so that there were posts with varied dates and construction in the site. To replicate the client’s site, the following Widgets were added the sidebar: Search, Archives, Categories, Pages, Recent Posts, Tag Cloud, Calendar. Calendar isn’t in the existing site, but, since it is a very ‘expensive’ SQL query to run, it made for a good benchmark.

The demo site is running on:

model name	: Intel(R) Celeron(R) CPU 2.40GHz
stepping	: 9
cpu MHz		: 2400.389
cache size	: 128 KB

with a Western Digital 80gb 7200RPM IDE drive. Since all of the benchmarking was done on the same machine without any config changes taking place between tests, our benchmarks should represent as even a test base as we can expect.

Regrettably, our underpowered machine couldn’t run the benchmark with 50 concurrent tests, nor, could it run the benchmarks with the Calendar Widget enabled. In order to get apachebench to run, we had to bump the number of requests down and reduce the number of concurrent tests.

These results are from Apache without Varnish.

Server Software:        Apache
Server Hostname:        varnish.cd34.com
Server Port:            80

Document Path:          /
Document Length:        43903 bytes

Concurrency Level:      10
Time taken for tests:   159.210 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      4408200 bytes
HTML transferred:       4390300 bytes
Requests per second:    0.63 [#/sec] (mean)
Time per request:       15921.022 [ms] (mean)
Time per request:       1592.102 [ms] (mean, across all concurrent requests)
Transfer rate:          27.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   7.0      0      25
Processing: 14785 15863 450.2  15841   17142
Waiting:     8209 8686 363.4   8517    9708
Total:      14785 15865 451.4  15841   17142

Percentage of the requests served within a certain time (ms)
  50%  15841
  66%  15975
  75%  16109
  80%  16153
  90%  16628
  95%  16836
  98%  17001
  99%  17142
 100%  17142 (longest request)

Normally we would have run the Varnish enabled test without the Calendar Widget, but, I felt confident enough to run the test with the widget in the sidebar. Varnish was configured with a 12 hour cache (yes, I know, I’ll address that later) and the ESI Widget was loaded.

Server Software:        Apache
Server Hostname:        varnish.cd34.com
Server Port:            80

Document Path:          /
Document Length:        45544 bytes

Concurrency Level:      50
Time taken for tests:   18.607 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      457980000 bytes
HTML transferred:       455440000 bytes
Requests per second:    537.44 [#/sec] (mean)
Time per request:       93.034 [ms] (mean)
Time per request:       1.861 [ms] (mean, across all concurrent requests)
Transfer rate:          24036.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.8      0      42
Processing:     1   92  46.2    105     451
Waiting:        0   91  45.8    104     228
Total:          2   93  46.0    105     451

Percentage of the requests served within a certain time (ms)
  50%    105
  66%    117
  75%    123
  80%    128
  90%    142
  95%    155
  98%    171
  99%    181
 100%    451 (longest request)

As you can see, even with the aging hardware, we went from .63 requests per second to 537.44 requests per second.

But, more about that 12 hour cache. The ESI Widget uses an Edge Side Include to include the sidebar into the template. Rather than just cache the entire page, we instruct Varnish to cache the page and include the sidebar. As a result, when a person surfs the site and goes from the front page to a post page, the sidebar doesn’t need to be regenerated when they go to the 2nd page. With wp-cache, it would have regenerated the sidebar Widgets and then cached the resulting page. Obviously, that 12 hour cache is going to affect the usability of the site, so, ESI widget purges the sidebar, front page and post page any time a post is updated or deleted or commented on. Voila, even with a long cache time, we are presented with a site that is dynamic and not delayed until wp-cache’s page cache expires. As this widget is a concept, I’m sure a little intelligence can be added to prevent the excessive purging in some cases, but, it does handle things reasonably well. There are some issues not currently handled with the ESI including how to handle users that are logged for comments. With some template modifications, I think those pieces can be handled with ESI to provide a lightweight method for the authentication portion.

While I have seen other sites mention Varnish and other methods to keep your wordpress installation alive in high traffic, I believe this approach is a step in the right direction. With the ESI widget, you can focus on your site, and let the server do the hard work. This methodology is based on a CMS that I have contemplated writing for many years, though, using Varnish rather than static files.

It is a concept developed in roughly four hours including the time to write the widget and do the benchmarking. It isn’t perfect, but does address the immediate needs of the one client. I think we can consider this concept a success.

If you don’t have the ability to modify your system to run Varnish, then you would be limited to running wp-cache and db-cache. If you can connect to a memcached server, you might consider running Memcached for WordPress as it will make quite a difference as well.

This blog site, http://cd34.com/blog/ is not running behind Varnish. To see the Varnish enabled site with ESI Widget, go to http://varnish.cd34.com/

Software Mentioned:

* Varnish ESI and Purge and Varnish’s suggestions for helping WordPress
* WordPress
* wp-cache
* db-cache

Sites used for reference:

* Supercharge WordPress
* SSI, Memcached and Nginx (with mentions of a Varnish/ESI configuration)

Varnish configuration used for ESI-Widget:

backend default {
.host = "127.0.0.1";
.port = "81";
}

sub vcl_recv {
 if (req.request == "PURGE") {
     purge("req.url == " req.url);
 }

 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
    unset req.http.cookie;
  }
  if (!(req.url ~ "wp-(login|admin)")) {
    unset req.http.cookie;
  }
}

sub vcl_fetch {
   set obj.ttl = 12h;
   if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
      set obj.ttl = 24 h;
   } else {
      esi;  /* Do ESI processing */
   }
}

June 28, 2009

cd34 — Varnish and Nginx with Joomla

Recently we had a client that had some performance issues with a Joomla installation. The site wasn’t getting an incredible amount of traffic, but, the traffic it was getting was just absolutely overloading the server.

Since the machine hadn’t been having issues before, the first thing we did was contact the client and ask what had changed. We already knew the site and database that was using most of the CPU time, but, the bandwidth graph didn’t suggest that it was traffic overrunning the server. Our client rescued this client from another hosting company because the site was unusable in during prime time. So, we’ve inherited a problem. During the move, the site was upgraded from 1.0 to 1.5, so, we didn’t even have a decent baseline to revert to.

The stopgap solution was to move the .htaccess mod_rewrite rules into the apache configuration which helped somewhat. We identified a few sections of the code that were getting hit really hard and wrote a mod_rewrite rule to serve those images direct from disk — bypassing Joomla serving those images through itself. This made a large impact and at least got the site responsive enough that we could leave it online and work through the admin to figure out what had gone wrong.

Some of the modules that had been enabled contributed to quite a bit of the performance headache. One chat module generated 404s every second for each person logged in to see if there were any pending messages. Since Joomla is loaded for each 404 file, this added quite a bit of extra processing. Another quick modification to the configuration eliminated dozens of bad requests. At this point, the server is responsive, the client is happy and we make notes in the trouble ticket system and our internal documentation for reference.

Three days later the machine alerts and our load problem is back. After all of the changes, something is still having problems. Upon deeper inspection, we find that portions of the system dealing with the menus are being recreated each time. There’s no built in caching, so, the decision is to try Varnish. Varnish has worked in the past for WordPress sites that have gotten hit hard, so, we figured if we could cache the images, css and some of the static pages that don’t require authentication, we can get the server to be responsive again.

Apart from the basic configuration, our varnish.vcl file looked like this:

sub vcl_recv {
  if (req.http.host ~ "^(www.)?domain.com$") {
     set req.http.host = "domain.com";
  }

 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
    unset req.http.cookie;
  }
}

sub vcl_fetch {
 set obj.ttl = 60s;
 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
      set obj.ttl = 3600s;
 }
}

To get the apache logs to report the IP, you need to modify the VirtualHost config to log the forwarded IP.

The performance of the site after running Varnish in front of Apache was quite good. Apache was left with handling only .php and the server is again responsive. It runs like this for a week or more without any issues and only a slight load spike here or there.

However, Joomla doesn’t like the fact that every request’s REMOTE_ADDR is 127.0.0.1 and some addons stop working. In particular an application that allows the client to upload .pdf files into a library requires a valid IP address for some reason. Another module to add a sub-administration panel for a manager/editor also requires an IP address other than 127.0.0.1.

With some reservation, we decide to switch to Nginx + FastCGI which removes the reverse proxy and should fix the IP address problems.

Our configuration for Nginx with Joomla:

server {
        listen 66.55.44.33:80;
	server_name  www.domain.com;
 	rewrite ^(.*) http://domain.com$1 permanent;
}
server {
        listen 66.55.44.33:80;
	server_name  domain.com;

	access_log  /var/log/nginx/domain.com-access.log;

	location / {
		root   /var/www/domain.com;
		index  index.html index.htm index.php;

           if ( !-e $request_filename ) {
             rewrite (/|\.php|\.html|\.htm|\.feed|\.pdf|\.raw|/[^.]*)$ /index.php last;
             break;
           }

	}

	error_page   500 502 503 504  /50x.html;
	location = /50x.html {
		root   /var/www/nginx-default;
	}

	location ~ \.php$ {
		fastcgi_pass   unix:/tmp/php-fastcgi.socket;
		fastcgi_index  index.php;
		fastcgi_param  SCRIPT_FILENAME  /var/www/domain.com/$fastcgi_script_name;
		include	fastcgi_params;
	}

        location = /modules/mod_oneononechat/chatfiles/ {
           if ( !-e $request_filename ) {
             return 404;
           }
        }
}

With this configuration, Joomla was handed any URL for a file that didn’t exist. This was to allow the Search Engine Friendly (SEF) links. The second 404 handler was to handle the oneononechat module which looks for messages destined for the logged in user.

With Nginx, the site is again responsive. Load spikes occur from time to time, but, the site is stable and has a lot less trouble dealing with the load. However, once in a while the load spikes, but, the server seems to recover pretty well.

However, a module called Rokmenu which was included with the template design appears to have issues. Running php behind FastCGI sometimes gives different results than running as mod_php and it appears that Rokmenu is relying on the path being passed and doesn’t normalize it properly. So, when the menu is generated, with SEF on or off, urls look like /index.php/index.php/index.php/components/com_docman/themes/default/images/icons/16×16/pdf.png.

Obviously this creates a broken link and causes more 404s. We installed a fresh Joomla on Apache, imported the data from the copy running on Nginx, and Apache with mod_php appears to work properly. However, the performance is quite poor.

In order to troubleshoot, we made a list of every addon and ran through some debugging. With apachebench, we wrote up a quick command line that could be pasted in at the ssh prompt and decided upon some metrics. Within minutes, our first test revealed 90% of our performance issue. Two of the addons required compatibility mode because they were written for 1.0 and hadn’t been updated. Turning on compatibility mode on our freshly installed site resulted in 10x worse performance. As a test, we disabled the two modules that relied on compatibility mode and turned off compatibility mode and the load dropped immensely. We had disabled SEF early on thinking it might be the issue, but, we found the performance problem almost immediately. Enabling other modules and subsequent tests showed marginal performance changes. Compatibility mode was our culprit the entire time.

The client started a search for two modules to replace the two that required compatibility mode and disabled them temporarily while we moved the site back to Apache to fix the url issue in Rokmenu. At this point, the site was responsive, though, pageloads with lots of images were not as quick as they had been with Nginx or Varnish. At a later point, images and static files will be served from Nginx or Varnish, but, the site is fairly responsive and handles the load spikes reasonably well when Googlebot or another spider hits.

In the end the site ended up running on Apache because Varnish and Nginx had minor issues with the deployment. Moving to Apache alternatives doesn’t always fix everything and may introduce side-effects that you cannot work around.

May 20, 2009

Ingvar Hagelund — The usage of Varnish revisited

Varnish is a high-performance HTTP accelerator. Working with Varnish is part of my day job. Among other things, I maintain the packages for Fedora and EPEL.

Some months have passed, and it is time to run my poking scripts again, looking for sites that run Varnish. There is no deep magic here. I just parse the available top lists that I know of, and peek at the HTML headers of the sites that are listed. If there are subsites linked from the front page of the site, I scan them too. This means that twitter.com shows up, though Twitter only runs Varnish on its search site. Subsites with a Varnish match are shown in parenthesis in the results.

For the Nordic countries, I have found quite good lists, that is, upload result lists from the probably most visited media sites in the respective countries. Remember of course, that these are generally pay-to-be-included lists, and there may exist sites with far more hits than the ones listed.

For the rest, I have used Alexa and Netcraft’s toolbar users’ lists with main sites summed up. With a few exceptions, they are a bit less interesting, as global .com and .net sites tend to cover most of the top 100 entries anyway.

Now for the results. Varnish is sponsored by large Norwegian sites, so it is no big surprise that there are a lot of hits in Norway. Of the TNS Gallup top list, Varnish runs at some every third site.

For Denmark, I use FDIM’s list. The results are more or less the same as in my last run. For Finland, I use TNS’ numbers again. There are +1 site in the TNS top list. For Sweden, I use the KIA Index list. There is -1 Varnish site there since last.

For a global overview, I have used Alexa’s top 500 list. This is the first time I have got any hit on that list. That is interesting. Twitter is probably the most well known site outside the geekosphere running Varnish now.

The whole World: Alexa's global top 500 list
Place  39 Varnish running on twitter.com (search.twitter.com)
Place  83 Varnish running on globo.com (globo.com)
Place 166 Varnish running on ifeng.com (ifeng.com)
Place 194 Varnish running on people.com.cn (bbs1.people.com.cn)
Place 227 Varnish running on hulu.com (hulu.com)
Place 282 Varnish running on wikia.com (wikia.com)
Place 467 Varnish running on perezhilton.com (perezhilton.com)

Probing the rest of the World would probably take too much time, so I have done a scan of Alexa’s and Netcraft’s lists for their toolbar users in the European countries.

All the gory details are available here.

Other worth mentioned sites that uses Varnish that does not show up on my Alexa’s global list may be Slashdot, WAT TV, The Pirate Bay, JDownloader, e.Republik, WOWwiki, the Brazilian news site Globo.com, the German IDG site PC Welt, and the Chinese computer tech site Csdn.net.

Do you know of other famous sites running Varnish? Use the comments.

February 02, 2009

Ingvar Hagelund — The usage of the varnish in the Nordic countries

The Varnish http accellerator has gained some general usage in the Nordic countries over the last months.

In Norway, using the top list from TNS Gallup , varnish now runs on 13 of the top 30, and 27 of the top 100 sites on the list. Including in these results are also major sites finn.no and nrk.no, new since my last probes.

In Denmark, using the top list from FDIM, varnish runs on 3 sites in the top 15.

In Finland, using the top list from TNS Gallup, varnish runs on 4 of the top 100 sites.

And finally, in Sweden, using the top list from the KIA-index, varnish runs on 7 sites in the top 100, and 13 sites of the whole list.

Remember of course, that these top sites are generally pay-to-be-included lists, and there may exist sites with far more hits than the ones on the lists. yr.no for instance, a very popular Norwegian weather site, does use varnish, but is not included on the TNS Gallup list.

For reference, I have also done som simple non-weighted probing of the most hitted web sites for netcraft toolbar users, showing probably a bit more geeky cut of the browsing population, actually showing that yr.no and the infamous Pirate Bay also uses varnish.

Disclaimer: Working with the developers of Varnish is part of my day job.

All details below.

#
# Norway
#
# TNS Gallup Topplisten
Place   1 Varnish running on vg.no (sport.vg.no) (skatt.vg.no) (interaktiv.vg.no) (www.vg.no) (www1.vg.no)
Place   4 Varnish running on startsiden.no (img.startsiden.no) (www.startsiden.no)
Place   5 Varnish running on nrk.no (www1.nrk.no) (www.nrk.no)
Place   6 Varnish running on finn.no (cache.finn.no)
Place  10 Varnish running on aftenposten.no (www.aftenposten.no)
Place  14 Varnish running on abcnyheter.no (www.abcnyheter.no)
Place  16 Varnish running on klikk.no (www.klikk.no) (skatt.klikk.no)
Place  20 Varnish running on e24.no (www.e24.no)
Place  23 Varnish running on bt.no (www.bt.no) (fotball.bt.no) (images.bt.no)
Place  24 Varnish running on dn.no (multimedia.dn.no)
Place  26 Varnish running on adressa.no (kundeservice.adressa.no) (www.adressa.no) (fotball.adressa.no) (berlin.adressa.no)
Place  28 Varnish running on dinepenger.no (www.dinepenger.no)
Place  30 Varnish running on ba.no (www.ba.no) (bildeserier.ba.no)
Place  31 Varnish running on aftenbladet.no (fotball.aftenbladet.no)
Place  41 Varnish running on rb.no (www.rb.no) (tjenester.rb.no)
Place  47 Varnish running on speaker.no (live.speaker.no)
Place  49 Varnish running on kvinneguiden.no (bilder.kvinneguiden.no)
Place  52 Varnish running on h-avis.no (livefotball.h-avis.no)
Place  59 Varnish running on nordlys.no (www.nordlys.no)
Place  66 Varnish running on idg.no (www.idg.no)
Place  69 Varnish running on budstikka.no (livefotball.budstikka.no) (www.budstikka.no)
Place  75 Varnish running on tvnorge.no (limefiles.tvnorge.no) (www.tvnorge.no)
Place  81 Varnish running on tu.no (www1.tu.no) (www.tu.no) (web.tu.no)
Place  82 Varnish running on vl.no (www.vl.no)
Place  87 Varnish running on nationen.no (www.nationen.no)
Place  92 Varnish running on rbk.no (www.rbk.no)
Place  98 Varnish running on journalisten.no (www.journalisten.no)
Place 103 Varnish running on dagligvarehandelen.no (www.dagligvarehandelen.no)

# Norwegian Netcraft toolbar users
Place   1 Varnish running on www.startsiden.no (www.startsiden.no)
Place   2 Varnish running on www.vg.no (www.vg.no)
Place   4 Varnish running on www.aftenposten.no (www.aftenposten.no)
Place   9 Varnish running on www.yr.no (www.yr.no)
Place  14 Varnish running on www.bt.no (www.bt.no)
Place  15 Varnish running on e24.no (e24.no)
Place  16 Varnish running on sokm.startsiden.no (sokm.startsiden.no)
Place  20 Varnish running on www.abcnyheter.no (www.abcnyheter.no)
Place  21 Varnish running on www.adressa.no (www.adressa.no)
Place  23 Varnish running on www.nrk.no (www.nrk.no)
Place  30 Varnish running on www1.vg.no (www1.vg.no)
Place  46 Varnish running on www.ba.no (www.ba.no)
Place  50 Varnish running on nrk.no (nrk.no) (www1.nrk.no) (www.nrk.no)
Place  51 Varnish running on vg.no (www.vg.no) (vg.no)
Place  57 Varnish running on www1.nrk.no (www1.nrk.no)
Place  64 Varnish running on go.nrk.no (go.nrk.no)
Place  68 Varnish running on www.norman.com (www.norman.com)
Place  79 Varnish running on www.op.no (www.op.no)
Place  87 Varnish running on www.klikk.no (www.klikk.no)
Place  88 Varnish running on atvs.vg.no (atvs.vg.no)

#
# Denmark
#
# FDIM toplist
Place   5 Varnish running on ekstrabladet.dk (multimedia.ekstrabladet.dk)
Place   9 Varnish running on dmi.dk (www.dmi.dk) (dmi.dk)
Place  14 Varnish running on jp.dk (multimedia.jp.dk)
Place 110 Varnish running on slankedoktor.dk (slankedoktor.dk)

# Danish Netcraft toolbar users
Place   4 Varnish running on ekstrabladet.dk (multimedia.ekstrabladet.dk)
Place   6 Varnish running on www.dmi.dk (www.dmi.dk)
Place  11 Varnish running on jp.dk (multimedia.jp.dk)
Place  18 Varnish running on www.komogvind.dk (www.komogvind.dk)
Place  58 Varnish running on www.spielmit.com (www.spielmit.com)

#
# Finland
#
# TNS Gallup Finland
Place  56 Varnish running on talouselama.fi (www.talouselama.fi) (talouselama.fi)
Place  61 Varnish running on tekniikkatalous.fi (tekniikkatalous.fi) (www.tekniikkatalous.fi)
Place  67 Varnish running on arvopaperi.fi (arvopaperi.fi) (www.arvopaperi.fi)
Place  96 Varnish running on marmai.fi (marmai.fi) (www.marmai.fi)
Place 124 Varnish running on mediuutiset.fi (mediuutiset.fi) (www.mediuutiset.fi)

# Finnish Netcraft toolbar users
Place  13 Varnish running on apareena.arvopaperi.fi (apareena.arvopaperi.fi)

#
# Sweden
#
# KIA-Index
Place  20 Varnish running on Sydsvenskan.se (Sydsvenskan.se)
Place  39 Varnish running on hd.se (websvcc.hd.se) (coachen.hd.se) (misc.hd.se) (media.hd.se) (hd.se)
Place  48 Varnish running on alltomstockholm.se (www.alltomstockholm.se) (alltomstockholm.se)
Place  81 Varnish running on nyteknik.se (www.nyteknik.se) (nyteknik.se)
Place  83 Varnish running on affarsvarlden.se (affarsvarlden.se) (www.affarsvarlden.se)
Place  90 Varnish running on nwt.se (nwt.se)
Place  99 Varnish running on bt.se (www.bt.se)
Place 102 Varnish running on smp.se (www.smp.se) (smp.se)
Place 112 Varnish running on barometern.se (www.barometern.se) (barometern.se)
Place 113 Varnish running on norran.se (vader.norran.se) (norran.se)
Place 130 Varnish running on blt.se (www.blt.se) (blt.se)
Place 245 Varnish running on pchemma.se (pchemma.se) (www.pchemma.se)
Place 250 Varnish running on ut.se (ut.se) (www.ut.se)

# Swedish Netcraft toolbar users
Place   6 Varnish running on www.aftonbladet.se (www.aftonbladet.se)
Place  24 Varnish running on www.svd.se (www.svd.se)
Place  29 Varnish running on torrents.thepiratebay.org (torrents.thepiratebay.org)
Place  38 Varnish running on aftonbladet.se (aftonbladet.se) (www.aftonbladet.se)
Place  44 Varnish running on www.e24.se (www.e24.se)
Place  63 Varnish running on www.tietoviikko.fi (www.tietoviikko.fi)

October 20, 2008

Ingvar Hagelund — Varnish in EPEL – packages for Red Hat Enterprise Linux and clones

I requested EPEL EL-4 and EL-5 tag and build for varnish on friday night. The packages will be included in the main EPEL repos at the next sync from testing, that is primo November. These packages are the same as we use in production, and should be quite ready for general usage. Packages are available at http://download.fedora.redhat.com/pub/epel/testing/ . There is a list of known bugs that will be fixed eventually in the 2.0 series at http://varnish.projects.linpro.no/wiki/TroubleLog

Please report any problems as tickets in Trac at http://varnish.projects.linpro.no/ (you’ll to register and login to file tickets). Bugs reported by RedHat’s Bugzilla will end up the same place over time, but we would prefer our the local ticket system.

October 15, 2008

Ingvar Hagelund — Who is using Varnish in Scandinavia?

Varnish is a high performance http accelerator.

As varnish-2.0 hopefully will be released in a few hours, it is time to check out who is using it again. I fixed up my small web usage toplist parsing scripts, and ran them looking for varnish. The sources are, for Norway, TNS Gallup’s top 100, for Denmark, FDIM’s top 250, and KIAIndex for Sweden. For comparison, I also used Alexa’s country lists.

Now, the results for this search are of course heavily coloured by the quality of the lists in use. TNS Gallup, FDIM, and KIAIndex’s lists are result of reported hits from customers/members. Sites that for various reasons, like, they do not want to pay for the service, do not report in, are of course not included in the lists. We know, for example that yr.no, an extremely popular weather site, uses varnish for some of their services, but is not listed. The same goes for the Pirate Bay. They are not included here. Alexa’s lists are based on the usage of their web browser tool bar. Who uses their tool bar. Most people do not, I would guess. Also, some of the numbers are old. FDIM’s last update is from August. And my parsing of KIAIndex is ugly and faulty at the best. Still, the results may give a rough hint of the usage of Varnish in Scandinavia.

Topplisten TNS Gallup (Norway)

Place   1 Varnish running on vg.no (interaktiv.vg.no) (www.vg.no) (sport.vg.no)  (atvs.vg.no) (www1.vg.no)
Place   4 Varnish running on startsiden.no (www.startsiden.no) (img.startsiden.no)
Place  10 Varnish running on aftenposten.no (www.aftenposten.no)
Place  14 Varnish running on e24.no (www.e24.no)
Place  17 Varnish running on abcnyheter.no (www.abcnyheter.no)
Place  22 Varnish running on klikk.no (www.klikk.no) (skatt.klikk.no)
Place  23 Varnish running on dn.no (multimedia.dn.no)
Place  25 Varnish running on bt.no (fotball.bt.no)
Place  27 Varnish running on adressa.no (kundeservice.adressa.no) (www.adressa.no) (fotball.adressa.no) (berlin.adressa.no)
Place  28 Varnish running on dinepenger.no (www.dinepenger.no)
Place  31 Varnish running on aftenbladet.no (fotball.aftenbladet.no)
Place  34 Varnish running on speaker.no (live.speaker.no)
Place  45 Varnish running on kvinneguiden.no (bilder.kvinneguiden.no)
Place  46 Varnish running on rb.no (tjenester.rb.no)
Place  48 Varnish running on amobil.no (www.amobil.no)
Place  54 Varnish running on idg.no (www.idg.no)
Place  62 Varnish running on tvnorge.no (limefiles.tvnorge.no) (www.tvnorge.no)
Place  63 Varnish running on budstikka.no (livefotball.budstikka.no)
Place  76 Varnish running on tu.no (www1.tu.no) (www.tu.no) (web.tu.no)
Place  79 Varnish running on rbk.no (www.rbk.no) (forum.rbk.no)
Place 101 Varnish running on dagligvarehandelen.no (www.dagligvarehandelen.no)

Comparison: .no-domains in Alexa's top list for Norwegian users

Place   2 Varnish running on http://vg.no
Place   6 Varnish running on http://startsiden.no
Place   8 Varnish running on http://aftenposten.no
Place  12 Varnish running on http://e24.no
Place  21 Varnish running on http://abcnyheter.no
Place  36 Varnish running on http://adressa.no

Foreningen af danske interaktive medier (Denmark) top 250, August numbers:

Place   4 Varnish running on dmi.dk (www.dmi.dk) (dmi.dk)
Place   7 Varnish running on ekstrabladet.dk (multimedia.ekstrabladet.dk)
Place  15 Varnish running on jp.dk (jp.dk) (multimedia.jp.dk)
Place  24 Varnish running on epn.dk (epn.dk)
Place  34 Varnish running on fpn.dk (fpn.dk)
Place  56 Varnish running on newz.dk (newz.dk)
Place 105 Varnish running on filmz.dk (filmz.dk)

Comparison: .dk-domains in Alexa's top list for Danish users

Place  13 Varnish running on http://dmi.dk
Place  17 Varnish running on http://jp.dk
Place  21 Varnish running on http://komogvind.dk
Place  28 Varnish running on http://epn.dk

KIAindex (Sweden) top 300 or something

Place  12 Varnish running on e24.se (e24.se)
Place  15 Varnish running on svd.se (svd.se)
Place  23 Varnish running on Sydsvenskan.se (Sydsvenskan.se)
Place  37 Varnish running on hd.se (websvcc.hd.se) (blogg.hd.se) (misc.hd.se) (media.hd.se) (hd.se)
Place  61 Varnish running on affarsvarlden.se (affarsvarlden.se) (www.affarsvarlden.se)
Place  68 Varnish running on nyteknik.se (www.nyteknik.se) (nyteknik.se)
Place  73 Varnish running on nwt.se (nwt.se)
Place  79 Varnish running on bt.se (www.bt.se) (bt.se)
Place  80 Varnish running on smp.se (www.smp.se) (smp.se)
Place  83 Varnish running on barometern.se (www.barometern.se) (barometern.se)
Place  85 Varnish running on norran.se (vader.norran.se) (norran.se) (m.norran.se)
Place  92 Varnish running on blt.se (www.blt.se) (blt.se)
Place 165 Varnish running on pchemma.se (pchemma.se) (www.pchemma.se)
Place 170 Varnish running on ut.se (ut.se) (www.ut.se)

.se-domains in Alexa's top list for Swedish users

Place   2 Varnish running on http://aftonbladet.se
Place  18 Varnish running on http://affarsvarlden.se
Place  20 Varnish running on http://svd.se
Place  44 Varnish running on http://sydsvenskan.se

November 28, 2007

Ingvar Hagelund — Fri programvare på 79 av Norges 100 mest besøkte nettsteder

TNS-Gallup har en morsom liste med de mest besøkte nettstedene i Norge. Jeg tok utgangspunkt i denne lista fra uke 46 i år (2007). Ved å snuse litt på http-headeren til disse nettstedene, og litt hjelp av Netcraft kan jeg presentere følgende påstand:

Av Norges 100 mest besøkte nettsteder kjører 74 på fri programvare. Om man tar med alle med Linux som operativsystem stiger tallet til 79. Det er også artig at Varnish har tatt så stor markedsandel på så kort tid. Her er lista.

Apache: 60%
Apache + Varnish: 8%
Microsoft IIS: 21%
Oracle: 5% (alle på Linux)
Oracle + Varnish 1%
Resin: 2%
Resin + Varnish: 1%
Zope + Varnish: 1%
nginx + Varnish: 1%

Hvis vi videre ser på hvilke nettsteder som bruker hva, kommer det fram at Microsoft IIS ikke dukker opp på lista før nummer 20 (opplysingen 1881).