Kris Wallsmith

Symfony2 ESI error: File name too long

We recently came across a strange, intermittent error at OpenSky:

(36)File name too long: Cannot map GET /_internal… HTTP/1.1 to file

We use Varnish to cache large sections of our pages that include follow/unfollow buttons. If the user is following a certain curator we show the unfollow button, otherwise we show the follow button.

Our render tag looked like this:

{% render 'AppBundle:Curator:list' with
    { following: app.user.sortedFollowingSlugs }
    { standalone: true } %}

When the current user is following just a few curators this works fine. The app renders the curator list with follow/unfollow buttons accurate for the current user and Varnish caches it for us. However, when a user was following all or nearly all of our curators we see the “file name too long” error and Apache would render its 403 error template (no idea why it chooses 403 in this case).

Unpack the WTF

To understand why this happens we need to dig into how Symfony2 implements ESI. When ESI is enabled and your app sits behind a reverse proxy that supports ESI, any call to Twig’s {% render %} tag with the standalone option set to true will render an <esi:include /> tag instead of rendering the action inline. Included in the src attribute of that tag is an URL with all the values you passed to the {% render %} tag encoded in.

The routing pattern used for the ESI URL looks like this:

/{controller}/{path}.{_format}

In the case above the {controller} value would be AppBundle:Curator:list and the {path} value would be equivalent to the following:

php > echo http_build_query(array('following' => array('bar', 'foo')));
following%5B0%5D=bar&following%5B1%5D=foo

Now imagine what the {path} value will look like when a user is following 100+ curators. Big, right? Turns out it would be too big for Apache to handle as a filename, which cannot exceed 255 characters, hence the “file name too long” error.

Query String to the Rescue!

Fortunately the fix is easy and does not require any hacking on Symfony. By moving the potentially large parameter to the ESI URL’s query string our character limit jumps from 255 up to 8,190 — lots more room!

{% render 'AppBundle:Curator:list' {
    standalone: true,
    query: { following: app.user.sortedFollowingSlugs }
} %}

Of course you would also need to update CuratorController::listAction() to look for the following parameter in the query string rather than in the action arguments, but that’s an easy change.

Stay Curious

Symfony makes a lot of cool things very easy for you, but don’t become lazy. Open up the source code and figure out how it works, what technologies you’re using under the hood, and what constraints may be lurking in a dark corner somewhere, waiting to pounce.

Type Matters

json_encode(array()) --> []
json_encode((object) array()) --> {}

Hello Spork! (aka “Forking PHP…”)

A few months ago I was tasked with speeding up the upload of assets to the OpenSky CDN, which was taking a few minutes each deploy. I ended up dividing the upload into multiple processes using pcntl_fork() and bringing the total time of the upload down to a matter of seconds.

Since then I’ve been working on wrapping some of the complexities of working with a parent and child processes into an OO library and am happy to announce this experimental library: Spork.

Its usage is pretty straight forward.

<?php

use Spork\ProcessManager;
use Spork\Deferred\DeferredFactory;

$pm = new ProcessManager(new DeferredFactory());
$pm->fork(function() {
    // do something in a child process...
    echo posix_getpid();
})->then(function($output) {
    // do something in the parent process...
    printf('Parent %d forked child %d!', posix_getpid(), $output);
});

You can pass any callable into the process manager’s fork() method. In return you get a nice little deferred object which you can use to queue more callables to run after the child process exits. Just like the jQuery object I ♥ so much, there are three basic methods: always(), done(), and fail(), each of which accept a callable as an argument. There is also a then() method, which is a convenience method for adding a done and a fail callback in one method call.

Case Studies

If you end up using Spork I would like to hear about it. Please post your own blog and I’ll link to it, or send me a little description of what you’re doing and I’ll post it here.

I ♥ Event Delegation

jQuery event delegation is one of the coolest things since way cooler than sliced bread.

This new hotness allows you to listen for an event farther up the DOM than where that event is triggered. If you understand event bubbling, you should be able to grok this fairly quickly.

It starts getting interesting when you look at the implications of adding this tool to your toolkit. Let’s look at a concrete example.

Ye Ole List

What application would be complete without a list? We all know the deal: a list of things, each with a “remove” button and then an “add” button at the bottom. Without event delegation we would wire up all these remove buttons something like this:

var $list = $("#list");
$list.find("a.remove").click(function(e) {
    e.preventDefault();
    $(this).closest("#list > li").remove();
});

This looks nice — but looks can be deceiving. The .click() function loops over the list of elements inside the current jQuery object and applies a listener to each one. So if there are n elements, you will have added n listeners. That bloat starts to show when you get into adding more rows to the list:

var template = $list.data("template");
$("#add_item").click(function(e) {
    e.preventDefault();
    $(template.replace(/__name__/, $list.children().length))
        .appendTo($list)
        .find("a.remove")
        .click(function(e) {
            e.preventDefault();
            $(this).closest("#list > li").remove();
        });
});

The remove listener is now defined in two places: first in the initial setup and again in the setup of each new row. We could move this listener to its own variable and reuse it, but that would only address the symptom of more code.

The more important fix is to reduce the number of listeners from n to 1. This is possible with event delegation:

var $list = $("#list");
$list.on("click", "a.remove", function(e) {
    e.preventDefault();
    $(this).closest("#list > li").remove();
});

This little beauty tells the #list element to listen for click events that bubble up from any of its a.remove progeny and run those events through a single listener. Since the listener lives on a common ancestor now, there is no need to add this listener when creating a new row:

var template = $list.data("template");
$("#add_item").click(function(e) {
    e.preventDefault();
    $(template.replace(/__name__/, $list.children().length)).appendTo($list);
});

There you have it! Go forth and profit with less, more performant code!

If you need to be able to tell others in the business when they can expect cached pages to refresh:

$response->setPublic();
$response->setExpires(new \DateTime(sprintf('+%d seconds', 300 - time() % 300)));

$response->isNotModified($request)

This is one of my favorite methods in all of Symfony2 and it’s buried in the docs. Take a moment to glory in its simplicity:

public function someHeavyAction(Widget $widget, Request $request)
{
    $response = new Response();
    $response->setLastModified($widget->getUpdatedAt());

    if ($response->isNotModified($request)) {
        return $response;
    }

    // do some heavy lifting
    $response->setContent($this->renderView(/* ... */));

    return $response;
}

As you can see, this method makes implementing cache validation a snap. Once you’ve determined what you are going to use for validation — a last-modified timestamp and/or an e-tag — set that value on a new response object and this method will take care of the rest.

It accepts the current request as an argument and compares the cache headers coming in on the request with those you have set on the response. If the method returns true simply return the 304 response. If the method returns false, generate a fresh response.

It’s this sort of method that makes the Symfony2 HttpFoundation component the clear choice for anyone working with PHP and HTTP.

Faster PHPUnit

I made a simple optimization to the test suite at OpenSky over the weekend and we are reaping big benefits. The premise is pretty straight forward. We use the setUp() method to create a lot of mock objects through our test suite. If these are allowed to accumulate they end up wasting a lot of space and slowing down your suite.

Luckily there is also a tearDown() method you can use to cleanup, and you can do it automatically if you use this base class:

<?php

abstract class BaseTestCase extends PHPUnit_Framework_TestCase
{
    protected function tearDown()
    {
        $refl = new ReflectionObject($this);
        foreach ($refl->getProperties() as $prop) {
            if (!$prop->isStatic() && 0 !== strpos($prop->getDeclaringClass()->getName(), 'PHPUnit_')) {
                $prop->setAccessible(true);
                $prop->setValue($this, null);
            }
        }
    }
}

Our buddy Jenkins is much happier now — builds are approximately 20% faster.