Last month I had the pleasure to hear Clifton Cunningham, CTO of the Mail Online (the online version of the Daily Mail) giving a talk about how they brought Node.js into the Publication.
To give some background, the Mail Online is now the largest online newspaper in the world, having overtaken the New York Times in January 2013. Every month they service over 150 million unique visitors serving in excess of 154 billion images (most of which are of Kim Kardashian).
The homepage itself is typically over 32,000 pixels long, and the Mail Online serves the entire thing in one go via Akamai. The side bar in the newspaper is so long in fact that they were featured in the Poke, saying that they were due to reach the 9th level of hell if they kept going the way they were going (the side bar contains 180 articles on average).
The Mail Online is a truly global business which runs 24 hours a day 7 days a week, with offices in London, New York and LA. Essentially they still broadcast the news, and because they use Akamai to deliver content, they can serve 150 million users with only 4 physical web servers.
Node and Mobile
Trouble has been afoot for a while though as mobile device traffic has been on the increase and it’s not possible to serve a 32,000 pixel webpage to a mobile device. Instead a more advanced approach is required where content for mobile needs to be served up in a more context aware way, where deciding who sees what content for mobile is a key enabler to a meaningful mobile experience for the readers of the publication.
Essentially Node.js came into the Mail Online because of mobile, and with it a change in how the publication architected things. The Mail Online now uses an event-based architecture, which uses lightweight micro-services.
The development resources in the company are full stack development teams, which are re-deployed into new project configurations on a regular basis.
First quick-win to start in production
The first app built by the Mail Online was a real time analytics platform based on Redis, it enables an editor to make decisions minute-by-minute on what articles to pace at the top of the home page.
Some other things the Mail Online have built with node:
Dynamic ad-slot management platform which optimizes and places ads on pages in a smart way based on page type, section of the site, device and geography. The Mail Online serves 10s of millions of pages each day, the optimisation of what ads go on what page for what device and geography is a key part of the business, and is how the publication makes money.
Editorialised e-commerce (Femail Fashion Finder): This is a stand-alone micro-app that editors can drop into articles about celebrities, it enables the sales of dresses, shoes, handbags etc .. that the celebrities are wearing.
Live Commentary app: The first iteration of the app is purely web-based, but will move into mobile, making the app highly interactive through the use of web-sockets. Initial sweet spots are live sporting events for the app, additional interesting use cases for the app will be 2nd screen application for things like the Oscars and other notable media events.
The Node stack:
Their Node stack is based around Express, with Moustache for templating. In terms of persistence pretty much everything is stored in Elastic Search and Redis is used as a caching layer for things like share counts and top articles etc.
The entire backlog of 1.2 million articles are stored in elastic-search, which is used to drive the site as well as applications around the content.
A micro-services architecture is used, which was inspired by Fred George, which is a slightly different take on the hapi plugin architecture, structuring applications to be maintainable as they get bigger is a key challenge going forward and micro-services is a solution to this. The MailOnline are also heavy users of Joyent (On Premise SDC and public cloud).
Journey to Node
Organisational Change (design == organization & fear != creativity)
The existing system was holding the organization back, and couldn’t scale to meet the growing demand of mobile and the editorial teams as the Mail Online grew globally. A project to replace it was underway, but had to be stopped as it simply re-implemented the same architecture (with many of the same challenges), but with additional separation between the ‘front end’ and ‘back end’ to mirror the technology organization at the time (a clear example of Conway’s law in action).
Working on this type of legacy monolithic platform is one predominantly of fear: fear of doing something that results in unintended consequences in another part of the code base. It’s impossible in this environment to come up with genuinely innovative and creative solutions, which is what was needed.
“Through inappropriate system design the organization created fear, and fear was blocking creativity”
Developer Driven Development and Programmer Anarchy
Enter Fred George (of Programmer Anarchy fame). Fred came on board as Chief Architect, and focused on supporting the organisational move from traditional enterprise technology team to a nimbler, highly skilled organisation built around a strong culture of personal responsibility and trust.
Over a number of months they applied some of the concepts from Programmer Anarchy, tailoring them for the Mail Online environment, which resulted in a method that Clifton prefers to call Developer Driven Development (DDD).
In DDD no unnecessary roles exist that separate developers from the customer, no unnecessary processes should be employed as a crutch (against fear).
The Rules are simple: developers, on a Table, own a project end-to-end and have the freedom to make their own choices. All major work is a team backlog that can be picked up by and Table. People can move freely among Tables (they can also be kicked out, and apply to move to other tables). Most Tables have 2-6 people on them. Tables have end-to-end responsibilities, including fixing it if it breaks at 3am. Tables choose whatever process works for them.
Clifton articulated the initial technology vision for the Mail Online, it was up to the team, execution of the vision was collectively owned by the team.
Tables are co-located, meters away from the news desk ‘the customers’, meaning it only takes minutes if something breaks for the communication between the developer and the user.
Killing Java with Clojure
In addition to the micro apps written to date, a Table also rebuilt the entire legacy Java frontend replacing the entire 155,000 lines of have with about 4000 lines of Clojure (accompanied by a large number of Moustache templates shared by Node).
Tools and processes used by Mail Online journalists are seen as a competitive advantage. The core authoring tools for the Mail Online are next in the firing line, and will likely be built on top of Node.
Ultimate take away from node.js at the Mail Online:
Node was not a risky choice for the Mail Online, initially it was used for a select group of obvious use cases, but now it has built up a head of steam and internal tools are being produced on a weekly basis at this point. As an organisation, the Mail Online haven’t contributed anything large back to the community yet but have a number of internal projects that are approaching this level of maturity, so watch this space.
Node made the team super productive, and this led to the freedom to implement the changes that were made, and this has resulted in problems being solved that genuinely people didn’t think could be solved.