RELEASE THE KRAKEN: HOW PAYPAL IS BEING REVOLUTIONIZED BY NODE.JS AND LEAN-UX
By Cian O'Maidin

At the beginning of September 2013, we had the pleasure of hearing Bill Scott from PayPal talk about the revolution that is happening there through the use of Node.js and Lean-UX practices.

Bill arrived in PayPal in 2011 to a risk-averse, “not invented here” culture where long shelf life was the norm. Things were so slow, in fact, that in 2011 even a simple content copy change could take as much as 6 weeks to change on the PayPal site.

Due to the slow release cycle times, the door had been left open for new entrants like Stripe and Square to enter the payments market. One of the root problems was that all the technology in PayPal was tangled up and not suited for rapid experimentation and build/measure/learn.

 

The existing stack was Java through and through. You didn’t write html, css or JavaScript – you wrote css in Java, you wrote html in Java and you wrote JavaScript in Java. Later the stack was changed to be more reasonable (JSP for templating). But even in that state the stack was not conducive to prototyping, and most of the UI bits lived on the server.

Under Bill Scott, some sharp folks joined from Netflix and Yahoo, and in early 2012 they began to flesh out a UI layer that could support rapid experimentation.

 

In April 2012, David Marcus became president of PayPal and Bill was given a small team and a project to re-invent checkout.

David is a startup person and gave them 6 weeks to launch a new checkout system (bear in mind that checkout is a system that generates $3.5Billion revenue for PayPal). It wasn’t possible to get Node.js production ready within 6 weeks as a large number of PayPal’s subsystem needed to be integrated into the Node.js system so Node was initially used as a rapid prototyping framework.

Java and Node in Parallel

David had a number of features in mind for the initial system, and he drew some mockups. With Node, Bootstrap and other open source technology it was possible to get the flow working in 3 days. This LeanUX project was called Hermes: God of Mobile (mobility), God of Agility (lean) and God of Commerce (checkout).

The team adopted a whiteboard-to-code build-measure-learn prototyping cycle where the concept becomes a living spec, or as Bill dubbed it, ‘a vicious cycle of goodness was embarked upon’. Every week, the team went from code to usability test and feedback, where they built, got feedback and learning and then implemented what they learned.

Bill’s team did this for 8 weeks, while at the same time, the Java application team had started to spin up the application controller layer and got it up and running. In the meantime, most of the initial experience was already written in Node.

kraken logo

Focus on re-use (running two horses)

The strategy was to try to use the legacy systems already in existence in PayPal – but to do this from Node.js. These are the steps involved in implementing a game-changing program inside PayPal:

Step 1: Utilize an Open Source Stack (use tools that lots of people use)

Express, Connect, Require.js, bring in JavaScript templating and other open source UI Goodness.

Step 2: Bootstrap with Bootstrap

Used Bootstrap to enable developers to create a new branded frontend in a few hours, enabling sketch to code.

Step 3: Use JavaScript Templating

JavaScript Templating can be run in client browser or server on the production stack.

Step 4: Make UI Bits portable to legacy

It was possible to take the Dust Templates and use them on top of the old Java-Stack (this meant that quick results were possible from the frontend work that was being done). This allowed the team was to drag and drop the UI bits from the prototyping stack (Node) to the production stack (Java), with Rhino (JavaScript for Java VM) enabling stack parity between the two.

Project Delorean (named after the car in Back to the Future): as PayPal still has a number of experiences running on top of an XML/C++ stack in production, they were able to insert V8 into the C++ stack. Just like with Rhino, this allowed them to create Dust templates and run these JavaScript templates on the legacy C++ stack.

JavaScript templating gave a clean way to cover across 3 stacks at once – Node.js prototype, Java Legacy and C++ Legacy – and provided a graceful way to nibble away at the old stacks, producing results all the way along.

Step 5: Make it easy to understand and easy to experiment

Building totally on open source stack enabled new hires to start and grasp what was going on within a few hours. Contrast this with the custom Java Spring framework where it took a 20-day training course just to become proficient in that environment.

A key goal was hiding all the complexity of PayPal from developers and allowing them to just get stuff done without over-complicating things, plus creating an environment where it’s possible to embrace learning and experimentation. This makes it easy to experiment with minimal investment. In other words, make it cheap and frictionless to experiment.

Step 6: Bring Node to Production:

In order to do this, it was necessary to enable all of the standard PayPal services – without looking like PayPal. In other words, do it in a friendly NPM way, and simplify creating an app in a few minutes with all of the PayPal services.

One Language to Rule them all

Bringing Node.js into PayPal removed the barriers between the layers in the software organization, Node.js and JavaScript all the way up from just above the services layer to the frontend layer, meaning you need fewer engineers.

In one project there was initially 25-30 Java engineers were required to build a simple controller layer. Later this was simplified to a smaller team. But in parallel, the Node.js team did it with just 1-2 node engineers!

Developer Love

PayPal got it down to the level where developers could build an initial deployable app in a few minutes, while integrated with all the PayPal services. Bill’s team (along with other key partners in PayPal) made the environment developer-friendly internally to the point where adoption teams that had been in PayPal in the past to help with the adoption of Java Stack were no longer necessary.

The Node.js stack was user-friendly to the point where they had removed all the bespoke complexity and scariness out of their tool set.

Once Node.js made it into PayPal, they had to almost have an anti-adoption team to slow things down.

Frictionless

To get people up and running, there are JavaScript classes, nice set of github internal docs and Yeoman scripts to produce flavors of apps. A simple app that connects to authentication, account services, experimentation, logging and other services can be finished in 10-15 minutes. Once engineers have gone through this experience, they become believers and in turn evangelists for Node.

Step 7: Node.js Wins

One stack to rule them; all the new applications in PayPal are now being built using Node.js. This makes the prototyping and production stacks one in the same.

Step 8: Release Modules to Open Source:

A core express module (Kraken) (link), i18n: managing content at scale,
security, app lifecycle middleware, and bootstrap capability.

 

Summary:

Head count: Node.js. Typically they are seeing anywhere from ⅓ to 1/10 the number of developers needed on the new stack vs the old stack.

Performance: In one set of Load & Performance tests done on the same app built on both the Java stack vs the Node stack showed a 10x throughput in scale.

All in on Node.js: all the new applications in PayPal are now being built using Node.js

Lines of code: In general they have seen code size shrink by factor of 3-5 by moving from the Java stack to the Node stack. Even the number of files shrunk by a similar factor.

For more information on the modules and frameworks PayPal are moving to open source, watch out for the following, which will be appearing on the PayPal github account over the coming months (the main page will be http://krakenjs.com):

Kraken-js

The framework that started it all. This is the glue that holds everything together. Kraken-js is built on top of the tried and true Express webapp framework, and further streamlines it. It gives you a clean webapp structure that will induce developers to write neat, well organized code. When used in combination with the other modules, it makes it easy to set up robust applications.

To make things even easier for adopters, we are also providing a Yeoman generator for Kraken apps. By simply typing `yo kraken` you will have a ready-to-run basic application.

Lusca

Out-of-the-box application security. Lusca is middleware that can be deployed over Express, and configured to plug common attack vectors. When used, it will:

• Enable Cross Site Request Forgery (CSRF) headers.

• Enable Content Security Policy (CSP) headers.

• Enable X-FRAME-OPTIONS headers to help prevent Clickjacking.

• Enable Platform for Privacy Preferences Project (P3P) headers.

Adaro

A DustJS template rendering plugin for Express. It is based on LinkedIn’s fork of DustJS, and includes DustJS-helpers by default.

Makara

An internationalization/localization support module. It allows for loading content bundles based on a context (eg: a user connecting from the US would receive the en_US content bundle, while a spanish user would receive the es_ES one ). The loaded bundles can then be used to decorate DustJS templates.

Kappa

One of the main challenges faced by the enterprise, is the need to deploy private modules in an efficient manner. Existing solutions often entail replicating the public npm registry in its entirety, in order to deploy a few packages privately. Kappa is a smart npm proxy that allows public and private npm packages to be fetched from the right place. This means that companies can have a private npm server that only holds their modules. When a project requires either private or public modules, Kappa will transparently deliver them. In addition it will soon support licensing checks, blacklisting, statistics, and more!

In addition to the headliner Kraken Suite modules, they are releasing the following supporting utilities:

shortstop

Sometimes JSON just isn’t enough for configuration needs. Occasionally it would be nice to use arbitrary types as values, but JSON is necessarily a subset of all available JS types. shortstop enables the use of protocols and handlers to enable identification and special handling of json values.

findATag

A string tokenizer, configured to recognize a particular pattern: {@[A-Za-z._] [attrs...]/}.

spud

A content bundle transcoder. It will convert to and from different formats, including .properties, .json, .4cb, etc.

express-enrouten

Route configuration middleware for expressjs. This module changes where/how express loads its routing, making for more organized applications.

kraken-webtools

A collection of Development-time tools for Kraken applications.

 

Subscribe to our monthly newsletter!
join the discussion