RELEASE THE KRAKEN: HOW PAYPAL IS BEING REVOLUTIONIZED BY NODE.JS AND LEAN-UX

By: Cian O'Maidin

At the beginning of September 2013, we had the pleasure of hearing Bill Scott from PayPal talk about the revolution that is happening there through the use of Node.js and Lean-UX practices.

Bill arrived in PayPal in 2011 to a risk-averse, “not invented here” culture where long shelf life was the norm. Things were so slow, in fact, that in 2011 even a simple content copy change could take as much as 6 weeks to change on the PayPal site.

Due to the slow release cycle times, the door had been left open for new entrants like Stripe and Square to enter the payments market. One of the root problems was that all the technology in PayPal was tangled up and not suited for rapid experimentation and build/measure/learn.

 

The existing stack was Java through and through. You didn’t write html, css or JavaScript – you wrote css in Java, you wrote html in Java and you wrote JavaScript in Java. Later the stack was changed to be more reasonable (JSP for templating). But even in that state the stack was not conducive to prototyping, and most of the UI bits lived on the server.

Under Bill Scott, some sharp folks joined from Netflix and Yahoo, and in early 2012 they began to flesh out a UI layer that could support rapid experimentation.

 

In April 2012, David Marcus became president of PayPal and Bill was given a small team and a project to re-invent checkout.

David is a startup person and gave them 6 weeks to launch a new checkout system (bear in mind that checkout is a system that generates $3.5Billion revenue for PayPal). It wasn’t possible to get Node.js production ready within 6 weeks as a large number of PayPal’s subsystem needed to be integrated into the Node.js system so Node was initially used as a rapid prototyping framework.

Java and Node in Parallel

David had a number of features in mind for the initial system, and he drew some mockups. With Node, Bootstrap and other open source technology it was possible to get the flow working in 3 days. This LeanUX project was called Hermes: God of Mobile (mobility), God of Agility (lean) and God of Commerce (checkout).

The team adopted a whiteboard-to-code build-measure-learn prototyping cycle where the concept becomes a living spec, or as Bill dubbed it, ‘a vicious cycle of goodness was embarked upon’. Every week, the team went from code to usability test and feedback, where they built, got feedback and learning and then implemented what they learned.

Bill’s team did this for 8 weeks, while at the same time, the Java application team had started to spin up the application controller layer and got it up and running. In the meantime, most of the initial experience was already written in Node.

kraken logo

Focus on re-use (running two horses)

The strategy was to try to use the legacy systems already in existence in PayPal – but to do this from Node.js. These are the steps involved in implementing a game-changing program inside PayPal:

Step 1: Utilize an Open Source Stack (use tools that lots of people use)

Express, Connect, Require.js, bring in JavaScript templating and other open source UI Goodness.

Step 2: Bootstrap with Bootstrap

Used Bootstrap to enable developers to create a new branded frontend in a few hours, enabling sketch to code.

Step 3: Use JavaScript Templating

JavaScript Templating can be run in client browser or server on the production stack.

Step 4: Make UI Bits portable to legacy

It was possible to take the Dust Templates and use them on top of the old Java-Stack (this meant that quick results were possible from the frontend work that was being done). This allowed the team was to drag and drop the UI bits from the prototyping stack (Node) to the production stack (Java), with Rhino (JavaScript for Java VM) enabling stack parity between the two.

Project Delorean (named after the car in Back to the Future): as PayPal still has a number of experiences running on top of an XML/C++ stack in production, they were able to insert V8 into the C++ stack. Just like with Rhino, this allowed them to create Dust templates and run these JavaScript templates on the legacy C++ stack.

JavaScript templating gave a clean way to cover across 3 stacks at once – Node.js prototype, Java Legacy and C++ Legacy – and provided a graceful way to nibble away at the old stacks, producing results all the way along.

Step 5: Make it easy to understand and easy to experiment

Building totally on open source stack enabled new hires to start and grasp what was going on within a few hours. Contrast this with the custom Java Spring framework where it took a 20-day training course just to become proficient in that environment.

A key goal was hiding all the complexity of PayPal from developers and allowing them to just get stuff done without over-complicating things, plus creating an environment where it’s possible to embrace learning and experimentation. This makes it easy to experiment with minimal investment. In other words, make it cheap and frictionless to experiment.

Step 6: Bring Node to Production:

In order to do this, it was necessary to enable all of the standard PayPal services – without looking like PayPal. In other words, do it in a friendly NPM way, and simplify creating an app in a few minutes with all of the PayPal services.

One Language to Rule them all

Bringing Node.js into PayPal removed the barriers between the layers in the software organization, Node.js and JavaScript all the way up from just above the services layer to the frontend layer, meaning you need fewer engineers.

In one project there was initially 25-30 Java engineers were required to build a simple controller layer. Later this was simplified to a smaller team. But in parallel, the Node.js team did it with just 1-2 node engineers!

Developer Love

PayPal got it down to the level where developers could build an initial deployable app in a few minutes, while integrated with all the PayPal services. Bill’s team (along with other key partners in PayPal) made the environment developer-friendly internally to the point where adoption teams that had been in PayPal in the past to help with the adoption of Java Stack were no longer necessary.

The Node.js stack was user-friendly to the point where they had removed all the bespoke complexity and scariness out of their tool set.

Once Node.js made it into PayPal, they had to almost have an anti-adoption team to slow things down.

Frictionless

To get people up and running, there are JavaScript classes, nice set of github internal docs and Yeoman scripts to produce flavors of apps. A simple app that connects to authentication, account services, experimentation, logging and other services can be finished in 10-15 minutes. Once engineers have gone through this experience, they become believers and in turn evangelists for Node.

Step 7: Node.js Wins

One stack to rule them; all the new applications in PayPal are now being built using Node.js. This makes the prototyping and production stacks one in the same.

Step 8: Release Modules to Open Source:

A core express module (Kraken) (link), i18n: managing content at scale,
security, app lifecycle middleware, and bootstrap capability.

 

Summary:

Head count: Node.js. Typically they are seeing anywhere from ⅓ to 1/10 the number of developers needed on the new stack vs the old stack.

Performance: In one set of Load & Performance tests done on the same app built on both the Java stack vs the Node stack showed a 10x throughput in scale.

All in on Node.js: all the new applications in PayPal are now being built using Node.js

Lines of code: In general they have seen code size shrink by factor of 3-5 by moving from the Java stack to the Node stack. Even the number of files shrunk by a similar factor.

For more information on the modules and frameworks PayPal are moving to open source, watch out for the following, which will be appearing on the PayPal github account over the coming months (the main page will be http://krakenjs.com):

Kraken-js

The framework that started it all. This is the glue that holds everything together. Kraken-js is built on top of the tried and true Express webapp framework, and further streamlines it. It gives you a clean webapp structure that will induce developers to write neat, well organized code. When used in combination with the other modules, it makes it easy to set up robust applications.

To make things even easier for adopters, we are also providing a Yeoman generator for Kraken apps. By simply typing `yo kraken` you will have a ready-to-run basic application.

Lusca

Out-of-the-box application security. Lusca is middleware that can be deployed over Express, and configured to plug common attack vectors. When used, it will:

• Enable Cross Site Request Forgery (CSRF) headers.

• Enable Content Security Policy (CSP) headers.

• Enable X-FRAME-OPTIONS headers to help prevent Clickjacking.

• Enable Platform for Privacy Preferences Project (P3P) headers.

Adaro

A DustJS template rendering plugin for Express. It is based on LinkedIn’s fork of DustJS, and includes DustJS-helpers by default.

Makara

An internationalization/localization support module. It allows for loading content bundles based on a context (eg: a user connecting from the US would receive the en_US content bundle, while a spanish user would receive the es_ES one ). The loaded bundles can then be used to decorate DustJS templates.

Kappa

One of the main challenges faced by the enterprise, is the need to deploy private modules in an efficient manner. Existing solutions often entail replicating the public npm registry in its entirety, in order to deploy a few packages privately. Kappa is a smart npm proxy that allows public and private npm packages to be fetched from the right place. This means that companies can have a private npm server that only holds their modules. When a project requires either private or public modules, Kappa will transparently deliver them. In addition it will soon support licensing checks, blacklisting, statistics, and more!

In addition to the headliner Kraken Suite modules, they are releasing the following supporting utilities:

shortstop

Sometimes JSON just isn’t enough for configuration needs. Occasionally it would be nice to use arbitrary types as values, but JSON is necessarily a subset of all available JS types. shortstop enables the use of protocols and handlers to enable identification and special handling of json values.

findATag

A string tokenizer, configured to recognize a particular pattern: {@[A-Za-z._] [attrs...]/}.

spud

A content bundle transcoder. It will convert to and from different formats, including .properties, .json, .4cb, etc.

express-enrouten

Route configuration middleware for expressjs. This module changes where/how express loads its routing, making for more organized applications.

kraken-webtools

A collection of Development-time tools for Kraken applications.

 

By: Cian O'Maidin

Cian O'Maidin is the co-founder and visionary CEO behind nearForm. Very early on he saw how Node.js could help the world work in a better way and is committed to making that happen through nearForm. He is an established Entrepreneur, Node Community Leader, the Creator of NodeDublin and now the Curator of NodeConf.EU. In 2012 Cian was voted amongst the most influential leaders in Irish technology.
  • Peter Elger

    Real interesting post, Thanks!

  • http://www.insitedesignlab.com/blog insitedesignlab

    Really inspiring stuff, as someone in a similar situation (trying to get out of tech debt) this definitely resonates with me.

  • Robert Read

    Nice post.

  • Greg Tomei

    I had no idea all this was going on at PayPal. Good stuff.

  • jhnlsn

    is Kappa a public project or something private to PayPal? Really interested in something like that and can’t really find anything about it on npm or google search

    • Bill Scott

      Drop me an email with your github username and next week I will give you access. Full public access is in a couple of weeks. We gave Cian a sneak peek.

      • Daniela

        I am also interested in Kappa, is it similar to what Nexus is for Maven?

      • berzniz

        Would also love(!) to take a look (github.com/berzniz). We’re

      • Ruan Botha

        Thank you Bill, this looks wonderful! Very excited to witness the “migration”. Do you believe more companies would be using Node.js in enterprise in the near future?

  • BillSaysThis

    Great post but where are the repo links?

    • Bill Scott

      They are coming. Sorry. We have been using this over the last year in our internal enterprise github. We had to do a bunch of renaming and are moving the libraries to githhub.com/paypal in the next week or so (it will be private repo for a week or so, then opened up). If you want access just email me at billwscott@gmail.com

      • http://www.morgancraft.com Morgan Craft

        really interested in the mentioned express-enrouten but I’ve googled and checked npm for anything that resembles it and struck out. Would love a link or more information.

        • Lenny Markus

          As Bill mentioned, we are >thisclose< to releasing the whole thing.
          We're basically going through the last of the internal legal requirements :)
          Stay tuned!

      • Brian Moeskau

        Link does not go where you think it goes…

    • Bill Scott

      In case you haven’t seen it: krakenjs.com will get you there.

  • grnadav

    Very nice post, but still looks like a lot of the old Not Invented Here is still there, and they have 10 (!!!) new librariesframeworks to show for it.

    • Bill Scott

      Actually we are purposely not adding a lot to existing frameworks. We have an allergy to the NIH syndrome. Our modules for security, internationalization, etc. are just there to help if you are using express based node apps.

      • http://www.codingninja.co.uk/ Zen Master

        Bill you say you don’t have “NIH” syndrome, however the above blog post suggests you have created a whole host of frameworks! why?

        Why write a ton of frameworks? I mean “findATag”? sounds a lot like regex to me? or am I mistaken?

        I hate frameworks just so you know my bias, I’d rather write raw javascript then strap a ton of frameworks just to display a few shiny things on the screen, but that’s just me.

        • Bill Scott

          findATag is just a specialization of string tokenizer handy for dust templates. If you look at our framework we could be criticized for actually it being so lightweight. Over the last year we have thrown away a bunch of node modules/libs that we started writing that once we saw a good open source replacement we started using that instead.

          It is interesting that you refer to a “mountain” of frameworks. We released a half dozen or so node modules and some yeoman generators. Not quite a mountain.

          I have started an “open source office” at PayPal because I do feel like if you look at our github.com/paypal there is a bunch of previous random stuff out there that has dubious value. I want to clean that up.

          The step to release Kraken was more about taking our core part of the framework (there is a lot more inside that is specific to the PayPal/eBay eco-system) and get it outside the walls of PayPal. That way it couldn’t get bastardized into a PP specific framework. Kraken is pretty simple (although the journey to get it to be simple is a story I will write about later) as it just adds some nice conventions and lifecycle stuff to express. The other modules are handy as they either don’t exist or didn’t exist in a way that was satisfactory or we felt made sense to fork.

          BTW, there is a nice example of a shopping cart end 2 end on krakenjs.com now. That might help with comprehension of the offering.

          • http://www.codingninja.co.uk/ Zen Master

            Maybe I’m sounding cranky, but it seems you are re-inventing an awful lot of things, that have already been solved.

            Want smart templating, heck just use PHP.

          • Bill Scott

            Bingo! You are cranky ;-)

            And at this point becoming a troll. I am happy to dialogue, but that is a silly comment (PHP being the answer).

          • http://www.codingninja.co.uk/ Zen Master

            Yes php was tongue in cheek, but as a side note, a lot of smart “templates” just end up replicating php in one form or another.

            Bill, if node.js works for your use case that’s great :)

            I personally don’t trust the claimed hype, as the benchmarks says otherwise.

    • Ryan Scheel

      Lots of libraries that do one thing and do it well. They are valuable additions to the Node ecosystem — some of them without good alternatives.

    • http://www.codingninja.co.uk/ Zen Master

      +1 I was thinking the same thing!

  • Aslam Tajwala

    NodeJS for the win!

    • Roe

      I’m not sure how NodeJS had anything to do with their success. It was just one of many tools they could have chosen to use to complete this effort. I see nothing remarkable here aside from the fact that a company like paypal would want to be an early adopter of technology that is so young and unproven.

      • Aslam Tajwala

        Agreed, this could be done with other similar technologies, but they chose Node, and they triumphed. Node is not a panacea for all software development – it does a lot of things, but it does a bunch of those really well. Just like anything else there is a fine line between winning & losing – the extra edge is all that matters. To conclude, All programming languages are turing complete; they could write there own stack with C if they liked and had the same performance gain (if not less) but that would take 10X the time.

        • http://www.codingninja.co.uk/ Zen Master

          no you are incorrect.

          The reason for the productivity boost was not due to Nodejs, it was due to they way the legacy system was designed!

          It has nothing to do with nodejs.

      • erhangundogan

        It is nearly 5 years old and maturing. The power of NodeJS comes from asynchronous design and javascript. That’s why it will be a chosen one on the web. nginx also good performer with same asynchronous design.

        You may see that Microsoft adopted it on Azure, and other big companies like Yahoo, eBay, LinkedIn, PayPal are all using it somehow in their structure. I think NodeJS proven its power.

        • http://www.codingninja.co.uk/ Zen Master

          A design which is easily replicated in other platforms and languages. React for PHP as an example.

          • erhangundogan

            There are others like twisted matrix, ruby event machine etc. any platform can implement event loop. But it doesn’t mean that they can survive and grow powerful. NodeJS made it so far. I do not definitely pick reactphp, because they have long way to go.

            NodeJS has great community and resources today. Besides it is javascript. It is going to rule the backend as it did frontend.

          • http://www.codingninja.co.uk/ Zen Master

            I love javascript, in fact I write pure javascript as part of my day job (look ma no frameworks ;)

            But I also write in a lot of other languages too. I love js, but I don’t want to use js in every situation!

            server side js, YUCK that idea died 10 years ago *cough* asp JScript anyone?

            Use the right tool for the right job.

            If I only knew JavaScript, then nodejs is the kind of hammer I would have invented:

            “If all you have is a hammer, everything looks like a nail”

          • erhangundogan

            I have used most languages and platforms like python, c#, php, java and their web platforms. I think that NodeJS is pretty straightforward and suitable for asynchronous web server design.

            Web server architecture must be very simple in it’s nature. You must answer requests and provide response. That’s all. Besides you can forward time consuming I/O operations to child processes. That’s how it works. So, i am not sure if you have good understanding on web server architecture.

            Btw, NodeJS is not just a hype, a lot of people recognized it already. You may use it or not, it is your choice.

          • http://www.codingninja.co.uk/ Zen Master

            I understand web servers, one of my abandoned projects called “S3ngularity” was building a web server. And contrary to your assertion that a web server is “simple” is not the case.

            It is “simple” in concept, in reality its a little more involved. Take a look at the following specs for a web server (snippet):

            RFC 1945, RFC 2616, RFC 2396, RFC 2671.

            Then you have to think about all the other wonderful things such as process isolation, nodejs is single threaded, if a part of your code crashes, it brings your entire “web server” down. That is simply unacceptable design for a “web server”.

          • erhangundogan

            I did not understand which part you are addressing in RFC documents as “more involved” with web server? Can you please be more specific? Besides these are protocol documents and do not describing web server infastructure.

            Furthermore you are addressing HTTP protocol. Well that can be acceptable, But we have new protocols like WebSockets today. And NodeJS is very acceptable design for that kind of technology.

            Interpreted languages has disadvantage for the exceptions, it is the truth. But as you may know faulty code and unhandled exception can bring any server down. NodeJS has exception handling mechanism. So it can recover from errors. Even it crashes and exit, NodeJS server can go live in a second and it can be respawn easily. I am pretty sure it is not possible for all kind of servers.

          • http://www.codingninja.co.uk/ Zen Master

            That’s not correct IIS for example has proper worker process isolation. if a application pool crashes it does not take down the entire web server. Nodejs is on the other hand will just die.

            This is why you have ridiculous work around such as “supervise” modules for nodejs. Which basically restart nodejs when it dies.

            Terrible software design for a web server,

          • erhangundogan

            Well, good luck to you with IIS…

          • http://www.codingninja.co.uk/ Zen Master

            I don’t need luck when I have well designed web servers, unlike nodejs which is a terrible design for a web server.

            Nodejs async IO event is cool***, in fact other platforms are copying this approach, so it does have one positive, I admit that. And in some cases nodejs may be useful.

            But when outlandish claims about performances are made, and the numbers do not actually reflect the claims it is plain hype(tm).

            ***But it is not original, it is how network based comms was done in the old days. Nodejs just brought it back into fashion of late.

    • http://ondrek.com/ Samuel Ondrek

      YEEEAH! :D

    • http://www.codingninja.co.uk/ Zen Master

      The back end API and services run on the JVM, what the front end apps are developed in make no difference, might as well have been PHP, probably would have been faster.

      • andrewvijay

        You seem to be really annoyed at node js. Pls keep calm and follow whatever you want. I guess people can judge their app’s performance in a few days. They at least get a new experience in a new framework. Continual learning Zen! Peace btw node and php :)

        • http://www.codingninja.co.uk/ Zen Master

          Hi Andrew,

          I’m calm :)

          You are right developers can use whatever tool they want.

          What I hate is that node.js which is a product of Joyent (a private company), is using FUD and hype and misinformation to push its own agenda and products.

          NearForm is a node.js consulting company and naturally has a stake also in heavily promoting node.js.

          Again use a tool for its merit, its a shame that people often mistake hype for “merit”.

          You have a good day sir :)

  • http://www.nosqlbr.com.br Suissa

    Very nice!

  • http://www.nosqlbr.com.br Suissa

    The only Kraken which I know is https://github.com/kraken-io/kraken-node

  • Marius Kubilius

    Kraken.js is supposed to be released soon according to this presentation: http://www.slideshare.net/lennymarkus/node-js-at-paypal

    • Bill Scott

      Yes. Sorry for the confusion :-)

      I should have anticipated that Cian would get a lot of press and at least had a placeholder up at krakenjs.com ;-)

  • Daniel Eck

    Awesome! I fav’d this talk months ago. Happy to hear adoption (and productivity) has landed. More fuel for the fire in my own adoption battle…

    RELEASE the KRAKEN!!!

  • geophoenix

    it took 25 java engineers to write java code : D and 1-2 node.js engineers to write code that does the same thing. i mean imagine the development cost going dramatically down.

    • Roe

      This didn’t seem like a fair comparison, of course a legacy system full of tech debt would take more engineers to get things done. The true test of their architecture and their tooling will not come around until this new framework as decayed over a similar period of time. I’d love to know how many people it’ll take to write node apps in their new environment 5 years from now.

      • Bill Scott

        To clarify.

        The Java team wanted to start with about 30 engineers (more that they had staff in place to use). Several of us argued that down to 5 java engineers.

        Then we had 1 node developer start playing catchup and added a second pretty quickly. The 2 node developers were able to get to parity in a couple of months. At that point we repurposed all of the devs (7) to doing Node.

  • Philip O’Brien

    Great read!

  • http://jobety.com/ Jobety

    This is very interesting. Any plans on making some of these open source?

    • Lenny Markus

      They are *all* going to be OS’d.
      We built them from the ground up with this in mind.

      • http://jobety.com/ Jobety

        That’s awesome to hear @lennymarkus:disqus

  • Nick Mason

    I love the post, but it’s more focussed on the fact that you’ve removed technical debt than the lean ux portion. Can you talk a little bit more about that? How do you plan to scale the UX stuff? Bootstrap is a starting point, but certainly not an ending point.

    • Bill Scott

      I have separate workshops on that. See slideshare.net/billwscott for some of my longer form talks on LeanUX.

      • Nick Mason

        Thanks, Bill.

  • JacksonTian

    Great!

  • Owls Rutherford

    Great post on how Node.js and staying opensource & lean can revolutionize the workflow. These technologies are true disruptors in the space.

  • vitmalina

    Great Post! In my team we are in a similar predicament. Everything is Java on the server side (and lots of developers and efforts). I am promoting adoption of node.js for the UI layer (Java will still do the rest), which makes a lot of sense. I hope articles like this will help me succeed. Thanks.

  • erhangundogan

    javascript is going to rule not only frontend but also backend of the web servers. and nodejs is the clear winner about lightweight and fast web apps.

    • Roe

      It’s not the clear winner, it’s just one of the only languages to make the attempt as of yet.

  • http://fordinteractive.com/ Andy Ford

    I can’t help but wonder how the hire of Douglas Crockford may have played into the decision to go for Node.

  • Guille

    Have you considered using Node.js also for backend services or you prefer to keep your existing Java stack for backend?

  • Jonathon Creenaune

    If I read this right, you were running dust templates through Rhino on the JVM … how did this work in production? Were there any hiccups?

  • http://www.codingninja.co.uk/ Zen Master

    The node fan boys will be rolling in soon, they forget the actual heavily lifting is done by JVM back-end, but node.js is all hipster and sounds cool and is “l337 and web-scale”.

    • strictnein

      20 Day training courses for horribly outdated platforms are super “web-scale”. As is requiring 10 times the developers. Also, you missed a couple of bits: “all the new applications in PayPal are now being built using Node.js” and “In one set of Load & Performance tests done on the same app built on both the Java stack vs the Node stack showed a 10x throughput in scale”. But keep refusing to upgrade your skillset, because Node.js is just a fad.

      • http://www.codingninja.co.uk/ Zen Master

        Have a look at some actual benchmarks (TechEmpower Round 7), node is squashed so badly its not even a joke:

        http://www.techempower.com/blog/2013/10/31/framework-benchmarks-round-7/

      • http://www.codingninja.co.uk/ Zen Master

        I’m a polygot engineer, your comment is based nothing but pure assumptions. I learn different langauges and platforms, frameworks every other day. The ones I like I stick with. I tried node.js, I just happen to see past the hype.

        If you even read the title, it takes about “LEAN UX”.

        Basically they were writting EVERYTHING in java (which is madness).

        Thats the part they changed, the core remains the same.

        • Bill Scott

          Actually you are conflating things. Earlier in the Java framework they tried to get everyone to write everything in Java. That was 2011. When I joined they had just moved to JSP. With a little twist reminiscent of GWT. The approach they took at that point is REALLY COMMON. I get lots of pings from companies living with some flavor of building webapps with the J2EE or Java/Spring/JSP frameworks. This is what I rescued PayPal from.

          So it is not fair to say well node was better than a twisted framework. It is better than a Java/JSP framework.

          And I have some cred to talk about that btw. I have built 4 flavors of Java/JSP frameworks since 1997. In fact when I joined Netflix in 2007 we went with Struts 2 since it was about the only decent (I say that with a smirk) framework available then.

          There are lots of reasons something like node shines as a web app framework. Even more than performance. A post for a later date.

    • Roe

      Node fan boys will be node fan boys. I would never go so far as to say that NodeJS is a fad, it’s just too early in it’s life-cycle for me to endorse it’s use. If it’s still around in a year then maybe I’ll take a look, for now though it’s just another new tool and I only use new tools when the requirements drive me there, I don’t like to use new tools just because their shiny and new.

      • Andrea Rossi

        Good point. Maybe you can even spend some of the time you’re not wasting by using Node to improve your grammar.

    • erhangundogan

      you do not get it do you? it is not matter of java or javascript, it is the matter of finishing product with fewer people, resources and having better performing product on the web. i am curious who is the fan boy here…

      • http://www.codingninja.co.uk/ Zen Master

        why?

        so that a few months down the line, the shambles that is left has to be fixed and completely re-achitectured again by the next developers?

        Developer productivity is important, but not at the cost of a poor implementations/design choices.

        The PayPal case, their legacy system did everything server side inclduing CSS and html, which is madness for sure. Now they are doing things a little more saner. Technically they could have used anything for the front end. So the use of node.js is nothing special in this case.

        • erhangundogan

          It could be anything like you said but they chosed NodeJS! and it was a good choice indeed. I am not sure if they could make it better with another one.

          • http://www.codingninja.co.uk/ Zen Master

            After looking at the recent benchmarks, the benefits of nodejs are diminished further. Even the creator has admitted that php with react framework can replicate nodes async style.

          • Bill Scott

            Obviously you made your mind up about node. I sat on a panel with LinkedIn’s head of mobile, the head of engineering at Google. The head of infra at eBay. And earlier with Eran Hammer, head arch at walmart labs (mobile). Additionally, Dav Glass who leads node arch at Yahoo! All of us have seen the same thing. In real web production, in real large scale systems, in real internet companies, node is outperforming the java stack in every case.

            Once Java has Narwhal in full swing that could be a different story. Narwhal brings evented IO to Java.

          • http://www.codingninja.co.uk/ Zen Master

            Bill that’s great, but I want actual numbers:

            http://philsturgeon.co.uk/blog/2013/11/benchmarking-codswallop-nodejs-v-php

            I just don’t buy the hype, its js engine, why would I want to use a js engine at the heart of a web server?

            Greate node is async, but at some point actual work has to be done somewhere in the pipeline.

          • Bill Scott

            That article is about scraping. Scraping is more about computation than about I/O. When you take in a page and then block on scraping that is completely misunderstanding and misusing node’s async capability. We posted numbers on our blog did yo not see them? https://www.paypal-engineering.com/2013/11/22/node-js-at-paypal/

            These numbers are actually holding up quite well (in fact as we are adding clustering it appears that Node will be 10x faster in our case.)

            Yes, actual work has to be done in the pipeline. I wonder if you understand our architecture. Node is at the webapp layer. It is nothing but an aggregator of data and a formatter. That by definition is what functional languages do well. And this aggregation comes from I/O. That is what async does well with evented I/O. No real computation (a little in formatting, but very little).

            We call services (non-blocking) like add a card, add a bank, remove a card, remove a bank, confirm a bank, send money, receive money and so on. These are not done in our node layer. They are done in the Java Layer and our node app calls a service for this asynchonously in a fire and forget model. The node process goes back to handling the event loop. but doesn’t block while send money happens. it can process tons more requests. Then an event happens where the money got sent (confirmation) and the callback is fired in node and the UI gets sent some JSON or some HTML gets formatted and the world is happy.

            The reason you want to use a JS engine at the heart of a web app server is why Ngnix is so much better than apache. evented i/o is better than threads for handling large amounts of i/o bound traffic.

            JS gets extra points due to it being a functional programming language, the use of closures that matches so well with asynch and the fact that V8 C++ engine powers it.

            Node just happens to be the most mature evented i/o, async, functional programming platform that is ubiquitous to the web.

            This is not really a Java vs JS question. It is modern web architecture has led us to see that at different layers in the stack there are different problems and concerns. Node nicely satisfies those.

            This is why there is so much interest in Scala, btw. It gives functional programming in the VM and becomes more terse and dynamic compared to classical inheritance schemes.

            No one is saying Java is dead. We are using it all over the place at PayPal and will continue to do so. Just not for the web app layer.

            Walmart uses it for fronting services as a proxy. If you think about it that is the same aggregrating/formatting challenge that the web apps have. There they have some god awful back end systems speaking in cryptic protocols that Node smooths out to be all RESTful/JSON services.

          • http://www.codingninja.co.uk/ Zen Master

            Which all comes back to my original point, that the front end web application could have been written in anything!

            The problem is that most people do not read pass the headlines.

            The tech press are already running away with the impression that nodejs is “l337″ and web-scale and “hey look PayPal have dropped the JVM and its all running off nodejs”.

          • erhangundogan

            yeah the question is who is going to use reactphp? how much time it will take people to learn and contribute? It is just silly option for this conversation, sorry.

          • http://www.codingninja.co.uk/ Zen Master

            I just demonstrated that nodejs is not special, in fact its “number 1 killer feature” (aysnc I/O) is already available in other platforms.

            Nodejs real world performance doesn’t live up to its Hype in actual benchmarks.

            But feel free to use nodejs if it meets your need.

    • Bill Scott

      You are funny (not sure if you meant to be ;-)

      We do our REST services in Java for a lot of reasons. Java really shines for compute intensive. NodeJS does not. Node really shines for I/O bound (evented IO is the solution). Java does not. Node follows the same architectural ideas as Ngnix and twisted/python.

      Now as a strong counterpart to your argument though look at what Walmart Labs is doing with NodeJS. They ran 100% of their mobile traffic (btw, this is a lot of traffic) to their Node framework on Black Friday. This twitter feed: https://twitter.com/search?q=%23nodebf&src=hash details it. Basically they got bored because of just how well it performed.

      I think a lot of node critics actually haven’t spent time looking at the core of it (the V8 google engine) which is a real key to how fast it is — a C++ engine). This coupled with advances in JavaScript runtime technology we are starting to see near native speeds with JS.

      • http://www.codingninja.co.uk/ Zen Master

        The standard JVM are written in C, so your argument about reaching native speeds are mute.

        Then there is the whole can of worms regarding compiled vs JIT, which I’m sure you know about.

        If the only thing that node has to offer is async I/O, then even this is not much of a reason as there are many things like it eg React for PHP, vert.x for JVM etc.

        • http://www.isights.org/ Michael Long

          “The standard JVM are written in C, so your argument about reaching native speeds are mute.”

          Huh? A JVM may be written in C (or C++), but that’s the byte-code interpreter. You know, the interpreter that cycles through and interprets the compiled Java byte-code? The interpreter that passes function calls to other interpreted byte-codes in the Java library? That interpreter.

          Now, you can Hotspot a JVM and improve performance by doing JIT compilation of the byte-code… which apparently places you back into the aforementioned “can of worms” territory.

          V8 compiles JavaScript source code directly into machine code when it is first executed. In fact, a server-based JIT compiler can be a lot more aggressive and take a little longer to do the compilation, since it’s likely that any compiled module will be accessed again by subsequent requests.

          PHP, which you often bring up below, is an interpreted language, most often running on the Zend II engine. PHP modules are parsed, converted to byte-code, and executed by the Zend engine.

          As such, it’s slower than JIT-compiled code by at least a full order of magnitude.

          • http://www.codingninja.co.uk/ Zen Master

            Michael,

            You are correct, however you forget that there is a lot of caching available. As an example in PHP’s case most PHP accelerators cache the compiled byte-code, so this entire argument about “its interpreted thus slow” does not apply.

            In the end, the real world benchmarks have shown that node.js so called “performance” simply does not stack up to its hype.

            Does that mean one should not use node.js?

            No, in certain contexts it is a good fit, so please lets put balance over hype.

          • http://www.isights.org/ Michael Long

            First, any caching options or strategy (caching
            compiled code, memcached, etc.) you’d care to name are pretty much available on both sides of the fence.

            It’s true that Zend’s “accelerator” caches PHP compiled byte-code, but that doesn’t change the fact that the compiled byte-code still has to be run through the Zend engine (byte-code interpreter) each and every time it’s executed.

            As opposed to how the V8 engine caches JS compiled MACHINE code, which in turn can be directly executed for each request.

            It might help to understand how the technologies work before making statements like, “”its interpreted thus slow” does not apply.”

          • http://www.codingninja.co.uk/ Zen Master

            Michael,

            No you are incorrect here, the accelerators (there are a lot of them) CAN also cache the COMPILED machine code, not just the byte code.

            So my statements still stands.

            I hope that helps.

          • http://www.isights.org/ Michael Long

            The difference between executing optimized cached byte code and “native code” is most assuredly not marginal, especially when the later is also optimized, cached, and compiled directly for the processor the code is running on.

            And seriously, you’re telling me that code running on code pretending to be a processor (managing function calls, stacks, “registers”, and so on) is faster than the same module running natively? Give me some of what you’re smoking, please.

            I have several PHP sites I support for clients on two of the largest PHP hosting services, WPEngine and Synthesis, and they most assuredly do not offer “native” performance.

            The HipHop JIT will be cool… once it’s released into the wild. The dev team is still regression testing it against most of the common frameworks and stacks.

            Evented PHP IO will be cool too… when available and when more sites are totally rewritten to use it.

            The benchmark for “real-world” PHP performance is Zend II, perhaps with an accelerator, and that’s on 95% of the current installations.

            No HipHop. No React. And no magic “machine code” interpreters that don’t seem to be on the list of the most common PHP accelerators.

            http://en.wikipedia.org/wiki/List_of_PHP_accelerators

          • http://www.codingninja.co.uk/ Zen Master

            byte code vs native in theory should be no contest right?

            In theory you are right, however reality is a lot more complex.

            Lets look at real number:

            C# Mono vs JavaScript V8 Engine:

            C# Wins (V8 is only wins 2 out of 10)

            http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=v8&lang2=csharp&data=u64

            Java 7 vs JavaScript V8 Engine:

            Java 7 wins (V8 is only wins 2 out of 10)

            http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=v8&lang2=java&data=u64

            Scala vs V8:

            http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=v8&lang2=scala&data=u64

            Again same story, the V8 engine (which node is built on) as not as fast as you would expect.

          • http://www.codingninja.co.uk/ Zen Master

            Disqus decided to kill my reply.

            I’m not going to rewrite that long reply again,

            I’ll just summarize:

            * In theory Native should be faster than VM byte code

            * In reality its more complicated than that.

            I can’t put to many links in as disqus nuked my last reply,

            However look at the benchmarks for

            http://benchmarksgame.alioth.debian.org/u64/javascript.php

            Java 7 vs V8

            Scala vs V8

            C# Mono vs V8

            In all the above instances V8 was slower in general. One would expect V8 to be faster but this is not the case in real cases.

          • http://www.isights.org/ Michael Long

            I believe we were just discussing interpreted PHP performance, but no matter.

            The Mono runtime contains a code execution engine that translates byte codes into native code. Java made HotSpot the default configuration in 1.3. Scala runs on the JVM, so again, HS.

            Your site is interesting, but isn’t effective in comparing algorithm X in language A to that of language B.

            Your site shows that JS, Java, and Mono have roughly the same execution times when running the same algorithms, and as a class handily beat out PHP, Ruby, and other languages that don’t do JIT native compilation.

            Major differences often come down to the algorithm used. For example, according to your link, PHP is about twice as fast as JS in the k-nucleotide test, but JS is 3x faster than PHP if PHP uses k-nucleotide #4.

            Compare algorithms for the fastest version, and you can see that the JS author is doing a lot more make-work parsing and tokenizing his own strings.

            The two algorithms are apples to oranges. In effect, it’s as if language A tested sorting 25 million records against B, and A won in the “sorting” category.

            A, however, used a Quickersort against B’s Bubble sort, and won handily. Duh.

            Get back with me when you can directly test A’s Quickersort against B’s Quickersort.

          • http://www.codingninja.co.uk/ Zen Master

            Yes we where talking about PHP but moved specifically to “Native Code Vs Optimized VM Byte Code”

            If you read the benchmark notes, you would have noticed that the fastest versions of the respective platform/language was picked for each test.

            You can always amend and submit your changes to those benchmarks if you think you can improve them.

            But it does clearly show that when considering speed, you have to consider the entire system and not just a small subset of a given system.

            As I have mentioned earlier, I agree that native machine code will be faster than VM Byte Code (if that is the only element you are focusing on).

            However in practice because that element is only a small part of a bigger picture it is never as simple as the theory.

            And once again, the root of all this discussion (at least in my mind) is that node.js is built on a lot of hype and simply not has fast as it is made out to be.

            Perhaps when I said that the difference between “Native Code and VM Byte Code is marginal”, I didn’t make the context clear. I should have said looking at the whole system (not just the machine code execuation end).

            Anyway discussions aside I hope you enjoy the rest of day Michael, forgive me if I’ve caused any offence! :)

  • buginsoup

    Hoping to see better reports and navigation in PayPal. With node.js it should be possible to provide such interface.

  • Dhamodharan Lakshmipathy

    Awesome ! Great Stuff ! !
    So when can we expect some feature in LIVE, which uses the above technology stack ?

  • Richard Migambi

    Good article, whether you like Nodejs or not, this is proof that it can do a good job

  • Cristian I.

    The Kraken has been released! krakenjs.com

  • Azat Mardanov

    Please capitalize the names such as Java, JavaScript, etc. Also you have some missing links.

  • Flying Horse

    Would be great to hear Bill’s take on Meteor, any plans to use it, or to make Kraken reactive like Meteor?