Data Hub platform development
Integrated team builds a data processing, analytics and visualisation platform in record time
Objective: Accelerate the development of core analytical tools. It needs to be globally-available, secure, performant and delivered on a very short timeline.
Solution: Successful delivery of an optimised, scalable and extensible data processing, analytics and visualisation platform.
Skycatch builds and sells technology to autonomously capture, process, and analyse highly accurate 3D aerial mapping and survey data for mining and construction companies globally. Skycatch has a world-class team of developers who consistently work with complex engineering challenges to provide their customers with the high quality outputs and digital tools they desire. Recently, in an effort to accelerate the development of some of their core analytical tools, they enlisted the help of NearForm to augment this deeply skilled in-house engineering team.
Delivering a major platform in a short timeframe
Skycatch had a major challenge on their hands with Data Hub. They needed to deliver a new future-proof product that would go beyond their competitors’ offerings. It needed to be globally-available, secure, performant and to meet the diverse needs of all their customers. To add to the challenge, Skycatch had to deliver on a very short timeline.
This also coincided with rapid growth in Skycatch’s other product lines which consumed significant resources in their advanced computer vision processing development team. So, to meet their aggressive goals, Skycatch chose to engage with NearForm to enable an efficient, accelerated product delivery while working alongside a similarly competent team.
The Skycatch engineering team has always worked within a very fast moving flow. Finding team members that can move at the same pace, while building advanced technology, is not easy. NearForm’s team, composed of many core Node.js engineers, is world-class and easily adapted to our pace to get the job done within our tight deadline.
Supplementing a team with the right mix of skills
Given the high level of business innovation and technical challenge, the key to project success had to be an in-depth collaboration, using the strengths of both Skycatch and NearForm. To that end, together we formed a fully integrated team leveraging the engineering and development talent within both companies from across North America and Europe.
NearForm created a team of diverse experts, including platform architects, developers and data visualisation experts to complement the experience and skill of the full stack engineering team at Skycatch. The teams kicked off the engagement by aligning business goals, defining user needs and discussing technical options. This also included an introduction and onboarding of NearForm to the Skycatch technology stack. NearForm and Skycatch mapped user journeys and identified both key features and project risks. Together, we validated assumptions by creating and reviewing a series of prototypes from low to very high fidelity.
With the vision for the product ready, the development team rapidly produced a highly-functional prototype that all parties could validate. At the same time, technical experts from both companies worked together to devise a data processing, analytics and visualisation platform architecture that would allow the product to be alternatively hosted on both the cloud and custom hardware devices in the field.
The platform development was progressing well when a key customer raised expectations by introducing many complex features needed for their specific use cases. At the request of the Skycatch product and engineering teams, NearForm decided to take on this new challenge to significantly accelerate their pace of development.
Drawing on their delivery experience, NearForm grew the team in a rapid but methodical manner. To ensure deadlines were met, NearForm first scaled out to two agile squads, and then three squads, while maintaining a significant velocity increase for each new team member.
NearForm brought extremely talented engineers and designers to the Data Hub project which was instrumental in meeting and matching our own engineering team’s high standards for quality and delivery. NearForm continually added top talent to the team on short notice, that allowed us to supplement our own resources and respond effectively and quickly to customer change requests.
Staying on track for delivery
It was important the team delivered each milestone both on time and with high quality as the client immediately applied rigorous tests and assurance checks. To ensure that we stayed on track, we took a risk-based approach, continuously analysing the project health and capturing any project risks for immediate mitigation. This allowed us to resolve impending issues, such as a build-up of work in progress, so that the team always remained focused.
One of the risk analysis inputs was the regular feedback from the squad retrospectives. This feedback allowed us to continually improve our processes and ways of working. We resisted the urge to postpone the retrospectives, even when time was short and the delivery pressure was high, as they were invaluable. Retrospectives also serve as an outlet to release some of the tensions that can build on the project. Likewise, regular one-to-one sessions between management and each team member ensured that we aired and resolved issues early and everyone stayed efficient and aligned.
We selected and implemented a new generation of technology to meet the ambitious designs, including the latest in cloud-based build pipelines, Kubernetes and serverless technology. Each member of the team worked to their limits to systematically deliver each milestone on time. The result was the successful delivery of an optimised, scalable and extensible data processing, analytics and visualisation platform.
With the combined resources of the NearForm and Skycatch teams, this project was delivered in under 9 months, something that would have taken a regular team 3-4 years and 10x the cost.
The story doesn’t end there though. The product delivered so far is the basis for many future innovations. The team at Skycatch remains fully enabled for ongoing development and operations, with NearForm at hand to accelerate the addition of new features and capabilities, if needed, to supplement the Skycatch internal team of engineers.
The NearForm team is at the forefront of their field and have expertise in best practices and the latest effective technologies. Each team member was highly motivated and completely dedicated to a very challenging project, from start to finish. NearForm proved to be so much more than a hired vendor – they lived and breathed as a natural extension of our own world-class engineering team.
How to approach design on such a fast-moving project?
NearForm started the engagement with Skycatch with a three-day discovery workshop. It helped the team immerse itself into the world of surveying construction sites. User flow map, prototypes and project scope were some of the outcomes of this initial phase. It clarified expectations and helped the team reach a consensus on where we were going as well as getting buy-in from senior stakeholders.
Beyond this kick-off phase, the product designer focused on two key aspects: task-oriented and scalable designs.
Enterprise-level products like this one have a rich collection of features. One of the designer’s responsibilities was to make sure everyone looks at the features from the users’ perspective. For example, construction site managers and superintendents use the application. They do not want a ruler or a marker. They need an accurate way to measure the volume of a dirt pile, because this impacts on how much they charge the contractor who will do the job. Understanding the underlying goal was key to designing a job-driven application that fits people’s workflow.
From start to finish, the designer and front-end engineers put their efforts into the design system as much as the application itself. The interface’s styles and behaviours inherited from the design system (design tokens, layouts, components); this ensured optimum reusability of components and consistency of the experience. Building a new flow does not require us to build everything from scratch. Instead, we can achieve 80% reuse of existing components. This results in significant time and effort savings.
How to work efficiently and remotely?
With team members working remotely and from many locations, the different time zones represented a major challenge. We located squads in overlapping time zones as much as possible. We kept the squads aligned by regularly sharing progress while actively working on dependencies. As well as regular sprint demos, developers enjoyed the technical show ‘n tell sessions, giving them a chance to walk through and get feedback on their latest work.
We minimised the need to meet face-to-face by regularly using video conferencing and chat communications. We also had occasional meet-ups on each continent to develop understanding, work through hard problems and build project energy.
We worked hard to maintain the story pipeline by planning ahead, and not skipping essential meetings, such as story grooming. To maximise efficiency, we ensured that we held just the right number of project meetings, with all attendees adding value. This reduced waste and frustration, enabling developers to focus on the tasks at hand.
How could so many engineers work effectively on the same code?
It was crucial that we started with a well-designed architecture. We defined a microservice-based architecture, with domain boundaries and clear component responsibilities, favouring composition over configuration. This has proven valuable in many ways, such as allowing developers to work in parallel without getting in each other’s way. It was especially useful when rapidly onboarding new people.
Engineers from both organisations worked collaboratively to deliver technology that would allow for fast, efficient development, including AWS native services, cloud-based build pipelines, and container orchestration through Kubernetes.
We designed the front end to accommodate many developers working on different features. We achieved this by using a modular architecture and feature plugins, each developed independently. We also created reusable components to save time later in the project. In addition, a monorepo allowed us to avoid complexity and unnecessary overhead.
This combination of leading technology, strong architecture and best practice engineering culture allowed us to maximise the efficiency of the engineering team and scale it out to the size needed to deliver the project on time.