Data Hub platform development

Skycatch builds and sells technology to autonomously capture, process, and analyse highly accurate 3D aerial mapping and survey data for mining and construction companies globally. Skycatch has a world-class team of developers who consistently work with complex engineering challenges to provide their customers with the high quality outputs and digital tools they desire. Recently, in an effort to accelerate the development of some of their core analytical tools, they enlisted the help of NearForm to augment this deeply skilled in-house engineering team.

Skycatch had a major challenge on their hands with Data Hub. They needed to deliver a new future-proof product that would go beyond their competitors’ offerings. It needed to be globally-available, secure, performant and to meet the diverse needs of all their customers. To add to the challenge, Skycatch had to deliver on a very short timeline.

This also coincided with rapid growth in Skycatch’s other product lines which consumed significant resources in their advanced computer vision processing development team. So, to meet their aggressive goals, Skycatch chose to engage with NearForm to enable an efficient, accelerated product delivery while working alongside a similarly competent team.

Given the high level of business innovation and technical challenge, the key to project success had to be an in-depth collaboration, using the strengths of both Skycatch and NearForm. To that end, together we formed a fully integrated team leveraging the engineering and development talent within both companies from across North America and Europe.

NearForm created a team of diverse experts, including platform architects, developers and data visualisation experts to complement the experience and skill of the full stack engineering team at Skycatch. The teams kicked off the engagement by aligning business goals, defining user needs and discussing technical options. This also included an introduction and onboarding of NearForm to the Skycatch technology stack. NearForm and Skycatch mapped user journeys and identified both key features and project risks. Together, we validated assumptions by creating and reviewing a series of prototypes from low to very high fidelity.

With the vision for the product ready, the development team rapidly produced a highly-functional prototype that all parties could validate. At the same time, technical experts from both companies worked together to devise a data processing, analytics and visualisation platform architecture that would allow the product to be alternatively hosted on both the cloud and custom hardware devices in the field.

The platform development was progressing well when a key customer raised expectations by introducing many complex features needed for their specific use cases. At the request of the Skycatch product and engineering teams, NearForm decided to take on this new challenge to significantly accelerate their pace of development.

Drawing on their delivery experience, NearForm grew the team in a rapid but methodical manner. To ensure deadlines were met, NearForm first scaled out to two agile squads, and then three squads, while maintaining a significant velocity increase for each new team member.

It was important the team delivered each milestone both on time and with high quality as the client immediately applied rigorous tests and assurance checks. To ensure that we stayed on track, we took a risk-based approach, continuously analysing the project health and capturing any project risks for immediate mitigation. This allowed us to resolve impending issues, such as a build-up of work in progress, so that the team always remained focused.

One of the risk analysis inputs was the regular feedback from the squad retrospectives. This feedback allowed us to continually improve our processes and ways of working. We resisted the urge to postpone the retrospectives, even when time was short and the delivery pressure was high, as they were invaluable. Retrospectives also serve as an outlet to release some of the tensions that can build on the project. Likewise, regular one-to-one sessions between management and each team member ensured that we aired and resolved issues early and everyone stayed efficient and aligned.

We selected and implemented a new generation of technology to meet the ambitious designs, including the latest in cloud-based build pipelines, Kubernetes and serverless technology. Each member of the team worked to their limits to systematically deliver each milestone on time. The result was the successful delivery of an optimised, scalable and extensible data processing, analytics and visualisation platform.

With the combined resources of the NearForm and Skycatch teams, this project was delivered in under 9 months, something that would have taken a regular team 3-4 years and 10x the cost.

The story doesn’t end there though. The product delivered so far is the basis for many future innovations. The team at Skycatch remains fully enabled for ongoing development and operations, with NearForm at hand to accelerate the addition of new features and capabilities, if needed, to supplement the Skycatch internal team of engineers.

NearForm started the engagement with Skycatch with a three-day discovery workshop. It helped the team immerse itself into the world of surveying construction sites. User flow map, prototypes and project scope were some of the outcomes of this initial phase. It clarified expectations and helped the team reach a consensus on where we were going as well as getting buy-in from senior stakeholders.

Beyond this kick-off phase, the product designer focused on two key aspects: task-oriented and scalable designs.

Enterprise-level products like this one have a rich collection of features. One of the designer’s responsibilities was to make sure everyone looks at the features from the users’ perspective. For example, construction site managers and superintendents use the application. They do not want a ruler or a marker. They need an accurate way to measure the volume of a dirt pile, because this impacts on how much they charge the contractor who will do the job. Understanding the underlying goal was key to designing a job-driven application that fits people’s workflow.

From start to finish, the designer and front-end engineers put their efforts into the design system as much as the application itself. The interface’s styles and behaviours inherited from the design system (design tokens, layouts, components); this ensured optimum reusability of components and consistency of the experience. Building a new flow does not require us to build everything from scratch. Instead, we can achieve 80% reuse of existing components. This results in significant time and effort savings.

With team members working remotely and from many locations, the different time zones represented a major challenge. We located squads in overlapping time zones as much as possible. We kept the squads aligned by regularly sharing progress while actively working on dependencies. As well as regular sprint demos, developers enjoyed the technical show ‘n tell sessions, giving them a chance to walk through and get feedback on their latest work.

We minimised the need to meet face-to-face by regularly using video conferencing and chat communications. We also had occasional meet-ups on each continent to develop understanding, work through hard problems and build project energy.

We worked hard to maintain the story pipeline by planning ahead, and not skipping essential meetings, such as story grooming. To maximise efficiency, we ensured that we held just the right number of project meetings, with all attendees adding value. This reduced waste and frustration, enabling developers to focus on the tasks at hand.

It was crucial that we started with a well-designed architecture. We defined a microservice-based architecture, with domain boundaries and clear component responsibilities, favouring composition over configuration. This has proven valuable in many ways, such as allowing developers to work in parallel without getting in each other’s way. It was especially useful when rapidly onboarding new people.

Engineers from both organisations worked collaboratively to deliver technology that would allow for fast, efficient development, including AWS native services, cloud-based build pipelines, and container orchestration through Kubernetes.

We designed the front end to accommodate many developers working on different features. We achieved this by using a modular architecture and feature plugins, each developed independently. We also created reusable components to save time later in the project. In addition, a monorepo allowed us to avoid complexity and unnecessary overhead.

This combination of leading technology, strong architecture and best practice engineering culture allowed us to maximise the efficiency of the engineering team and scale it out to the size needed to deliver the project on time.