Tech Excellence: Building a Microservices Product in 6 Months

Building a software product from scratch takes a lot of technical and business knowledge from everybody with a stake in the product’s success. It’s a considerable challenge when the product is internal — but when it needs to be a high-usage platform which must also be integrated with various third-party systems, the complexity becomes formidable. Godel’s experience building products like this with its clients has spotlighted a number of interesting business problems. Here, we explain the approaches taken to solving them.

A Looming Deadline

In the scenario we’re discussing here, the business that needed to develop this product is in a highly demanding market — e-commerce retail. Demand surges are unpredictable but one date in the diary is always going to cause a spike: Black Friday. Six months prior, the business brought the challenge to Godel.

Taking these circumstances into account, Godel started with a Discovery Phase. The Godel team chosen to help build the product consisted of senior developers, business analysts and quality assurance engineers. Their first two weeks on the project were spent on-site at the client’s office — everybody was fully invested in learning the necessary core system knowledge. Rather than running multiple planning sessions with smaller groups, this format combined with the seniority of everyone involved was how the team quickly achieved foundational understanding the initial product requirements.

Setting A Solid Foundation

Laying a foundation based on complete system knowledge was essential for fast delivery. For this product, the team needed to ensure that the first line of code written for a feature could go all the way to production without any setbacks. The CI/CD pipelines, test automation framework and real production environments had to be ready for deployments before anything could be deployed. With this in mind, before touching product development the team built an infrastructure and environment designed for the rapid deployment they had in mind.

It was clear from the outset that a high engineering bar would be necessary if the product was to be released on schedule — there was very little room for error. To facilitate this Godel ensured no compromises were made on test coverage or documentation. The «definition of done» was extremely clear and specific, and included a rigorous testing approach that combined unit and automated testing. If the «definition of done» was not met by a feature, then it would not be considered production-ready.

Choosing The Right Architecture

Developing the product required the team to make the right decision at every step of architecture planning. First, a microservices architecture was an absolute must. The product would be plugged into dozens of external platforms, each of which with vastly different levels of process. A risk from this was that if a problem arose with one system, the whole product would break — not an option for this product. Microservices mitigate this risk (depending on how they are built) by isolating specific services into containers so that they can run independently, as part of the wider product.

The product was to be hosted in Azure since that was the client’s existing cloud provider. However, specific technology decisions still needed to be made. The product would need to handle a huge amount of data processing and given the client’s domain the volume of data would peak and trough at unpredictable times. Combined with the many external integrations that the product would interface with, Kubernetes was the clear choice of containerisation technology to tackle this challenge. Kubernetes orchestrates load up and down depending on system usage and handles each container independently so that performance can remain optimal across the whole product.

Integrating With Third Party Systems

There were many third parties that the product needed to be integrated with. Each of them had its own interface — some with clear documentation and RESTful services, and others with limited (FTP/SFTP/API (XML, SOAP etc)) systems that present a more complex challenge. The team was faced with a mountainous task: how could they build a product that has the same top-line functionality, but so many nuances within each integration?

The golden solution was performance testing, as part of a wider continuous deployment approach. The Godel team had built a second production environment which was essentially a simulation of real-life use. Every single deployment went into this pipeline and was load-tested nearly to death with emulations of sky-high loads. It was how the team could present the product with a «worst-case scenario», see how it reacted and understand why components did not work — without discovering these errors in a live situation.

Ensuring Peak Performance

Performance testing was how the team knew the product could handle all manner of scenarios in live production. Still, further measures were necessary in ensuring constant availability. Another technology decision the team made early on was the introduction of a message queue system (Azure Service Bus). The messaging system promoted asynchronous communication within the product — when one container had an issue, requests would not back up in the messaging queue behind that issue until it was resolved. Instead the specific microservice would simply get the next message while the issue was resolved — with no impact on the overall product’s availability.

One important element to success for the team was autonomy. As mentioned earlier the team was made of very experienced people. Each individual developer took full ownership of a slice of the product, adhering completely to the quality thinking approach whilst taking their code all the way to deployment. There was never a separation between development and testing: the quality assurance engineers would ensure complete and intensive test coverage at all times, whilst the developers maintained first-rate code quality and documentation.

Team autonomy also facilitated faster decision-making; everyone knew their part of the system at such a fundamental level that they could be trusted to choose the best technologies needed. With that said, the team could still work cross-functionally. When a feature needed to be deployed quickly, developers could work on new areas of the product easily because of the wider system knowledge that they had understood since Sprint Zero.

A Successful Product

Every step taken above was due to the team’s dedication to delivering a high-quality product ahead of schedule. This was achieved. Over 30 microservices went into creating a product which can handle around 100 requests a second from its third-party integrations. Every integration necessary was completed in time so the product could perform at its full capacity on time. Most importantly, setting a foundation for quality created a product which is scalable and can be built upon — more integrations can be completed easily because a core approach had been rigorously defined from the outset. There was no need to backtrack or re-develop, despite the tight schedule, because quality was baked into the product from day one.