Why Data Engineering is “a Marathon, and not a Sprint”

It isn’t easy to summarise the complex and rewarding job of a Data Engineer and the way they affect so many lives, so we spoke with Valiantsin Shkvarko, Lead Software Engineer at Godel, who provides an insight into his role, and explains why he loves it so much…

In recent years, there has been more importance placed on data engineers since their work has the potential to change millions of peoples’ lives for the better — but how? Firstly, the data engineer is responsible for collecting masses of data, which is then analysed, meaning it can be understood in ways which then allow us to make informed business decisions for our clients, which in turn can help them increase their efficiency, and often, profits. Not only this, but our work is most likely to have a ‘domino effect’ onto our clients’ customers, whose lives are also impacted in a positive way. Being aware of this is a great boost for morale — it gives me great satisfaction!

For example, data engineering is the force behind the notification we get when there has been suspicious activity on our bank accounts. The end user will only receive the notification, but behind this is masses of data which has been collected by an engineer, so I often refer to us as being ‘invisible heroes». It’s also important the data we use as engineers is kept up to date, and used in a timely manner, or it minimises the impact of the benefits for the end user. If the data is not kept up to date and we don’t react to it quickly, then the person is at risk of losing large amounts of money. Similarly, if we don’t use the data efficiently, then we would miss out on opportunities which have a positive impact on our clients. Additionally, it’s essential that we work with large sets of data, as this is what makes our findings much more qualitative, and therefore impacts the reliability of the information we present to our clients. By using data of such a nature, we can find hidden patterns, which in turn allow us to accurately estimate costs of infrastructure and forecast its’ ROI.

It’s clear that there have been rapid technological advances we have seen in data engineering, particularly in the last couple of years, but none of it would be possible without data engineers. Because of these advances, we are seeing more and more clients move away from traditional legacy systems, towards more modern and mature technologies, and it’s us who help bridge this gap. In the past, the data would have been stored in warehouses, but transitioning this data to the cloud is much more cost effective for our clients and saves them a lot of time. Overall, these more mature systems help us to understand the data better, by combining the best of traditional and modern systems, which presents us with optimal results. This means that our clients’ are able to reach their goals at a much more rapid rate, and it gives the team great satisfaction knowing that we have been a contributing factor to this. Not only this but working with the latest technology makes our time at work much more exciting.

Although I wouldn’t consider myself a businessman, given my knowledge, I would recommend that businesses pay attention to data engineering and the positive impact it could have on them.

Personally, I think data engineering is different to any other role because I don’t necessarily see it as a ‘job’, rather it is as integral component of my day-to-day life. Data Engineers are so fascinated with thinking of new solutions, whether it be for work or personal life, that we can never fully «switch off» from work as it’s something which affects us every second of every day and we are constantly looking to make improvements which affect ourselves, our clients, and their customers. Data engineers are always striving for more — helping the people we are working with to be the best versions of themselves and helping clients to become the best in their market. I think this is what makes our role different, too because our job is never complete, and I would say this is what gives data engineers a ‘kick’ — to continue to better their own work.

No two days are the same as a data engineer, but the best days are when we get fresh new challenges from customers as we love exploring ways and solutions which will enable our clients to feel fulfilled and reach their goals. Sometimes this can be a huge task, but we love it, nonetheless, as it really tests our level of teamwork. However, we see the end product and the impact it has made on our client’s — that’s the real reward.

The work we do is so complicated and hard to explain that there aren’t many people who can understand it. Additionally, there are a huge list of technologies which a data engineer should know, which is very impressive and highlights why it is so difficult to become an established data engineer. I’d say it’s a huge climb, but the view is worth it. The Data Engineering team is really something special. An excellent example is the Trainline project team, led by Sergey Oshin, Godel’s Head of Data. His entire team has a truly professional approach to their work, whilst having a great balance of soft, hard and technical skills. I believe this is important because it aids communication, which is a necessary component of teamwork, especially in the early stages of development, between both the team and the client.

To conclude, I think the most challenging part of my job is understanding as accurately and as fully as possible what results our clients expect from our work. In line with this, I would definitely say the best part of the job is when this has been achieved and it highlights to us the rewards which are a product of our work. These rewards effect the lives of many people, especially when we work with healthcare businesses, when our systems are helping doctors and their patients. This is especially true now, when the world is facing new challenges and must discover new resolutions to overcome these. With that in mind, I have faith that we will revert to some sort of ‘normal’ because the data is never wrong.