The customer is a leading provider of intelligence and data to government agencies and the defence sector. They had migrated their core data ingestion platform to AWS, but with a traditional ‘lift and shift’ mindset that retained a monolithic SQL Server and Windows server based architecture at its core. They were not yet taking full advantage of the cloud, but had removed a dependency on an unreliable and costly legacy hosting provider.
The ingestion platform was slow and took 12 hours to run, gathering data from a variety of sources across the internet. The platform design did not facilitate scaling to increase throughput. Additionally, the system was relatively inflexible in terms of design – requiring input from developers when web sources changed, and a complete application update to adjust configuration. As a result, Analysts tended to silo their own data sets rather than use the central system - meaning they spent valuable time gathering the data rather than analysing it.
D55 completely rebuilt the application architecture, recoding the application where necessary. Core technologies include Amazon DynamoDB, AWS Lambda and AWS Event Bridge. Data ingestion takes less than two minutes to complete, as opposed to 12 hours previously – and includes 35x more data sources, now numbered in the thousands.
The platform now includes a full audit trail of data sources, which can be adjusted by analysts swiftly as a simple configuration change, rather than a development-led software coding effort. Analysts are enabled to be significantly more efficient and can now spend time on value added content creation rather than slowly aggregating disparate data sources. Event driven business logic has been created and, when any content changes, analysts are proactively notified of the delta. Similarly, processing logic has been added to automatically detect and use AI services to translate (for example) Russian or Chinese text to English. AI is further put to use on data classification by adding meta-data. This process is intelligent and spots data points that previously would have needed human input – such as noting the difference between Paris, France and Paris, Texas.
In the six-month period following the project, revenues have materially increased following an improved level of service quality.
While D55 led the platform redevelopment, we also worked actively with the client’s teams to upskill and enable them for the future so they can extend and develop the platform further.
Plans for 2024 include experimenting with the use of GenerativeAI technologies to augment and support the analysts work.
