Removing Scalability Limitations and Improving Deployment Capabilities with AWS Serverless

Overview

CoreLogic is the largest provider of property information, analytics and property-related risk management services in Australia and New Zealand.

As a well established and respected information and analytics provider, they rely on many core systems to service both their internal and client needs. Over the years the business has continued to grow with customers expecting faster and more in-depth insights, and systems being required to cope with ever increasing demand.

Specialising in property, one of their core systems used throughout the NZ business was their property matching solution, held in an on-premise Oracle database. Industry Data were employed to re-design this solution to operate within the AWS cloud, removing issues that had arisen from lack of scalability, visibility, reporting and deployment.

Take a look at the CoreLogic website to learn more about what they do: corelogic.com.au

Business Challenges

  • Scaling limitations of the legacy system meant client and internal requests could not be serviced as required
  • Legacy system incured large vendor licence fees
  • Slow response to change due to lack of continuous integration/deployment
  • Limited visibility into system metrics and client usage
“Industry Data came in and understood the problem that we were trying to solve and some of the scalablity issues that we were having. Was able to build out solution in some of Amazon's services that more than met our needs.”

--Glynn Willis, CoreLogic Australia

Project Outcomes

Using AWS Serverless architecture and the components detailed below, Industry Data were able to migrate the current CoreLogic matching system into the AWS Cloud, and achieve the following business outcomes:

  • Solution elastically scales to manage unpredictable usage whilst improving latency - improved customer experience, greater system resilience, reduced internal resource, reduced risk
  • Full system logging to improve visibility into system usage to aid development and enhance internal analytics
  • CI/CD pipeline for improved business agility - quickly deploy changes with reduced risk
  • Improved customer satisfaction, system is always available, sub-millisecond response times to calls now achieved
  • De-coupled serverless architecture allowing for independent scaling of compute components, as well as resilience to quickly recover from failure
  • Potential 45% reduction in monthly fees (average AWS monthly charge versus Oracle standard edition licence cost) + reduction in internal resources

The Industry Data Solution

Industry Data decided on an AWS serverless architecture to meet the scaling requirements identified. API calls to the matching service were unpredictable and sporadic, and in some cases required the matching of many millions of records. The solution had to be scaled at speed to allow for many concurrent calls to be made and results returned, without incurring large costs of managing pre-warmed servers that could be left idle for large periods.

AWS Lambda was chosen for its ability to scale at speed and be hyper-ready to respond without any of the costly overheads of managing a fleet of servers. Adding SQS allowed for a de-coupling of the application components to increase fault tolerance and allow for continuous deployment with reduced risk of loss of data. DynamoDB was selected as a further serverless component to run the matching, whilst S3 provided the backbone storage.

The AWS services utilised within this solution were: AWS Lambda / Amazon DynamoDB / Amazon API Gateway / Amazon EC2 / Amazon S3 / Amazon SQS

The serverless architecture developed for this solution is documented below. For more information on serverless computing by AWS please visit: AWS Serverless Architecture

Architecture diagram for an AWS serverless record matching solution