Using AWS Serverless to Mine Global Data for Analytics, Research and Marketing
A proprietary data mining solution utilising an AWS serverless architecture and services was originally designed and built by Industry Data to create an Australia wide business directory, containing all known businesses, contacts and additional information required for analysis and marketing.
To stay ahead of the competiton the business database had to be updated daily by locating all new or changed data sources across the web. A serverless architecture was designed to allow for multiple updates from unpredictable workloads without the need to pre-empt scaling requirements.
Since go-live this technology has been used time and time again to mine information from sources all over the world. The solution is infintiely scalable and as all data is landed in S3 it is easily consumed by downstream services.
Take a look at the original product website here: Industry Prospects
- Ability to mine data daily to apply the latest updates and changes from across the web
- Unpredicable and potentially large workloads required to run in minimal time to ensure all data is collected and processed
- Multiple workloads required to concurrently mine from global locations
- Data outputs must be easily integrated with downstream processes for collation and analysis
“Industry Prospects is Australia's best maintained B2B prospect database. Our proprietary web bot scours the web and collects new and changing business data. It is the work horse that needs no rest.”