page-template,page-template-right_sidebar,page-template-right_sidebar-php,page,page-id-2143764,strata-core-1.0.5,ajax_fade,page_not_loaded,wpb-js-composer js-comp-ver-6.2.0,vc_responsive

Designing the workflows and framework


As a final stage in the data lake setup, the following was done to ensure data was mined and reported as per the requirement:

    • Developed the DevOps process
    • Monitored/alerts/scheduling for failures/success
    • ETL Framework based on AWS Glue to discover data and store the associated metadata in the AWS Glue Data Catalog
    • Catalogued data was searchable using Elastic Search and available for various business applications through various SQL queries.

Author :
Aditya Sathyadev, Co-founder and Director – BizAcuity
IIT (BHU), IIM (Calcutta)

Monitor your usage in cloud