Enterprise Data Warehouse Migration from SQL Server to Snowflake

About Client 

The client is a billion-dollar home leasing company with an estimated 100,000+ rental homes across the United States.

Background

Home leasing companies heavily rely on data to manage properties, tenant info, finances, and maintenance schedules. Efficient data management ensures organized, accessible, and secure information for seamless operations.  Legacy systems like SQL Server have been valuable tools in managing large datasets and performing complex queries. However, as data volumes and the need for real-time analytics increased, these systems faced limitations in processing speed and scalability.

Cloud-based solutions like Snowflake help in resolving these issues. By leveraging the scalability and elasticity of the cloud, Snowflake enables home leasing companies to process large volumes of data rapidly and efficiently.  Snowflake’s scalability and optimization empower companies to extract actionable insights efficiently, improving decision-making and operational effectiveness.

Challenges 

  • Complexity in Maintenance: Managing and optimizing the SQL Server-based EDW, including legacy Microsoft server stack like SSIS (SQL Server Integration Services)was a complex and time-consuming processes
  • Performance and Scalability: The scalability of SQL Server was restricted, with its resources shared among analysts and data engineers. When multiple users access it simultaneously, performance suffers, leading to delays in resolving queries and other issues for users
  • Operational Costs: The operational costs associated with maintaining and upgrading the SQL Server EDW were high, with estimates reaching approximately $250K, annually

Solution

The client decided to migrate its EDW from legacy Microsoft server stacks to a modern data stack on Snowflake, the cloud-based data platform designed for performance and scalability. The migration involved:

  • Modernizing ETL pipelines: Moving ETL processes from SSIS to AWS Glue and leveraging DBT (Data Build Tool) for data transformations reduced processing time from 3.5 hours to 1.5 hours, freeing up valuable system resources
  • Connecting SSAS cube to Snowflake: The existing SSAS cube which was used to aggregate and pre- calculate data was configured to point to the new Snowflake data source, ensuring continuity in reporting without disrupting business operations during the transition
  • Migration of Data Sources: Data from resources such as Salesforce ERP and Yardi CRM were successfully migrated to Snowflake, consolidating and aligning these data sources into a unified platform
  • Centralizing data in Snowflake: With this migration, Snowflake has become the single source of truth for data analytics, simplifying data management and fostering collaboration for the client

 Outcomes

  • Improved Accuracy: The migrated SSAS cube accurately reflected data in Snowflake, ensuring reliable financial reporting
  • Enhanced Performance: System performance increased by 40%, leading to faster response times for complex queries. This allows for quicker insights and faster decision-making
  • Reasonable Operational Costs: By migrating the Enterprise Data Warehouse to Snowflake, the client has slashed operational costs to just $17K, leading to a remarkable 93% savings. This cost reduction directly translates into enhanced profitability and financial efficiency for the business.
  • Reduced ETL Cycle Time: Migrating ETL processes to AWS Glue and DPT cut the processing time in half, freeing up valuable server resources
  • Faster Disaster Recovery: Snowflake’s architecture enables data recovery within an hour compared to days with SQL Server, minimizing downtime and ensuring business continuity
  • Increased Collaboration: With ample resources available on Snowflake, multiple users can work simultaneously without performance issues, fostering improved collaboration among BI, analysts, and data engineers