Being a leader in the F&B with a workforce of 500+ associates and a distribution network spread across the US; they had ambitious growth plans. The company’s IT sector receives a wide variety of data such as Sales, Supply Chain, Formula, Shelf space etc. For the company to make a meaningful impact, they wanted to connect data and empower all areas of business with modern, scalable data and analytics. CloudIQ Technologies helped bring all of the data into a Unified Analytics Platform using Azure Data Bricks to enable Sales Analytics and ML use cases.
The company uses Microsoft Dynamics as their ERP system. In turn, they used JET Data Manager to extract and load the data from ERP system, which in turn will be pushed to the Business Intelligence system. This data was then used to build the Business Dashboards, sales and other business reports using PowerBI. The company also receives data from their co-packers, sales data from POS, marketing data, formula, shipment, distribution and many more.
The data received from these different systems do not have a consistent pattern or format. Data is received either to FTP or API or SharePoint etc as csv, or json or excel etc. Each department within the company would build SSIS packages to extract the data from their respective partners and build reports. There is no one central place where all the data is made available and scalable, making it tedious to build reports and analytics, provide data insights, and applied Machine learning.
CloudIQ wanted to select a modern, scalable, robust Data Platform on which all Data Engineering, Data Science and Exploratory Data Analysis will be facilitated and run for all areas of business within the company. The goal was to bring all data into unified Analytics Platform to enable subsequent Sales Analytics and ML use cases. CloudIQ chose Databricks Delta Lake as a solution to store all data in one Data platform. In addition, CloudIQ also built a framework for extract, process and load the data which is being received from different partners.
In addition, CloudIQ also built a framework for extract, process and load the data which is being received from different partners.The config driven framework is built to extract the data from different data sources, like SQL, ftp, api etc and load its raw format into the iron tier. The iron tier is also built to listen to a stream so that it stores only incremental data instead of storing the complete dataset. The data from iron tier, is then processed into bronze tier.
This will be Databricks table which is cleansed. Reports can be built directly from the Bronze tier. Data is then enriched, joined with other bronze tables, grouped, and loaded into the silver tier. Aggregations, summary, rollups, and statistical data is processed and in gold tier.
PowerBI can now connect to Databricks to any of the above tiers using spark connector to retrieve the data from Databricks tables and build custom reports. Databricks offers other capabilities like maintaining history, access control, zoning of the datasets, data catalog for easy discovery.
The company now has a modern, robust data platform. Unified Data Analytics Platform has helped the company to accelerate innovation by unifying data science with engineering and business.
Microsoft Azure Data Bricks, Azure SQL Managed Instance, Azure DevOps
CloudIQ is a leading Cloud Consulting and Solutions firm that helps businesses solve today’s problems and plan the enterprise of tomorrow by integrating intelligent cloud solutions. We help you leverage the technologies that make your people more productive, your infrastructure more intelligent, and your business more profitable.
Chennai One IT SEZ,
Module No:5-C, Phase ll, 2nd Floor, North Block, Pallavaram-Thoraipakkam 200 ft road, Thoraipakkam, Chennai – 600097
© 2023 CloudIQ Technologies. All rights reserved.
Get in touch
Please contact us using the form below