Being a leader in the F&B with a workforce of 500+ associates and a distribution network spread across the US; they had ambitious growth plans. The company’s IT sector receives a wide variety of data such as Sales, Supply Chain, Formula, Shelf space etc. For the company to make a meaningful impact, they wanted to connect data and empower all areas of business with modern, scalable data and analytics. CloudIQ Technologies helped bring all of the data into a Unified Analytics Platform using Azure Data Bricks to enable Sales Analytics and ML use cases.

Customer challenges

The company uses Microsoft Dynamics as their ERP system. In turn, they used JET Data Manager to extract and load the data from ERP system, which in turn will be pushed to the Business Intelligence system. This data was then used to build the Business Dashboards, sales and other business reports using PowerBI. The company also receives data from their co-packers, sales data from POS, marketing data, formula, shipment, distribution and many more.

The data received from these different systems do not have a consistent pattern or format. Data is received either to FTP or API or SharePoint etc as csv, or json or excel etc. Each department within the company would build SSIS packages to extract the data from their respective partners and build reports. There is no one central place where all the data is made available and scalable, making it tedious to build reports and analytics, provide data insights, and applied Machine learning.

Partner solution

CloudIQ wanted to select a modern, scalable, robust Data Platform on which all Data Engineering, Data Science and Exploratory Data Analysis will be facilitated and run for all areas of business within the company. The goal was to bring all data into unified Analytics Platform to enable subsequent Sales Analytics and ML use cases. CloudIQ chose Databricks Delta Lake as a solution to store all data in one Data platform. In addition, CloudIQ also built a framework for extract, process and load the data which is being received from different partners.

In addition, CloudIQ also built a framework for extract, process and load the data which is being received from different partners.The config driven framework is built to extract the data from different data sources, like SQL, ftp, api etc and load its raw format into the iron tier. The iron tier is also built to listen to a stream so that it stores only incremental data instead of storing the complete dataset. The data from iron tier, is then processed into bronze tier.

This will be Databricks table which is cleansed. Reports can be built directly from the Bronze tier. Data is then enriched, joined with other bronze tables, grouped, and loaded into the silver tier. Aggregations, summary, rollups, and statistical data is processed and in gold tier.

PowerBI can now connect to Databricks to any of the above tiers using spark connector to retrieve the data from Databricks tables and build custom reports. Databricks offers other capabilities like maintaining history, access control, zoning of the datasets, data catalog for easy discovery.

Customer benefits

The company now has a modern, robust data platform. Unified Data Analytics Platform has helped the company to accelerate innovation by unifying data science with engineering and business.

• Reports are being generated using the data from one unified platform instead of pulling from data from different data sources.
• Data is being quickly prepared, cleaned at a massive scale with no limitations. These new data sources are being added to the company’s Data Lake with very less effort from the Data Engineering team that can be used to enable the ML use cases or other business use cases.
• The platform has enabled us to continuously train and deploy ML models for all of our artificial intelligence applications.
• Data access has been provided based on the respective groups and business needs

MS Products / Services used:

Microsoft Azure Data Bricks, Azure SQL Managed Instance, Azure DevOps