November 12, 2019
Integrating Data With Azure Data Factory
Azure Data Factory (ADF) is a data integration service that enables customers to move, process and store large quantities of data with ease.
This demo will show how to setup a complete ADF pipeline. In this pipeline, we will:
- Load data from a SQL Server table into the raw zone of a data lake
- Conform the data in the data lake using an Azure Data Lake Analytics job
- Load the conformed data into an Azure SQL Data Warehouse for use with BI tools
Along the way, we'll touch on subjects such as writing U-SQL queries, automating pipelines with triggers, and some of the pros and cons of serverless technologies.
Azure Data Factory is a great solution when:
- I need to continuously load data from on-premise into the cloud, or from one place in the cloud to another
- I need the scalability and flexibility to accommodate any amount of data stored in any number of storage technologies
- I want to minimize the amount of custom code I develop so I can focus on the business logic