Deploying Azure Data Factory With CI/CD Using Azure Pipeline

This video will focus on how to use Azure Data Factory CI/CD Pipelines and promote ETL Pipelines from Dev environment to UAT then production. How to create an Azure DevOps pipeline to deploy Azure Data Factory pipelines, link services, triggers, datasets, etc to several Azure Data Factories.

Learn a complete tutorial for creating Azure Data Factory CI/CD pipelines with Git enabled that use different branches for developing new features for ETL pipelines and other Azure Data Factory components.

This will cover, creating three Azure Data Factories, two Azure Storage accounts, Azure Release pipeline with Azure Artifacts based on Git repo and deploying Azure Data Factory ARM templates files to another environment.

It will show how to develop all the Azure Data Factory components in the feature branch. Then how to raise a pull request to merge those changes with the master branch. After that publish Azure Data Factory development which will create adf_publish branch which will include the ARM template that describes the current Azure Data Factory Environment. And by using this ARM template, we can deploy it to UAT and Production environments.

Commands:


Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $false


Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $true


Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false


Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $true

-factoryName "[target ADF name]" -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""

-factoryName [target ADF name] -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""

-factoryName "ADF-Radwan-UAT" -AzureBlobStorage1_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan1;AccountKey=xx;EndpointSuffix=core.windows.net" -AzureBlobStorage2_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan2;AccountKey=xxx;EndpointSuffix=core.windows.net"

-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false


 About the Author
-----------------------------
Video:
   • Mohamed Radwan is a Principal DevOps Consu...  
Blog:
https://mohamedradwan.com
Linked-in
  / mohamedahmedradwan  
GitHub
https://github.com/MohamedRadwan-DevOps
Twitter:
  / mradwan06  

DevOps #Azure #Azure_Data_Factory_tutorial #Azure_training #Azure_DevOps