Create an Azure Data Factory using REST API - Azure Data Factory (2023)

  • Article
  • 12 minutes to read
  • Version 1
  • Current version

APPLIES TO: Create an Azure Data Factory using REST API - Azure Data Factory (1)Azure Data Factory Create an Azure Data Factory using REST API - Azure Data Factory (2)Azure Synapse Analytics

Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation. Using Azure Data Factory, you can create and schedule data-driven workflows (called pipelines) that can ingest data from disparate data stores, process/transform the data by using compute services such as Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics, and Azure Machine Learning, and publish output data to data stores such as Azure Synapse Analytics for business intelligence (BI) applications to consume.

This quickstart describes how to use REST API to create an Azure Data Factory. The pipeline in this data factory copies data from one location to another location in an Azure blob storage.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

Note

We recommend that you use the Azure Az PowerShell module to interact with Azure. See Install Azure PowerShell to get started. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.

  • Azure subscription. If you don't have a subscription, you can create a free trial account.
  • Azure Storage account. You use the blob storage as source and sink data store. If you don't have an Azure storage account, see the Create a storage account article for steps to create one.
  • Create a blob container in Blob Storage, create an input folder in the container, and upload some files to the folder. You can use tools such as Azure Storage Explorer to connect to Azure Blob storage, create a blob container, upload input file, and verify the output file.
  • Install Azure PowerShell. Follow the instructions in How to install and configure Azure PowerShell. This quickstart uses PowerShell to invoke REST API calls.
  • Create an application in Azure Active Directory following this instruction. Make note of the following values that you use in later steps: application ID, clientSecrets, and tenant ID. Assign application to "Contributor" role at either subscription or resource group level.

Note

For Sovereign clouds, you must use the appropriate cloud-specific endpoints for ActiveDirectoryAuthority and ResourceManagerUrl (BaseUri).You can use PowerShell to easily get the endpoint Urls for various clouds by executing “Get-AzEnvironment | Format-List”, which will return a list of endpoints for each cloud environment.

(Video) 40. Working with Token based REST API in Azure Data Factory

Set global variables

  1. Launch PowerShell. Keep Azure PowerShell open until the end of this quickstart. If you close and reopen, you need to run the commands again.

    Run the following command, and enter the user name and password that you use to sign in to the Azure portal:

    Connect-AzAccount

    Run the following command to view all the subscriptions for this account:

    Get-AzSubscription

    Run the following command to select the subscription that you want to work with. Replace SubscriptionId with the ID of your Azure subscription:

    Select-AzSubscription -SubscriptionId "<SubscriptionId>"
  2. Run the following commands after replacing the places-holders with your own values, to set global variables to be used in later steps.

    $tenantID = "<your tenant ID>"$appId = "<your application ID>"$clientSecrets = "<your clientSecrets for the application>"$subscriptionId = "<your subscription ID to create the factory>"$resourceGroupName = "<your resource group to create the factory>"$factoryName = "<specify the name of data factory to create. It must be globally unique.>"$apiVersion = "2018-06-01"

Authenticate with Azure AD

Run the following commands to authenticate with Azure Active Directory (AAD):

$credentials = Get-Credential -UserName $appIdConnect-AzAccount -ServicePrincipal -Credential $credentials -Tenant $tenantID

You will be prompt to input the password, use the value in clientSecrets variable.

If you need to get the access token

GetToken

Create a data factory

Run the following commands to create a data factory:

(Video) REST API Fetching only Required Records and Writing to Azure SQL DB in Azure Data Factory

$body = @"{ "location": "East US", "properties": {}, "identity": { "type": "SystemAssigned" }}"@$response = Invoke-AzRestMethod -SubscriptionId ${subscriptionId} -ResourceGroupName ${resourceGroupName} -ResourceProviderName Microsoft.DataFactory -ResourceType "factories" -Name ${factoryName} -ApiVersion ${apiVersion} -Method PUT -Payload ${body}$response.Content 

Note the following points:

  • The name of the Azure Data Factory must be globally unique. If you receive the following error, change the name and try again.

    Data factory name "ADFv2QuickStartDataFactory" is not available.
  • For a list of Azure regions in which Data Factory is currently available, select the regions that interest you on the following page, and then expand Analytics to locate Data Factory: Products available by region. The data stores (Azure Storage, Azure SQL Database, etc.) and computes (HDInsight, etc.) used by data factory can be in other regions.

Here is the sample response content:

{ "name":"<dataFactoryName>", "identity":{ "type":"SystemAssigned", "principalId":"<service principal ID>", "tenantId":"<tenant ID>" }, "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>", "type":"Microsoft.DataFactory/factories", "properties":{ "provisioningState":"Succeeded", "createTime":"2019-09-03T02:10:27.056273Z", "version":"2018-06-01" }, "eTag":"\"0200c876-0000-0100-0000-5d6dcb930000\"", "location":"East US", "tags":{ }}

Create linked services

You create linked services in a data factory to link your data stores and compute services to the data factory. In this quickstart, you only need create one Azure Storage linked service as both copy source and sink store, named "AzureStorageLinkedService" in the sample.

Run the following commands to create a linked service named AzureStorageLinkedService:

Replace <accountName> and <accountKey> with name and key of your Azure storage account before executing the commands.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/linkedservices/AzureStorageLinkedService?api-version=${apiVersion}"$body = @"{ "name":"AzureStorageLinkedService", "properties":{ "annotations":[ ], "type":"AzureBlobStorage", "typeProperties":{ "connectionString":"DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>" } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/linkedservices/AzureStorageLinkedService", "name":"AzureStorageLinkedService", "type":"Microsoft.DataFactory/factories/linkedservices", "properties":{ "annotations":[ ], "type":"AzureBlobStorage", "typeProperties":{ "connectionString":"DefaultEndpointsProtocol=https;AccountName=<accountName>;" } }, "etag":"07011a57-0000-0100-0000-5d6e14a20000"}

Create datasets

You define a dataset that represents the data to copy from a source to a sink. In this example, you create two datasets: InputDataset and OutputDataset. They refer to the Azure Storage linked service that you created in the previous section. The input dataset represents the source data in the input folder. In the input dataset definition, you specify the blob container (adftutorial), the folder (input), and the file (emp.txt) that contain the source data. The output dataset represents the data that's copied to the destination. In the output dataset definition, you specify the blob container (adftutorial), the folder (output), and the file to which the data is copied.

Create InputDataset

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/InputDataset?api-version=${apiVersion}"$body = @"{ "name":"InputDataset", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":{ "type":"AzureBlobStorageLocation", "fileName":"emp.txt", "folderPath":"input", "container":"adftutorial" } } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response 

Here is the sample output:

(Video) 32. Copy data from REST API which sends response in Pages using Azure data factory

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/datasets/InputDataset", "name":"InputDataset", "type":"Microsoft.DataFactory/factories/datasets", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":"@{type=AzureBlobStorageLocation; fileName=emp.txt; folderPath=input; container=adftutorial}" } }, "etag":"07011c57-0000-0100-0000-5d6e14b40000"}

Create OutputDataset

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/OutputDataset?api-version=${apiVersion}"$body = @"{ "name":"OutputDataset", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":{ "type":"AzureBlobStorageLocation", "folderPath":"output", "container":"adftutorial" } } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/datasets/OutputDataset", "name":"OutputDataset", "type":"Microsoft.DataFactory/factories/datasets", "properties":{ "linkedServiceName":{ "referenceName":"AzureStorageLinkedService", "type":"LinkedServiceReference" }, "annotations":[ ], "type":"Binary", "typeProperties":{ "location":"@{type=AzureBlobStorageLocation; folderPath=output; container=adftutorial}" } }, "etag":"07013257-0000-0100-0000-5d6e18920000"}

Create a pipeline

In this example, this pipeline contains one Copy activity. The Copy activity refers to the "InputDataset" and the "OutputDataset" created in the previous step as input and output.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartPipeline?api-version=${apiVersion}"$body = @"{ "name": "Adfv2QuickStartPipeline", "properties": { "activities": [ { "name": "CopyFromBlobToBlob", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }, "enableStaging": false }, "inputs": [ { "referenceName": "InputDataset", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "OutputDataset", "type": "DatasetReference" } ] } ], "annotations": [] }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/pipelines/Adfv2QuickStartPipeline", "name":"Adfv2QuickStartPipeline", "type":"Microsoft.DataFactory/factories/pipelines", "properties":{ "activities":[ "@{name=CopyFromBlobToBlob; type=Copy; dependsOn=System.Object[]; policy=; userProperties=System.Object[]; typeProperties=; inputs=System.Object[]; outputs=System.Object[]}" ], "annotations":[ ] }, "etag":"07012057-0000-0100-0000-5d6e14c00000"}

Create pipeline run

In this step, you trigger a pipeline run. The pipeline run ID returned in the response body is used in later monitoring API.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartPipeline/createRun?api-version=${apiVersion}"$response = Invoke-AzRestMethod -Path ${path} -Method POST $response.content 

Here is the sample output:

{ "runId":"04a2bb9a-71ea-4c31-b46e-75276b61bafc"}

You can also get the runId by using following command

($response.content | ConvertFrom-Json).runId

Parameterize your pipeline

You can create pipeline with parameters. In the following example, we will create an input dataset and an output dataset that can take input and output filenames as parameters given to the pipeline.

Create parameterized input dataset

Define a parameter called strInputFileName , and use it as file name for dataset.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/ParamInputDataset?api-version=${apiVersion}"$body = @"{ "name": "ParamInputDataset", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strInputFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strInputFileName", "type": "Expression" }, "folderPath": "input", "container": "adftutorial" } } }, "type": "Microsoft.DataFactory/factories/datasets"}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/datasets/ParamInputDataset", "name": "ParamInputDataset", "type": "Microsoft.DataFactory/factories/datasets", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strInputFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strInputFileName", "type": "Expression" }, "folderPath": "input", "container": "adftutorial" } } }, "etag": "00000000-0000-0000-0000-000000000000"}

Create parameterized output dataset

Define a parameter called strOutputFileName , and use it as file name for dataset.

(Video) How to Read the data from Rest API and write to Blob Storage in ADF By using REST Connector in ADF

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/ParamOutputDataset?api-version=${apiVersion}"$body = @"{ "name": "ParamOutputDataset", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strOutPutFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strOutPutFileName", "type": "Expression" }, "folderPath": "output", "container": "adftutorial" } } }, "type": "Microsoft.DataFactory/factories/datasets"}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/datasets/ParamOutputDataset", "name": "ParamOutputDataset", "type": "Microsoft.DataFactory/factories/datasets", "properties": { "linkedServiceName": { "referenceName": "AzureStorageLinkedService", "type": "LinkedServiceReference" }, "parameters": { "strOutPutFileName": { "type": "string" } }, "annotations": [], "type": "Binary", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "fileName": { "value": "@dataset().strOutPutFileName", "type": "Expression" }, "folderPath": "output", "container": "adftutorial" } } }, "etag": "00000000-0000-0000-0000-000000000000"}

Create parameterized pipeline

Define a pipeline with two pipeline level parameters: strParamInputFileName and strParamOutputFileName. Then link these two parameters to the strInputFileName and strOutputFileName parameters of the datasets.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartParamPipeline?api-version=${apiVersion}"$body = @"{ "name": "Adfv2QuickStartParamPipeline", "properties": { "activities": [ { "name": "CopyFromBlobToBlob", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }, "enableStaging": false }, "inputs": [ { "referenceName": "ParamInputDataset", "type": "DatasetReference", "parameters": { "strInputFileName": { "value": "@pipeline().parameters.strParamInputFileName", "type": "Expression" } } } ], "outputs": [ { "referenceName": "ParamOutputDataset", "type": "DatasetReference", "parameters": { "strOutPutFileName": { "value": "@pipeline().parameters.strParamOutputFileName", "type": "Expression" } } } ] } ], "parameters": { "strParamInputFileName": { "type": "String" }, "strParamOutputFileName": { "type": "String" } } }}"@$response = Invoke-AzRestMethod -Path ${path} -Method PUT -Payload $body$response.content

Here is the sample output:

{ "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelines/Adfv2QuickStartParamPipeline", "name": "Adfv2QuickStartParamPipeline", "type": "Microsoft.DataFactory/factories/pipelines", "properties": { "activities": [ { "name": "CopyFromBlobToBlob", "type": "Copy", "dependsOn": [], "policy": { "timeout": "7.00:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings", "recursive": true } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }, "enableStaging": false }, "inputs": [ { "referenceName": "ParamInputDataset", "type": "DatasetReference", "parameters": { "strInputFileName": { "value": "@pipeline().parameters.strParamInputFileName", "type": "Expression" } } } ], "outputs": [ { "referenceName": "ParamOutputDataset", "type": "DatasetReference", "parameters": { "strOutPutFileName": { "value": "@pipeline().parameters.strParamOutputFileName", "type": "Expression" } } } ] } ], "parameters": { "strParamInputFileName": { "type": "String" }, "strParamOutputFileName": { "type": "String" } } }, "etag": "5e01918d-0000-0100-0000-60d569a90000"}

Create pipeline run with parameters

You can now specify values of the parameter at the time of creating the pipeline run.

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartParamPipeline/createRun?api-version=${apiVersion}"$body = @"{ "strParamInputFileName": "emp2.txt", "strParamOutputFileName": "aloha.txt"}"@$response = Invoke-AzRestMethod -Path ${path} -Method POST -Payload $body$response.content$runId = ($response.content | ConvertFrom-Json).runId

Here is the sample output:

{"runId":"ffc9c2a8-d86a-46d5-9208-28b3551007d8"}

Monitor pipeline

  1. Run the following script to continuously check the pipeline run status until it finishes copying the data.

     $path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelineruns/${runId}?api-version=${apiVersion}" while ($True) { $response = Invoke-AzRestMethod -Path ${path} -Method GET $response = $response.content | ConvertFrom-Json Write-Host "Pipeline run status: " $response.Status -foregroundcolor "Yellow" if ( ($response.Status -eq "InProgress") -or ($response.Status -eq "Queued") -or ($response.Status -eq "In Progress") ) { Start-Sleep -Seconds 10 } else { $response | ConvertTo-Json break } }

    Here is the sample output:

     { "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelineruns/ffc9c2a8-d86a-46d5-9208-28b3551007d8", "runId": "ffc9c2a8-d86a-46d5-9208-28b3551007d8", "debugRunId": null, "runGroupId": "ffc9c2a8-d86a-46d5-9208-28b3551007d8", "pipelineName": "Adfv2QuickStartParamPipeline", "parameters": { "strParamInputFileName": "emp2.txt", "strParamOutputFileName": "aloha.txt" }, "invokedBy": { "id": "9c0275ed99994c18932317a325276544", "name": "Manual", "invokedByType": "Manual" }, "runStart": "2021-06-25T05:34:06.8424413Z", "runEnd": "2021-06-25T05:34:13.2936585Z", "durationInMs": 6451, "status": "Succeeded", "message": "", "lastUpdated": "2021-06-25T05:34:13.2936585Z", "annotations": [], "runDimension": {}, "isLatest": true }
  2. Run the following script to retrieve copy activity run details, for example, size of the data read/written.

     $path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelineruns/${runId}/queryActivityruns?api-version=${apiVersion}" while ($True) { $response = Invoke-AzRestMethod -Path ${path} -Method POST $responseContent = $response.content | ConvertFrom-Json $responseContentValue = $responseContent.value Write-Host "Activity run status: " $responseContentValue.Status -foregroundcolor "Yellow" if ( ($responseContentValue.Status -eq "InProgress") -or ($responseContentValue.Status -eq "Queued") -or ($responseContentValue.Status -eq "In Progress") ) { Start-Sleep -Seconds 10 } else { $responseContentValue | ConvertTo-Json break } }

    Here is the sample output:

     { "activityRunEnd": "2021-06-25T05:34:11.9536764Z", "activityName": "CopyFromBlobToBlob", "activityRunStart": "2021-06-25T05:34:07.5161151Z", "activityType": "Copy", "durationInMs": 4437, "retryAttempt": null, "error": { "errorCode": "", "message": "", "failureType": "", "target": "CopyFromBlobToBlob", "details": "" }, "activityRunId": "40bab243-9bbf-4538-9336-b797a2f98e2b", "iterationHash": "", "input": { "source": { "type": "BinarySource", "storeSettings": "@{type=AzureBlobStorageReadSettings; recursive=True}" }, "sink": { "type": "BinarySink", "storeSettings": "@{type=AzureBlobStorageWriteSettings}" }, "enableStaging": false }, "linkedServiceName": "", "output": { "dataRead": 134, "dataWritten": 134, "filesRead": 1, "filesWritten": 1, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "copyDuration": 3, "throughput": 0.044, "errors": [], "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (East US)", "usedDataIntegrationUnits": 4, "billingReference": { "activityType": "DataMovement", "billableDuration": "" }, "usedParallelCopies": 1, "executionDetails": [ "@{source=; sink=; status=Succeeded; start=06/25/2021 05:34:07; duration=3; usedDataIntegrationUnits=4; usedParallelCopies=1; profile=; detailedDurations=}" ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" }, "durationInQueue": { "integrationRuntimeQueue": 0 } }, "userProperties": {}, "pipelineName": "Adfv2QuickStartParamPipeline", "pipelineRunId": "ffc9c2a8-d86a-46d5-9208-28b3551007d8", "status": "Succeeded", "recoveryStatus": "None", "integrationRuntimeNames": [ "defaultintegrationruntime" ], "executionDetails": { "integrationRuntime": [ "@{name=DefaultIntegrationRuntime; type=Managed; location=East US; nodes=}" ] }, "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelineruns/ffc9c2a8-d86a-46d5-9208-28b3551007d8/activityruns/40bab243-9bbf-4538-9336-b797a2f98e2b" }

Verify the output

Use Azure Storage explorer to check the file is copied to "outputPath" from "inputPath" as you specified when creating a pipeline run.

Clean up resources

You can clean up the resources that you created in the Quickstart in two ways. You can delete the Azure resource group, which includes all the resources in the resource group. If you want to keep the other resources intact, delete only the data factory you created in this tutorial.

(Video) How to Use Rest API to Query Azure Data Factory From Web or By Using Postman ADF Tutorial 2022

Run the following command to delete the entire resource group:

Remove-AzResourceGroup -ResourceGroupName $resourcegroupname

Run the following command to delete only the data factory:

Remove-AzDataFactoryV2 -Name "<NameOfYourDataFactory>" -ResourceGroupName "<NameOfResourceGroup>"

Next steps

The pipeline in this sample copies data from one location to another location in an Azure blob storage. Go through the tutorials to learn about using Data Factory in more scenarios.

FAQs

Can Azure Data Factory connect to REST API? ›

Azure Data Factory allows you to add your required REST API Url and the authentication method to connect REST API to Azure products such as Azure SQL Database & Blob Storage.

How do I run Azure Data Factory pipeline using REST API? ›

Execute Azure ADF Pipeline using REST API
  1. Subscription ID.
  2. Tenant ID.
  3. Client ID.
  4. Client Secret.
  5. Authenticate using client_credentials.
  6. Request Type : POST.
  7. URL: https://login.microsoftonline.com/{{tenant_id}}/oauth2/token.
  8. Request Body:
Dec 5, 2019

How do I trigger Azure Data Factory pipeline from API? ›

To manually trigger a pipeline or configure a new scheduled, tumbling window, storage event, or custom event trigger, select Add trigger at the top of the pipeline editor. If you choose to manually trigger the pipeline, it will execute immediately.

Is Azure Data Factory an API? ›

Linked Services - REST API (Azure Data Factory)

How to connect REST API to database? ›

Connect from Postman

Using the excellent Postman app for accessing your database is a great way to explore and test your data REST API. Just install the Postman application and paste in a database REST url to get started. For authentication you also need to add a secure token as a HTTP Header value.

How do I call REST API from Azure pipeline? ›

Example
  1. From your pipeline definition, select the ellipsis button (...), and then select Add an agentless job.
  2. In your new agentless job, select the + sign to add a new task.
  3. Search for the Invoke REST API task. ...
  4. Select your Connection type and your Service connection. ...
  5. Select Save & queue when you are done.

How do I deploy REST API to Azure? ›

Publish the API app to Azure App Service
  1. In Solution Explorer, right-click the project and select Publish.
  2. In the Publish dialog, select Azure and select the Next button.
  3. Select Azure App Service (Windows) and select the Next button.
  4. Select Create a new Azure App Service. ...
  5. Select the Create button.
Nov 3, 2022

How do I create a dynamic pipeline in Azure data Factory? ›

The first step within the Azure Data Factory pipeline is to add a Lookup activity, which will retrieve the list of table names to load into the cloud from the metadata table. That set of table names will then be passed into a ForEach loop activity in order to process each table within the metadata list.

How do I run Azure rest APIs with Postman? ›

Use the Azure CLI to get a bearer token that you'll use to make API requests in Postman. Set up a Postman collection and configure the Postman REST client to use your bearer token to authenticate.

What is the best possible way to get started on creating an Azure data factory? ›

Advanced creation in the Azure portal
  1. Launch Microsoft Edge or Google Chrome web browser. ...
  2. Go to the Azure portal data factories page.
  3. After landing on the data factories page of the Azure portal, click Create.
  4. For Resource Group, take one of the following steps: ...
  5. For Region, select the location for the data factory.
Dec 13, 2022

How can I invoke an Azure Data factory pipeline from C# code? ›

  1. Specify the source Azure Blob information.
  2. Specify the Azure Cosmos DB information.
  3. Specify the Linked Service Names and Dataset Names.
  4. We can authenticate and create a data factory management client.
  5. Create data factory and wait.
  6. Create an Azure Blob Storage linked service.
  7. Create an Azure Cosmos DB linked service.
Oct 16, 2018

What are the types of triggers in Azure Data factory? ›

Azure Data Factory Triggers come in three different types: Schedule Trigger, Tumbling Window Trigger, and Event-based Trigger.

Does Azure support REST API? ›

The Azure Support REST API enables you to create and manage Azure support tickets programmatically. Using the API, you can: Open a technical, billing, subscription management, or subscription and service limits (quota) support ticket. Get a list of support tickets and detailed information about each ticket.

Does Azure have REST API? ›

Welcome to the Azure REST API Reference. Representational State Transfer (REST) APIs are service endpoints that support sets of HTTP operations (methods), which provide create/retrieve/update/delete access to the service's resources.

Is Azure data/factory ETL or ELT? ›

With Azure Data Factory, it is fast and easy to build code-free or code-centric ETL and ELT processes.

How do I call SOAP API from Azure Data Factory? ›

How to Connect to a SOAP API from Azure Data Factory
  1. Step 1 - Create Linked Service. Begin by creating a linked service. ...
  2. Step 2 - Create Dataset. Create a new dataset for your API. ...
  3. Step 3 - Create Pipeline. To use your SOAP API connector, create a pipeline. ...
  4. Step 4 - Determine SOAP Headers and Body.
Jul 28, 2021

What are the limitations of Azure Data Factory? ›

Version 2
ResourceDefault limitMaximum limit
Total number of entities, such as pipelines, data sets, triggers, linked services, Private Endpoints, and integration runtimes, within a data factory5,000Contact support.
Total CPU cores for Azure-SSIS Integration Runtimes under one subscription64Contact support.
29 more rows

Does Azure Devops support REST API? ›

A required HTTP method (also known as an operation or verb), which tells the service what type of operation you are requesting. Azure REST APIs support GET, HEAD, PUT, POST, and PATCH methods.

Videos

1. Data Factory: API data to Azure Table storage
(javiBI)
2. Read the data from Rest API JSON by using REST Connector and Write to Azure SQL DB in ADF
(TechBrothersIT)
3. Azure Data Factory (ADF) Quick Tip: Invoking REST APIs Using External Call Transform
(Data Platform Central)
4. DataFactory: Extract data from SharePoint Online
(javiBI)
5. Azure Data Factory - Executing a Pipeline from Azure Logic Apps using REST End Point!
(Azure Made Simple)
6. #26. Copy data from REST API which sends the response in Pages| Azure Data Factory Tutorial |
(GeekCoders)
Top Articles
Latest Posts
Article information

Author: Prof. Nancy Dach

Last Updated: 12/22/2022

Views: 5696

Rating: 4.7 / 5 (77 voted)

Reviews: 92% of readers found this page helpful

Author information

Name: Prof. Nancy Dach

Birthday: 1993-08-23

Address: 569 Waelchi Ports, South Blainebury, LA 11589

Phone: +9958996486049

Job: Sales Manager

Hobby: Web surfing, Scuba diving, Mountaineering, Writing, Sailing, Dance, Blacksmithing

Introduction: My name is Prof. Nancy Dach, I am a lively, joyous, courageous, lovely, tender, charming, open person who loves writing and wants to share my knowledge and understanding with you.