Azure Data Lake Store uses Azure Active Directory for authentication. Before authoring an application that works with Azure Data Lake Store, you must decide how to authenticate your application with Azure Active Directory (Azure AD). The two main options available are:

• End-user authentication
• Service-to-service authentication

Both these options result in your application being provided with an OAuth 2.0 token, which gets attached to each request made to Azure Data Lake Store.

We have created the end-user authentication method by following this link: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal. The user creating it should be configured as the owner of the resource and also as an admin in the active directory resource.

In the Azure portal, inside all resources, click on the Azure Active Directory resource:

Click on App registrations and New application registration:

Provide a name and URL for the application. Select Web app / API for the type of application you want to create and click on Create. The URL will look something like this: https://<your domain>.com/ADFV2BookApp.

Once the application is registered, we need to grant it permissions on the Blob Storage and the Azure Data Lake Analytics resource. Open Data Lake Analytics Resource | Access Control IAM and add the application as a contributor:

Similarly for the Blob Storage: open the Blob Storage Resource | Access Control IAM and add the application as a contributor.

So now we are ready to add the U-SQL task. Go back to the new pipeline and move the U-SQL task to the right; drag the arrow from the copy task to the U-SQL task:

Click on the U-SQL task and choose the new ADLA linked service. Test the connection; it should be successful:

In the Script tab, you can either load a script from the blob storage or from your local machine. The script should be the same one as we have previously used.

In our case, the U-SQL script described in the next section is saved on the blob Storage, so we are choosing the blob storage and then loading the script to the task. Make sure that the script was tested so that it does not fail.

You can validate, debug, and publish changes:

Ensure that it runs successfully. You can check it from the notification icon on the top right:

The last step is to verify that the output file was created. You can do that by opening the Blob Storage Resource | Storage Explorer.