Handling Paginated Results from Microsoft Graph with Azure Data Factory

Paginated Graph Results with @odata.nextlink in Azure Data Factory

UpdatedJanuary 2, 2025

Handling Paginated Results from Microsoft Graph with Azure Data Factory

I love backpacking and adventuring, but I am particularly obsessed with exploring the Grand Canyon and its surrounding tributaries.

My hobbies include photography, sports, reading and PC gaming. I also have a love/hate relationship with a vintage Suzuki motorcycle.

Father of two remarkable children and husband to an equally remarkable woman.

"I'll only be this young for one more day"

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Custom Table of Contents for SharePoint Page Libraries - JSON View Formatting

Create a Table of Contents for Site Page libraries in SharePoint. Easily implement site-wide, via web parts. Supports auto updates and additions.

Jan 19, 2026

Custom Table of Contents for SharePoint Page Libraries - JSON View Formatting

Custom Logging in Power Automate

Easy, practical DIY approach using SharePoint Online lists

Jan 1, 2025

Create Planner Tasks with Microsoft Form Attachments

Step-by-Step Guide to Linking Form Attachments to Planner Tasks with Power Automate and Logic Apps

Jun 2, 2023

Create Planner Tasks with Microsoft Form Attachments

Extract Excel Worksheet Data Without Tables - Power Automate

A reasonably sound method to extract unstructured data from Excel worksheets

Sep 22, 2022

Extract Excel Worksheet Data Without Tables - Power Automate

Ham Sandwich

10 posts

Articles and step-by-step guides for Microsoft Power Platform, M365 and SharePoint Online. Plus, a few Azure and Copilot Studio tidbits thrown in here and then.

This topic is covered in a handful of blog posts, forum topics, etc. but none of the information I found recently got me 100% of the way there. That can happen with methods involving two evolving services - Azure Data Factory and Microsoft Graph - information becomes out of date.

Here's what worked for me in early 2023!

Prerequisites

Even though some of these are detailed topics, I am going to gloss over them since they are covered in-depth elsewhere on the web:

Register an Azure App for use as a service principal (later) with your data factory
Grant the application appropriate roles for your intended use of Graph
Store the client secret value for the app in Azure Key Vault
Enable managed identities for your data factory
Give the managed identity for your data factory the appropriate role on the Key Vault (to retrieve stored client secret)
Configure a data sink to store the Graph results, and ensure the data factory has permission to write there
Create a new Linked Service within ADF (type: REST)
1. Set the Authentication type as Service Principal
2. Use the Azure Key Vault method to retrieve the client secret from AKV
3. AAD Resource: https://graph.microsoft.com/
4. Parameterize the Base URL value

This will get you something like this:

Dataset, Pipeline and Activity Setup

Create a new Dataset using the new Linked Service you created (above steps)
1. You can parameterize the baseURL value if you'd like to use one Dataset for multiple Graph queries.
Otherwise, provide a fixed Graph API endpoint value within the Dataset, like:

https://graph.microsoft.com/v1.0/users?$select=id,accountEnabled,createdDateTime,department,displayName,employeeId,employeeType,jobTitle,mail,userPrincipalName,userType,manager,employeeHireDate&$expand=manager($select=id)
Create a new Pipeline and add a Copy Data activity
Configure the Copy Data activity to use the Dataset you created
Delete the automatically-supplied Pagination criteria (RFC5988)
Add a new Pagination Rule, like this:

Your source definition within the Copy activity should look something like this:

"source": {
    "type": "RestSource",
    "httpRequestTimeout": "00:01:40",
    "requestInterval": "00.00:00:00.010",
    "requestMethod": "GET",
    "paginationRules": {
        "AbsoluteUrl": "$['@odata.nextLink']"
    }
}

Now, when your Copy Data activity executes, it will keep fetching results from Graph until the @odata.nextlink is no longer present in the result body, at which point it will start the sink activity.

Note: this is my first time using the new Neptune editor on Hashnode ... forgive any oddities with the numbered list formatting... I am in a bit of a hurry and will try to fix the markdown later

Command Palette

Comments

More from this blog

Prerequisites

Dataset, Pipeline and Activity Setup