Using ADF Utilities Custom ARM Template Parameters Definition

In the previous post, I covered the use of ADF Utilities to validate and export ARM Templates for Azure Data Factory deployments. The process also covers the option to use ARM Template parameters to provide environment specific configuration.

ARM templates are limited to 256 parameters in the template, so there are situations where some of the parameters need not to be exposed to avoid reaching this limit. Also in some cases the default name of the parameter is too long and need to be shortened for ease of use.

There are other cases where you might need to parameterise a property in the ARM template, for example the frequency of a trigger, but this property is not exposed as a parameter to the ARM template.

In this post:

• ARM Template Parameters Definition File
• Linked Services and Global Parameters
  • Summary of Syntax
• Shortening Global Parameters Names
• Changing Trigger Settings
• Additional Examples

ARM Template Parameters Definition File

The arm-template-parameters-definition.json file (item 4 in the screen shot below) provides additional instructions to the export process in ADF utility to parameterise the ARM template. This file must be created under the root of the ADF code folder.

The usage of the file is described in the ARM templates custom parameters section in the online docs.

In this post I will provide some explanation and examples for using this definition file.

Linked Services and Global Parameters

When customising ARM template parameter definitions, start by including the entry for linked services, and optionally ADF global parameters. Whilst these are included in the ARM templates by default, introducing the parameters definition file without explicitly including these entries will omit the linked services and global params sections from the output ARM templates and parameters.

{
    "Microsoft.DataFactory/factories/linkedServices": {
        "*": {
            "properties": {
                "typeProperties": {
                    "*": "="
                }
            }
        }
    },
    "Microsoft.DataFactory/factories": {
        "properties": {
            "globalParameters": {
                "*": {
                    "value": "="
                }
            }
        }
    }
}

The above example will achieve the following two results:

• Linked Services: properties under typeProperties for all entries of type Microsoft.DataFactory/factories/linkedServices are parameterised.
• Global Parameters: value property for all entries under globalParameters is parameterised.

In both cases, property name is set to "*", which indicates that all properties at the specified level in the template should be parameterised.

The "value" of the property was set to "=" keeping the current value as default the output parameters file, other options include - to not keep the value in the parameter file, and | as a special case for secretes from Azure Key Vault.

The above example has almost the same effect as the default behavior (see what the output was before) of ADF Utils export command. Notice that the region for Data Factory deployment is not included any more:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "factoryName": {
            "value": "magdatafactory-dev-mutaz"
        },
        "storagegen2_properties_typeProperties_url": {
            "value": "https://magdatafactorysotrage.dfs.core.windows.net"
        },
        "dataFactory_properties_globalParameters_global_param1_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_another_param_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_and_another_one_value": {
            "value": "v1"
        }
    }
}

Summary of Syntax

Setting the value of a property as a string indicates that you want to parameterize the property. Use the format <action>:<name>:<stype>.

• <action> can be one of these characters:
  • = means keep the current value as the default value for the parameter.
  • - means don’t keep the default value for the parameter.
  • | is a special case for secrets from Azure Key Vault for connection strings or keys.
• <name> is the name of the parameter. If it’s blank, it takes the name of the property. If the value starts with a - character, the name is shortened. For example, AzureStorage1_properties_typeProperties_connectionString would be shortened to AzureStorage1_connectionString.
• <stype> is the type of parameter. If <stype> is blank, the default type is string. Supported values: string, securestring, int, bool, object, secureobject and array.

Click here for for the full custom parameter syntax specs.

Shortening Global Parameters Names

In the above example, I want to shorten the name of global_param1 to p1, while keeping all the other global parameters unchanged. This can be achieved by adding another filter in the definition file for handling this requirement:

{
    ...
    "Microsoft.DataFactory/factories": {
        "properties": {
            "globalParameters": {
                "global_param1": {
                    "value": "=:-p1"
                },
                "*": {
                    "value": "="
                }
            }
        }
    }
    ...
}

• "global_param1" entry definition, with "value" property set to "=:-p1" instructs the ADF utils to keep create the parameter with the name changed to p1.

After running the export command, the output of this setting is:

{
    ...
        "dataFactory_p1": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_another_param_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_and_another_one_value": {
            "value": "v1"
        }
    ...
}

Notice how the parameter name is now dataFactory_p1, instead of the default full path name dataFactory_properties_globalParameters_global_param1_value.

Changing Trigger Settings

In this example, a trigger defined the dev environment, is configured to run on a particular day, e.g on Saturday, but when deployed to the testing environment, the trigger needs to be configured to run on a different day, e.g on Sunday.

The schedule of a trigger is not a property that can be parameterised by default in ADF UI. To change that, we will modify the definition file and include a filter to inject a parameter in the correct place in the ARM template for trigger schedule.

First here is a sample weekly trigger in the ARM template, this trigger runs on a specific day of the week:

  ...
  {
    "name": "OnceAWeekTrigger",
    "type": "Microsoft.DataFactory/factories/triggers",
    "apiVersion": "2018-06-01",
    "properties": {
            "annotations": [],
            "runtimeState": "Stopped",
            "pipelines": [],
            "type": "ScheduleTrigger",
            "typeProperties": {
                "recurrence": {
                    "frequency": "Week",
                    "interval": 1,
                    "startTime": "2021-05-25T22:59:00Z",
                    "timeZone": "UTC",
                    "schedule": {
                        "weekDays": [
                            "Sunday"
                        ]
                    }
                }
            }
        }
    }
    ...

The weekDays property under typeProperties : recurrence : schedule needs to be parameterised for resources of type Microsoft.DataFactory/factories/triggers in the ARM template.

The following entry in the ARM template parameters definition can achieve this requirement, it specifies that you want to parameterise weekDays property, and give it a shortened name weekdays, and that this property is of type array.

    ...
    "Microsoft.DataFactory/factories/triggers": {
        "properties": {
            "typeProperties": {
                "recurrence": {
                    "schedule": {
                        "weekDays": "=:-weekDays:array"
                    }
                }
            }
        }
    }
    ...

After running the ADF Utilities export function, the ARM template is modified as follows:

     ...
        {
        "name": "[concat(parameters('factoryName'), '/OnceAWeekTrigger')]",
        "type": "Microsoft.DataFactory/factories/triggers",
        "apiVersion": "2018-06-01",
        "properties": {
            "annotations": [],
            "runtimeState": "Stopped",
            "pipelines": [],
            "type": "ScheduleTrigger",
            "typeProperties": {
                "recurrence": {
                    "frequency": "Week",
                    "interval": 1,
                    "startTime": "2021-05-25T22:59:00Z",
                    "timeZone": "UTC",
                    "schedule": {
                        "weekDays": "[parameters('OnceAWeekTrigger_weekDays')]"
                    }
                }
            }
        }
    ...

And the parameters file will include the new weekDays parameter entry:

{
    ...
        "OnceAWeekTrigger_weekDays": {
            "value": [
                "Sunday"
            ]
        }
    ...
}

Additional Examples

Online documentation include additional examples of the same technique to achieve some common tasks, like parameterising a Databricks interactive cluster ID property in a linked service.