Mutaz’s Data Insights!

Using ADF Utilities Custom ARM Template Parameters Definition

2021-07-01T00:00:00+10:00

In the previous post, I covered the use of ADF Utilities to validate and export ARM Templates for Azure Data Factory deployments. The process also covers the option to use ARM Template parameters to provide environment specific configuration.

ARM templates are limited to 256 parameters in the template, so there are situations where some of the parameters need not to be exposed to avoid reaching this limit. Also in some cases the default name of the parameter is too long and need to be shortened for ease of use.

There are other cases where you might need to parameterise a property in the ARM template, for example the frequency of a trigger, but this property is not exposed as a parameter to the ARM template.

In this post:

ARM Template Parameters Definition File
Linked Services and Global Parameters
- Summary of Syntax
Shortening Global Parameters Names
Changing Trigger Settings
Additional Examples

ARM Template Parameters Definition File

The arm-template-parameters-definition.json file (item 4 in the screen shot below) provides additional instructions to the export process in ADF utility to parameterise the ARM template. This file must be created under the root of the ADF code folder.

The usage of the file is described in the ARM templates custom parameters section in the online docs.

In this post I will provide some explanation and examples for using this definition file.

Linked Services and Global Parameters

When customising ARM template parameter definitions, start by including the entry for linked services, and optionally ADF global parameters. Whilst these are included in the ARM templates by default, introducing the parameters definition file without explicitly including these entries will omit the linked services and global params sections from the output ARM templates and parameters.

{
    "Microsoft.DataFactory/factories/linkedServices": {
        "*": {
            "properties": {
                "typeProperties": {
                    "*": "="
                }
            }
        }
    },
    "Microsoft.DataFactory/factories": {
        "properties": {
            "globalParameters": {
                "*": {
                    "value": "="
                }
            }
        }
    }
}

The above example will achieve the following two results:

Linked Services: properties under typeProperties for all entries of type Microsoft.DataFactory/factories/linkedServices are parameterised.
Global Parameters: value property for all entries under globalParameters is parameterised.

In both cases, property name is set to "*", which indicates that all properties at the specified level in the template should be parameterised.

The "value" of the property was set to "=" keeping the current value as default the output parameters file, other options include - to not keep the value in the parameter file, and | as a special case for secretes from Azure Key Vault.

The above example has almost the same effect as the default behavior (see what the output was before) of ADF Utils export command. Notice that the region for Data Factory deployment is not included any more:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "factoryName": {
            "value": "magdatafactory-dev-mutaz"
        },
        "storagegen2_properties_typeProperties_url": {
            "value": "https://magdatafactorysotrage.dfs.core.windows.net"
        },
        "dataFactory_properties_globalParameters_global_param1_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_another_param_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_and_another_one_value": {
            "value": "v1"
        }
    }
}

Summary of Syntax

Setting the value of a property as a string indicates that you want to parameterize the property. Use the format <action>:<name>:<stype>.

<action> can be one of these characters:
- = means keep the current value as the default value for the parameter.
- - means don’t keep the default value for the parameter.
- | is a special case for secrets from Azure Key Vault for connection strings or keys.
<name> is the name of the parameter. If it’s blank, it takes the name of the property. If the value starts with a - character, the name is shortened. For example, AzureStorage1_properties_typeProperties_connectionString would be shortened to AzureStorage1_connectionString.
<stype> is the type of parameter. If <stype> is blank, the default type is string. Supported values: string, securestring, int, bool, object, secureobject and array.

Click here for for the full custom parameter syntax specs.

Shortening Global Parameters Names

In the above example, I want to shorten the name of global_param1 to p1, while keeping all the other global parameters unchanged. This can be achieved by adding another filter in the definition file for handling this requirement:

{
    ...
    "Microsoft.DataFactory/factories": {
        "properties": {
            "globalParameters": {
                "global_param1": {
                    "value": "=:-p1"
                },
                "*": {
                    "value": "="
                }
            }
        }
    }
    ...
}

"global_param1" entry definition, with "value" property set to "=:-p1" instructs the ADF utils to keep create the parameter with the name changed to p1.

After running the export command, the output of this setting is:

{
    ...
        "dataFactory_p1": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_another_param_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_and_another_one_value": {
            "value": "v1"
        }
    ...
}

Notice how the parameter name is now dataFactory_p1, instead of the default full path name dataFactory_properties_globalParameters_global_param1_value.

Changing Trigger Settings

In this example, a trigger defined the dev environment, is configured to run on a particular day, e.g on Saturday, but when deployed to the testing environment, the trigger needs to be configured to run on a different day, e.g on Sunday.

The schedule of a trigger is not a property that can be parameterised by default in ADF UI. To change that, we will modify the definition file and include a filter to inject a parameter in the correct place in the ARM template for trigger schedule.

First here is a sample weekly trigger in the ARM template, this trigger runs on a specific day of the week:

  ...
  {
    "name": "OnceAWeekTrigger",
    "type": "Microsoft.DataFactory/factories/triggers",
    "apiVersion": "2018-06-01",
    "properties": {
            "annotations": [],
            "runtimeState": "Stopped",
            "pipelines": [],
            "type": "ScheduleTrigger",
            "typeProperties": {
                "recurrence": {
                    "frequency": "Week",
                    "interval": 1,
                    "startTime": "2021-05-25T22:59:00Z",
                    "timeZone": "UTC",
                    "schedule": {
                        "weekDays": [
                            "Sunday"
                        ]
                    }
                }
            }
        }
    }
    ...

The weekDays property under typeProperties : recurrence : schedule needs to be parameterised for resources of type Microsoft.DataFactory/factories/triggers in the ARM template.

The following entry in the ARM template parameters definition can achieve this requirement, it specifies that you want to parameterise weekDays property, and give it a shortened name weekdays, and that this property is of type array.

    ...
    "Microsoft.DataFactory/factories/triggers": {
        "properties": {
            "typeProperties": {
                "recurrence": {
                    "schedule": {
                        "weekDays": "=:-weekDays:array"
                    }
                }
            }
        }
    }
    ...

After running the ADF Utilities export function, the ARM template is modified as follows:

     ...
        {
        "name": "[concat(parameters('factoryName'), '/OnceAWeekTrigger')]",
        "type": "Microsoft.DataFactory/factories/triggers",
        "apiVersion": "2018-06-01",
        "properties": {
            "annotations": [],
            "runtimeState": "Stopped",
            "pipelines": [],
            "type": "ScheduleTrigger",
            "typeProperties": {
                "recurrence": {
                    "frequency": "Week",
                    "interval": 1,
                    "startTime": "2021-05-25T22:59:00Z",
                    "timeZone": "UTC",
                    "schedule": {
                        "weekDays": "[parameters('OnceAWeekTrigger_weekDays')]"
                    }
                }
            }
        }
    ...

And the parameters file will include the new weekDays parameter entry:

{
    ...
        "OnceAWeekTrigger_weekDays": {
            "value": [
                "Sunday"
            ]
        }
    ...
}

Additional Examples

Online documentation include additional examples of the same technique to achieve some common tasks, like parameterising a Databricks interactive cluster ID property in a linked service.

Using ADF Utilities for building ADF ARM Templates

2021-07-01T00:00:00+10:00

Azure Data Factory portal is the tool for authoring and publishing ADF artifacts, that is pipelines, data sets, linked services, etc. And whilst the portal allows you to integrate with Git code repos for source control, you are still required to validate and then publish the code manually to a live instance of ADF to generate deployment ARM templates.

In this series of posts, I will use ADF Utilities for validating and building ADF ARM templates. The ARM templates can then be used to deploy the ADF solution artifacts to another instance of ADF.

In this post:

Development Environment Setup
Deployment Workflow using ADF Utilities
Preparation to Run ADF Utils
Validate and Export Commands
Deploy the ARM template
- Customising ARM Template Parameters Definition

Development Environment Setup

For this exercise, my setup is:

ADF Development:
- dev-mutaz: developer/data engineer instance to build the ADF solution and its artifacts. ADF Portal is the tool of choice to visually author pipelines. This instance of the ADF service is integrated with a code repo, all changes are saved and committed to the code repo. Manual validation and publishing is required to promote changes and make them live in this ADF instance. -dev-adfutils: DevOps engineer instance to build and test process for generating ARM template files, custom parameters, pre/post and deployment scripts.
ADF Testing:
- test-cicd: Automated deployment instance, this resembles higher environments like test, pre-prod or prod environments. The subject of deploying to higher environments using CI/CD is covered in a following post.

The rest of this post is focused on preparing a local environment to run ADF Utilities using command line, this local setup can be used to for validating and exporting ARM templates, using custom parameters with ARM, and customising parameters definitions and unit testing pre/post and deployment scripts. This outcome of this step is then used to deploy an ADF solution to a new instance, dev-adfutils from local.

Deployment Workflow using ADF Utilities

In this process:

Azure Data Factory Studio is used to build the solution in a dev instance
Changes to ADF artifacts are automatically saved and committed to a Git repo collaboration branch (or feature branch)
Code is pulled locally from feature branch and ADF Utils tool is run to validate ADF JSON artifacts
ADF Utils export command is used to build ARM templates and the accompanying template parameters files
Azure CLI is used to deploy the ARM template to a live instance of ADF
Test that deployment customisation is correct

Preparation to Run ADF Utils

Git Integration

Ensure that the development instance of ADF is connected to a Git repo.

Folder Structure

Clone the feature branch locally and open it in VSCode.

Item	Description
(1)	ADF code folder, as specified by root folder when setting up the git integration.
(2)	Build subfolder, manually created and contains the file `package.json`, this file is used to install ADF Utils. ARM templates and other build artifacts will be stored in this folder.
(3)	ADF publishing settings file.
(4)	ARM templates parameters definition file, more on this file later.

Install ADF Utilities

To prepare for using ADF Utils, you will need to create the npm package file (package.json) to install the utilities. I suggest placing it inside a build subfolder, as indicated by number (2) in the previous screenshot.

package.json file:

{
    "scripts": {
        "build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
    },
    "dependencies": {
        "@microsoft/azure-data-factory-utilities": "^0.1.5"
    }
}

Install nodejs, if not already installed on the local development computer.

In the command line, or a nodejs console, run package installation command from the build folder:

npm install

The npm installation will create a folder named node_modules, add this folder to the .gitignore as you do not need to commit it to the source code repo.

Validate and Export Commands

The commands to validate ADF json solution artifacts and export corresponding ARM templates will process the source code from the local clone of the Git repo, you need to provide the details of the ADF dev instance so that these scripts can parse and process correctly, the following are values to update in the command lines later:

Values	Description
Subscription ID	subscription id in which your dev instance is created
Resource Group Name	resource group name in which your dev instance is created
ADF Name	name of the ADF dev instance

Validate Command

The validate command requires a publish_config.json file in the ADF code root folder, it specifies the name of the publishing branch for creating ARM templates when using the publish command in the UI.

{"publishBranch":"adf_publish"}

The command to validate the the ADF json code, similar to the validate all command in the ADF UI, is:

npm run build validate c:\git\adf /subscriptions/<Subscription ID>/resourceGroups/<Resource Group Name>/providers/Microsoft.DataFactory/factories/<ADF Name>

Export Command

The export command will process the ADF code files to generate corresponding ARM templates and parameters file. You need to run this command after validation is completed successfully.

npm run build export c:\git\adf /subscriptions/<Subscription ID>/resourceGroups/<Resource Group Name>/providers/Microsoft.DataFactory/factories/<ADF Name> "ArmTemplate"

The command create an output folder, ArmTemplate, with the following two key files: ‘ArmTemplateforFactory.json’ and ARMTemplateParamatersForFactory.json.

At this stage, you can make a copy of the ARMTemplateParamatersForFactory.json for environment specific parameter values. The template parameters file by default includes connection properties of linked services.

To further control parameters output file, we will look at the use of arm-template-parameters-definition.json later in the post.

Include Global Parameters in the ARM template

ADF global parameters can also be included in the ARM template parameters, you can achieve this by enabling the option to add global parameters to tha ARM template through ADF Portal UI.

After making the change, ADF will add the option includeFactoryTemplate in publish_config.json file.

{"publishBranch":"adf_publish","includeFactoryTemplate":true}

Running the export command after enabling global parameters in the ARM template will inject global parameters in the ARM template parameters file.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "factoryName": {
            "value": "magdatafactory-dev-mutaz"
        },
        "storagegen2_properties_typeProperties_url": {
            "value": "https://magdatafactorysotrage.dfs.core.windows.net"
        },
        "dataFactory_properties_globalParameters_global_param1_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_another_param_value": {
            "value": "v1"
        },
        "dataFactory_properties_globalParameters_and_another_one_value": {
            "value": "v1"
        },
        "dataFactory_location": {
            "value": "australiaeast"
        }
    }
}

The names of parameters take on the full name of the property in the ARM template. For example, storagegen2_properties_typeProperties_url is the url property of the linked service storagegen2. Similarly for global parameters, dataFactory_properties_globalParameters_another_param_value is the paramaterised value for ADF global parameter another_param. These names can be shortened by using the ARM custom parameters definition template, which makes it easier when later on you need to write custom parameters files.

Deploy the ARM template

The result ARM template and paramaters can be deployed to an ADF instance, dev-adfutils in this example, using various methods, e.g:

Deploy using Azure Portal: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-portal
Deploy using Cli: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-cli
Deploy using Azure pipelines: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-tutorial-pipeline

To deploy the ARM template from local computer, you need first to install Azure CLI, update the deployment parameters file, in this case I created a copy with correct values named dev-adfutils.paramaters.json, and run the command az deployment group create command to deploy to Azure.

$rgname = 'datafactorydev'
$templatefile = '.\ArmTemplate\ARMTemplateForFactory.json'
$parameters = '@.\dev-adfutils.parameters.json'
$deploymentname = $('ExampleDeployment_'+$(Get-Date).tostring("yyyyMMdd.HHmmss"))
$subscriptionname = 'Azure MSDN 01'


echo $('switch subscription to ' + $subscriptionname)
az account set --subscription $subscriptionname
az account list --output table

echo $('create deployment '  + $deploymentname + ', in rg: ' + $rgname)
az deployment group create --name $deploymentname --resource-group $rgname --template-file $templatefile --parameters $parameters

Customising ARM Template Parameters Definition

There are cases where you want to override the parameters template, these include reducing the number of parameters in the deployment or parameter names. But there are also situations where you may need to parameterise a property that is not offered as a parameter by default, like for instance schedule properties of an ADF trigger.

The next post on Using the ADF Utilities Custom ARM Template Parameters Definition covers the options available through the use of arm-template-parameters-definition.json file in ADF Utils to achieve this granular level of customisation.

Number Plate Recognition

2020-07-12T00:00:00+10:00

Automatic Number Plate Recognition (ANPR) is used in a variety of applications, such as in parking and ticketless fee automation. In such applications, cameras are usually mounted at the entry points to capture the number plate for entering and exiting vehicles, and perhaps most known, and loathed, use of ANPR is fixed speed cameras. In all these applications, the camera is in a fixed location, with a controlled set up to capture the picture and then process it to detect the number plate.

But what if we needed ANRP with handheld devices, and not in a fixed and controlled setup where image parameters are always the same.

In this example, I will utilise Microsoft’s custom vision service to detect number plate objects in a picture, then pass the number plate object to the Read API service in Azure Cognitive Services to extract the plate number.

The Approach

In this example, a custom object detection model is used first, followed by an OCR step to recognise numbers/digits in a number plate with a high level of certainty. I am assuming that images could come in different qualities, and could be captured by handheld devices (as opposed to a fixed camera with controlled settings). A typical pipeline would involve steps to:

Load and pre-process the image
Detect plate in the image
Image segmentation
Character recognition

A model is trained and published using customvision.ai to detect the number plate object in the image. OpenCV is then used for image processing steps and Read API in Azure Cognitive Services is used for its OCR functionality.

Object Detection using CustomVision.ai

To get started, head over to https://customvision.ai, sign in and create a new project.

Custom vision allows you to create two types of models:

Image Classification: to tag the entire image
Object Detection: to find and tag content within an image

Create an object detection model for this example.

Upload and label training images

Once the project is created, upload sample images. In this case, I uploaded 20 pictures of Sydney taxis that I got using image search on the internet.

After loading the training images, you will need to go through them and label the number plates.

The screenshot above shows a training image, with a manually labelled number plate in the image.

Train model

Start the training process once all training images are labelled. The first time training is kicked off; it could take a while to finish. For this example, I used quick training.

Once training is complete, the tool will present the performance metrics of the trained model. It is essential to observe the precision and recall of the trained model to determine the relevance and quality of the model to the application requirements. Additional training may be required if these results are not satisfactory.

Last, by using the Quick Test function, you could validate the model against a holdout data set. This tool is also useful to understand the type of response to expect from the prediction API.

Publish trained model

Once a trained model is ready and is providing adequate results, the model will need to be published as a REST API to be consumed by downstream applications.

Publishing generates the URL and authentication key for the model.

The code

A custom vision model was trained to detect a number plate in an image, and the predicition API for the trained model was published to be used in this example. This Jupyter notebook (link) on GitHub illustrates how to put together the object detection and then OCR to detect and recognise the text in a number plate.

The following sections explain the different parts of the code.

APIs

APIs for custom vision and computer vision is used in this example. The secrets.ini file in the code example should include the access keys and URLs for the published customvision.ai prediction API (from the previous section) and API to Azure Cognitive Services instance.

The secrets file is formatted as follows:

[custom_vision]
key = <your published prediction api key>
imgurl = <URL for custom vision api, should end with /image>

[computer_vision]
key = <your computer vision api key>
url = https://australiaeast.api.cognitive.microsoft.com/ 

The requests library in python is used to make calls to the APIs in the code example.

Custom Vision API Call

To make the call to custom vision API, the code first loads the image as a byte array, and using opencv library, the image is serialised into a cv2 image for diplaying and visualisation later on.

data = open('anpr_samples/bike.jpg', 'rb').read()
 # decode the image file as a cv2 image, 
 # useful for later to display results
img = cv2.imdecode(np.array(bytearray(data), dtype='uint8'), 
    cv2.IMREAD_COLOR)

The REST request must pass in the authentication key to the published custom vision model and the data array for the loaded image.

custom_vision_headers = {
    'Content-Type': 'application/octet-stream', 
    'Prediction-Key': custom_vision_key}
custom_vision_resp = requests.post(url=custom_vision_imgurl, 
    data=data, 
    headers=custom_vision_headers).json()

In the following section, the respose from the REST API call is examined to find the detected object. The code below is simplistic in its logic, as it only looks at the returned hit with the highest probability, a more complex logic maybe required to example more hits in the returned result, and not just the highest probablity hit.

 # inspect the top result, based on probability 
hit = pd.DataFrame(custom_vision_resp['predictions'])
    .sort_values(by='probability',ascending=False)
    .head(1)
    .to_dict()
print(hit)

Here we can see that a plate was detected with a probablity of 0.92, the bounding box element of the response represents the location of the object relative to the size of the image.

    {
        'probability': {30: 0.926827848}, 
        'tagId': {30: '1752966c-d1e1-46f5-924b-505d90a16981'}, 
        'tagName': {30: 'plate'}, 
        'boundingBox': {30: {
            'left': 0.1388305, 
            'top': 0.486726224, 
            'width': 0.09517826, 
            'height': 0.079084456
            }
        }
    }

The bounding box dimensions can be used to draw a box around the detected plate.

 # extract the bounding box for the detected number plate 
boundingbox = list(hit['boundingBox'].values())[0]
l,t,w,h = (boundingbox['left'], 
    boundingbox['top'], 
    boundingbox['width'], 
    boundingbox['height'])

 # extract bounding box coordinates and dimensions are scaled 
 # using image dimensions 
polylines1 = np.multiply([[l,t],[l+w,t],[l+w,t+h],[l,t+h]], 
    [img.shape[1],img.shape[0]])

 # draw polylines based on bounding box results
img2 = cv2.polylines(img, 
    np.int32([polylines1]), 
    1, (255,255,0), 4, 
    lineType=cv2.LINE_AA )

Now that the number plate is detected and located, a cropped image of the plate is saved to a new image object. The cropped plate image is then used for character recognition.

 # crop the image to the bounding box of the rego plate

crop_x = polylines1[:,0].astype('uint16')
crop_y = polylines1[:,1].astype('uint16')

img_crop = img2[np.min(crop_y):np.max(crop_y), 
    np.min(crop_x):np.max(crop_x)]

 # display the detected rego plate region
plt.imshow(cv2.cvtColor(img2, cv2.COLOR_BGR2RGB))

Text Recognition

In this section, the cropped image for the detected rego plate is sent to the text extraction API.

Extracting text requires two API calls: One call to submit the image for processing, the other to retrieve the text found in the image.

First the cropped image is convered to a byte array before it is send to the text recognition API.

crop_bytes =bytes(cv2.imencode('.jpg', img_crop)[1])
 # make a call to the text_recognition_url
response = requests.post(
    url=text_recognition_url, 
    data=crop_bytes, 
    headers={
        'Ocp-Apim-Subscription-Key': subscription_key, 
        'Content-Type': 'application/octet-stream'})

The response of the text recognition API call holds the callback URL to retrieve the extracted text. This URL is saved, and then the API is polled for results.

 # Holds the URI used to retrieve the recognized text.
response.raise_for_status()
operation_url = response.headers["Operation-Location"]

 # The recognized text isn't immediately available, so poll to wait for completion.
analysis = {}
poll = True
while (poll):
    response_final = requests.get(
        response.headers["Operation-Location"], 
        headers={'Ocp-Apim-Subscription-Key': subscription_key})
    analysis = response_final.json()
    print(analysis)
    time.sleep(1)
    if ("recognitionResults" in analysis):
        poll = False
    if ("status" in analysis and analysis['status'] == 'Failed'):
        poll = False

A successful recognition result object includes the detected lines and text/words within these lines.

    {
        'status': 'Succeeded', 
        'recognitionResults': [
            {
                . . . 
                'lines': [
                    {
                        'boundingBox': [******], 
                        'text': 'EDY.60', 
                        . . .
                }]
        }

All is left is to extract the text from the response object.

for i,l in enumerate(analysis['recognitionResults'][0]['lines']): 
    print(i, ': text found: ', [w['text'] for w in l['words']])

0 : text found:  ['EDY.60']

Summary

The approach for number plate recognition illustrated in this post is simple yet very effective. No particular knowledge of neural network techniques is required to build a solution with this approach, as it relies on the use of cognitive services and APIs.

Few options can be considered to improve the results further:

Supplying additional training images in different orientations to enhance the output of object detection
Post-processing detected objects and examining all returned results of the object detection step
Training a custom OCR model to improve the text recognition outcomes

Create a pandas DataFrame from a dict

2020-06-10T00:00:00+10:00

It is a simple task to turn your dictionary into a pandas dataframe, however, everytime I need to do that I find my self going back to docos and stackoverflow to remember how! So I decided to write a short instructions page for future reference.

dictionaries

Often data comes to you in a form of a dictionary object. There are types of dictionaries based on orientation that I deal with in this post:

Type 1: dictionary item holds value pairs to be converted to rows, example dict1 = { 'a': 1, 'b': 2, 'c': 3}
Type 2: dictionary items are columns, example dict2 = {'name': ['a', 'b', 'c'], 'value' = [1, 2, 3]}

type 1: items are rows

In the first case, I want to convert each item, key-value pair, in the dictionary into a row in the dataframe. Use the dictionary items() method to get a list of key-value pairs in tuples, specify column names when creating the dataframe.

Type 1 dictionaries tend to be long, and are usually faced when interacting with libraries, like for example getting the properties of nodes in a graph, with each key being the node name, the value is the value of property in question.

import pandas as pd 
dict1 = { 'a': 1, 'b': 2, 'c': 3}
df1 = pd.DataFrame(dict1.items(), columns=['name', 'value'])

print(df1)

output:

  name  value
  a      1
  b      2
  c      3

type 2: items are columms

This type has a columnar structure, the dictionary in this case is usually not long (depending on number of columns). Dictionary values store arrays that map to column values, with the dictionary key mapping to the column name in the dataframe.

In this case you just need to pass in the dictionary object as is when creating the dataframe, there is no need to specify the column names as they are already specified by the keys of the dictionary.

import pandas as pd 
dict2 = { 'name': ['a', 'b', 'c'], 'value': [1 ,2 ,3]}
df2 = pd.DataFrame(dict2)

print(df2)

output:

  name  value
  a      1
  b      2
  c      3

- mutaz .

Visualisation Tools!

2019-11-30T00:00:00+11:00

In this series, I attempted the same visualisation task in three different tools: PowerBI, Tableau, and R, to compare the methods and approach for visualising the same dataset and producing the same/similar results.

PowerBI: Using Power BI to explore the NSW Penalty Notices data set
Tableau: First Time Tableau User!
R language: Using R markdown and ggplots for data visualisation

In all three examples, I started with data preparation tasks, then getting some initial insights through summarisation to explore min, max and other summary stats such as the median. Then I used stack bar charts to compare values in different categories. I explored different ways to control the style and formatting of such plots.

Reusability in PowerBI and Tableau is done through embedding, copying and pasting visualisation elements. In R, one can create functions to encapsulate the code for plots.

We use visualisations to communicate insights, in this series, applying a forecasting model is one way to show predicted increase of fines revenue overtime. Both PowerBI and Tableau have built-in features to produce time-series forecasts, in R the user must first fit a forecasting model, use it to produce a forecast data set then visualise the results.

Both PowerBI and Tableau provide online services to publish the developed reports and dashboards for sharing with others. Rmarkdown can be compiled into a document and shared with others offline, as a PDF or Word document, or online through services such as RPubs.

All of the above and more are explored in more detail in the three blog posts.

In summary, I’d pick PowerBI over any other tool to produce data narratives for communicating insights with impact, Tableau is an equally capable tool. So between these two tools, it often comes down to personal preference and availability of the tool within your organisation. Both tools are suitable self-serve BI tools for business users and data scientists.

For complex analysis of data, exploratory data analysis, predictive and forecast modeling, I would pick R language with ggplot, or python with libraries such as matplotlib and seaborn (which I did not cover in this series) as the tools of choice.

My personal preference remains PowerBI though! what’s yours?

first_post <- ‘Hello World!’

2019-11-01T00:00:00+11:00

Shortly after Microsoft announced the availability of its Cognitive Services APIs in 2015 I started experimenting with services such as language understanding, computer vision, voice services. All have pre-built models and available through Web APIs to use in app development.

This picture was done using the Face APIs, try it here: Face Cognitive API

Then enter the bot framework; a framework to build conversational user experiences. I started creating bot applications that use cognitive services and integrated into the bot language understanding to detect user intent. I added advanced search capabilities. The result was a new intelligent user experience that allowed users to interact with the application through natural language, and converse with the bot using voice.

This was my introduction to the world of AI. I also started to deal with big data through my work with customers and learned a thing or two about machine learning. Now I am hooked, and I started dabbling with some statistical analytics, learned how to look at numbers differently, and the next thing I know, I enrolled to study data science.

I started my Masters in Data Science and Innovation (MDSI) at UTS in Feb 2018, and since then I completed different projects covering the different aspects of data science:

Data acquisition and cleansing
Data wrangling using packages such as dplyr in R and pandas in python
Applied different predictive analytics techniques, such as linear regression, clustering, classification on data
Applied time-series forecasting on real energy data set: data acquisition of weather observations, short-term load forecast of energy demand
I learned how to use plotting libraries, mainly ggplot2 and matplotlib to generate graphs
Delved deeper into data visualisations and practiced creating narratives from data using visualisations, for example In this post I took a deeper look a the NSW traffic offenses and penalties to discover some unexpected insights!
Researched different deep learning approaches and built my first neural network from scratch.
Built an image captioning model (and this)that combines concepts from convolutional neural networks (CNN), transfer learning and long short term memory (LSTM) networks.

The latest addition to my learning and data science portfolio is the work I did as part of the Innovation Lab subject during the Spring 2019 term. In this subject, I worked on computational materials science research, applying different skills I acquired in data wrangling and visualisation to analyse computational materials science results. More to come on this topic in a later post!

I am setting up this site to bring my blogs under one platform, here I am using github pages and jekyll.

I will use this platform to reflect on my learning journey and share public notebooks of ML/AI problems and projects I work on. I hope you enjoy it!

Mutaz

Green energy trends in Australia

2019-05-19T00:00:00+10:00

In my previous post I looked at electricity demand and generation in Australia, and where electricity supply comes from by comparing the different energy sources used for power generation, and especially the use of non-renewable and green fuels to produce electricity.

In this post, I explore the use of renewable energy sources to understand how renewable fuels are contributing to the growth of energy supply in Australia.

Towards cleaner energy in Australia

2019-05-17T00:00:00+10:00

Australia is a growing nation, its demand of electricity grew 10 fold in the last 60 years. Electricity demand in Australia reached 258,000 gigawatt hours (GWh) in 2018, compared to 23,000 GWh in 1960.

Reforming NSW road fines system!

2019-05-04T00:00:00+10:00

As a resident and a road user in NSW, I have a general knowledge of road rules and penalties based on my own experience! All of us are affected by government policies and practices to monitor road usage and issue fines.

The road rules set a framework for safe and efficient movement of traffic on NSW roads. It is understandable that road fines and penalties are meant to deter and detect dangerous driving to protect persons and properties. But how far has the government of NSW gone in issuing penalty notices and collecting revenue from road-related offences? And is it all justified?

Using R markdown and ggplots for data visualisation

2019-04-14T00:00:00+10:00

R Markdown is a file format for creating dynamic documents with R by writing in markdown language. The document contains chunks of embedded R code and content blocks. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents that can be used to document and share the results of data processing and analysis including visualisations with others.

ggplot is a data visualization package for the statistical programming language R. It implements a Grammar of Graphics as a scheme for data visualization which breaks up graphs into semantic components such as scales and layers.

In this blog post I will be exploring the use of R Markdown with ggplot to produce visualisations and communicate data insights.

Getting Started

To get started with R Markdown (rmd), the user must create an R Markdown, or a Notebook file. Typically this will be done in an R integrated development environment such as RStudio, a tool that most data scientists are familiar with.

Once you have created the rmd file, you now ready to start writing code to perform the usual data preparation and exploration activities.

All the code used to prepare for this blog post is published as an R Markdown on rpubs at this link: http://rpubs.com/mutazag/NSW-Road-Offences-and-Penalties. The process I followed is summarised in the table of content for this R Markdwon file.

The Data Set

The data set used for this post is the same NSW Roads Offences and Penalties data that I used in my previous posts for exploring the use of PowerBI and Tableau for building data visualisations and stories.

I am using the tidyverse package for loading and manipulating the data set. Some basic data preparation was required to correct the data type for the OFFENCE_MONTH field, and to select a subset of fields that are required for this analysis.

# load the data set while setting the timezone locale to "Australia/Sydney" 
dfraw <- read_csv(file="dfosf001.csv", 
               locale = locale(tz ="Australia/Sydney"))

# examine the data set and the data specs
dfraw %>% head()
dfraw %>% spec()


# changing the data type for OFFENCE_MONTH to date
dfraw$OFFENCE_MONTH <- dfraw$OFFENCE_MONTH %>% dmy()

# selecting a sub set of columns 
df <- dfraw %>% 
  select(OFFENCE_FINYEAR, OFFENCE_MONTH, OFFENCE_CODE, OFFENCE_DESC, FACE_VALUE, TOTAL_NUMBER, TOTAL_VALUE)

Producing summaries of the data set is another data manipulation task that are commonly performed at this stage of the analysis.

In the example below, a summary data set is created to inspect the data range of the offences, total number and value of offences, minimum and maximum offence face value, and total number of records in this data set.

Data Exploration

This basic summary is already producing some insights: the minimum offence face value is $20, whilst the maximum offence face value is a whopping $18,455.

At this stage, the user may want to find out what other offences attract large fines. To produce such result in R, the user must query the data, produce a data frame with the required results and then produce a visualisation that meets their requirements, such as the stacked bar chart below.

The above visualisation is based on a bar chart plot. To produce the aesthetics in this plot, I applied a theme setting to control the borders, grid lines and general style elements, specify the text labels for axes and flip the plot coordinate. These tasks are achieved using the code below.

df %>%   group_by(OFFENCE_DESC) %>% 
  summarise(Value = first(FACE_VALUE)) %>% 
  top_n(5, Value  ) %>% 

  # Specify x and y axes 
  ggplot(aes(x = reorder(OFFENCE_DESC,Value), y = Value)) +

  # Choose a bar chart
  geom_bar(stat ="identity") + #geom_label(aes(label=Value)) +

  # apply a theme 
  theme_light() + 

  # set x axis label 
  labs (x = "Offence Description") +

  # flip coordinates (x axis becomes y axis)
  coord_flip()

The user often needs to continue transforming the data set to make and suitable for producing the different required visuasliations. For example, performing aggregate summaries on numerical fields (such as the Total Value of offence) by financial year or Offence Description first, then calling the appropriate plotting functions to display the results in a visualisation

Reproduce and Reuse Plotting Code

One of the key benefits of using a coded approach to generating visualisations is the ability to reproduce and reuse the code with different data sets.

In this example, the R function below de-clutters a plot by setting the theme for the plot then applying additional aesthetic to remove panel grid.

# decluttering theme 
theme_dvn <- function() {
  retTheme = theme_minimal() + 
    theme(panel.grid = element_blank())
  
  return(retTheme)
}

Some simple formatting tasks require extensive code, for example, formatting numbers to display suffixes instead of the whole number. The code below is used to achieve this task. This function is good example of code re-usability, as I did not have to write it my self, instead I was able to reuse the work of others that already solved this problem.

# numbers metric formatter 
format_si <- function(...) {
  # Format a vector of numeric values according
  # to the International System of Units.
  # http://en.wikipedia.org/wiki/SI_prefix
  #
  # Based on code by Ben Tupper
  # https://stat.ethz.ch/pipermail/r-help/2012-January/299804.html
  # Args:
  #   ...: Args passed to format()
  #
  # Returns:
  #   A function to format a vector of strings using
  #   SI prefix notation
  #
  
  function(x) {
    limits <- c(1e-24, 1e-21, 1e-18, 1e-15, 1e-12,
                1e-9,  1e-6,  1e-3,  1e0,   1e3,
                1e6,   1e9,   1e12,  1e15,  1e18,
                1e21,  1e24)
    prefix <- c("y",   "z",   "a",   "f",   "p",
                "n",   "µ",   "m",   " ",   "k",
                "M",   "G",   "T",   "P",   "E",
                "Z",   "Y")
    
    # Vector with array indices according to position in intervals
    i <- findInterval(abs(x), limits)
    
    # Set prefix to " " for very small values < 1e-24
    i <- ifelse(i==0, which(limits == 1e0), i)
    
    paste(format(round(x/limits[i], 1),
                 trim=TRUE, scientific=FALSE, ...),
          prefix[i])
  }
}

The entire plotting process can be wrapped in a function, and the function can be designed to accept a dataframe object, along with any other parameters, then generate the plot according to logic in the code. The following function produces a bar chart after applying the de-cluttering theme and number formatting functions prepared earlier.

# plotting function
topPlot <- function(df, v){
  
  df %>% group_by(OFFENCE_FINYEAR) %>% 
    summarise(Value = sum(TOTAL_VALUE), Number = sum(TOTAL_NUMBER)) %>% 
    ggplot(aes(x=OFFENCE_FINYEAR, y=!!as.name(v))) + 
    geom_bar(stat = "identity") + 
    labs( y = v, 
          x = "Financial Year", 
          title = paste0(v, " of offences by Financial Year")) +
    scale_y_continuous(label=format_si()) +
    theme_dvn()-> returnPlot
  
  return(returnPlot)
}

Generating different bars for different parameters becomes as easy as invoking the function by passing in the dataframe for plotting.

df %>% topPlot("Number")

df %>% topPlot("Value")

The above code generated the following two plots.

More complex scenarios can be achieved with high levels of code re-usability as shown in the example below. The data frame was prepared to calculate top parking offences for each year separately, the loop in the code applied the plotting settings on each of the sub sets to create a list of plots. At the end of the process the plots are displayed for each year by calling the print function on the list of plots.

# aggregates data set by year 

df %>% mutate(Year = year(OFFENCE_MONTH)) %>% 
  group_by(Year, OFFENCE_DESC) %>%
  summarise(Value = sum(TOTAL_VALUE)) %>% 
  #tally (TOTAL_VALUE) %>% 
  top_n(10, Value) %>% 
  ungroup() -> df2


# loop to create a list of plots by year
pl <- c()

for (y in 2013:2019){
  df2 %>% 
    filter(Year == y) %>% 
    ggplot(aes(x = reorder(OFFENCE_DESC,Value), y = Value)) +
    facet_grid(cols = vars( Year)) +
    geom_bar(stat ="identity") + #geom_label(aes(label=Value)) +
    theme_dvn() + 
    labs( x = "Offence Type")+
    scale_y_continuous(label=format_si())+
    coord_flip() -> p
  
  pl[[toString(y)]] = p
}

# display the list of plots 
print(pl)

The output of this code is a series of visualisations, similar to the one shown below, one for each year from 2013 to 2019.

Insights from Visualisations

So far, we’ve seen how R and ggplot can be used for data exploration activities. R provides a great platform for machine learning and advanced data analytics.

In this example, I am using the platform to forecast the number of offences and revenue. The example demonstrates a very basic time series forecasting process using the forecast library in R.

First, I needed to prepare the data in a time series object format.

df %>% group_by(OFFENCE_MONTH) %>% 
  summarise(Value = sum(TOTAL_VALUE), 
            Number = sum(TOTAL_NUMBER)) %>% 
  ungroup() %>% 
  select(Value, Number) -> tsdf
ts1 <- ts(tsdf, 
          start = c(year(dfSummary$`From Date`), month(dfSummary$`From Date`)), 
          frequency = 12) %>% head(-1)

# time series plotting function is prepared for re-usability 
plot_ts <- function (obj, endyear = 2019){
 return(
   autoplot(obj) + theme_dvn() + scale_y_continuous(label = format_si()) + scale_x_continuous(breaks = 2013:endyear)
 )
}

plot_ts(ts1[,1]) + labs(y="Value [AUD]")

The plot shows the change in revenue overtime. All the plot formatting functions that were prepared earlier are used to produce this visualisation.

Then using the forecast library, I am training an ARIMA model and forecasting for the next 24 months.

# Train the model
revenueFit <- auto.arima(ts1[,1])

# Forecast
revenueForecast <- forecast(revenueFit, h =2*12)

# Plot the output 
plot_ts(revenueForecast, endyear = 2021) + labs(y="Value [AUD]", title="Revenue Forecast")

The output in this case shows historical and foretasted revenue. The shaded areas represent the forecast confidence intervals.

It is important to note that the forecasting performed in this example is for explanatory purposes only, and is only meant to demonstrate the ability to forecast with this data set.

Producing meaningful forecasts requires deeper analysis of the timeseries data set and its decomposition, and proper validation of forecast errors and accuracy on a test data set before accepting any of the forecasting results.

Publishing the Output

RMD allows you to control the theme used to format the output document and whether to include a table of content or not in the output.

The user could achieve this by including document output attribute in the header of the rmd file show in the code snippet below.

---
title: "NSW Road Offences and Penalties"
output: 
  html_document: 
    fig_caption: yes
    theme: lumen
    toc: yes
---

The knitr library in RStudio provides the feature to publish the output as an html page, or other document formats such as Microsoft Word.

The ‘knitted’ output can then be published to the web for sharing with others on platforms such as Rpubs, as shown in the published document for this exercise on this link.

Conclusion

This post demonstrated some of the possibilities for using a coded approach to generate data visualizations with R and ggplot. It walked through the process to prepare the data set for specific plotting requirements, then calling the plotting functions and customising the elements of the plot.

Control of output, reproducibility of results and reusable code are some of the key benefits of this approach, but it requires technical knowledge of data preparation and plotting libraries, and coding experience. It is mostly an approach to use data scientists or developers. With careful design, developers can write modular code to minimise code duplication.

It is hard to compare this approach to other business intelligence tools, such as Power BI and Tableau, as the purpose and target user base for R, ggplot and RMarkdown is not necessarily the typical business user or data analyst. This approach is very flexible and great for complex data preparation and analysis, something that data scientists need in their tool set.