A few weeks ago I did an interiview with Nagaraj Venkatesan – a good friend of mine who works for Microsoft Singapore.

You can watch video below – I talk about all things DevOps – where it relates to data, Microsoft Fabric and my thoughts on the impact of AI:
https://www.youtube.com/watch?v=jBseLdRH1FU

I hope you enjoy it as much as I did doing it.

Yip.

I love KQL so much I even made a video and if you compare it to my last blog post – yeah my hair has changed a bit…
And my beard.
And my clothing…

Check out Data Exposed here – it is an AWESOME site:

https://learn.microsoft.com/en-us/shows/data-exposed/

If you want to see a longer version of a session on KQL then do visit my last blog post here

I recently did a talk to the Data Platform Down Under User group and it covers a whole heap of stuff that I’ve written about in this series.

Check it out below:

I was talking to an ex-client earlier this week and he saw the drafts of my blog posts around DevOps infra-as-code spinning up Azure Data Explorer and he said – but what about ARM templates…..

Now this is a trigger. I do not like ARM templates.
But…

Some things pay the mortgage and back when we worked together I spun up their whole platform using ARM (this was before bicep was a thing).
So here it is for you Scott… you lover of ARM (and a good red wine).

Here are the steps to provision an Azure Data Explorer cluster using an ARM template.

Prerequisites:

• An Azure subscription
• Basic knowledge of Azure Resource Manager templates
• Being a lover of JSON (just kidding – I have to write this…)

Step 1: Create the ARM Template The first step is to create the ARM template. The template defines the resources that will be provisioned in Azure. We’ll create a simple template that provisions an Azure Data Explorer cluster.

Here’s an example ARM template that provisions an Azure Data Explorer cluster:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Kusto/clusters",
      "apiVersion": "2021-04-01-preview",
      "name": "[variables('adxClusterName')]",
      "location": "[variables('location')]",
      "sku": {
        "name": "Standard_L16s",
        "tier": "Standard"
      },
      "properties": {
        "enableDiskEncryption": false,
        "dataRetentionTime": "365.00:00:00",
        "trustedExternalTenants": []
      }
    }
  ],
  "variables": {
    "adxClusterName": "adx-cluster",
    "location": "[resourceGroup().location]"
  }
}

This template provisions an Azure Data Explorer cluster with the name “adx-cluster” and the SKU “Standard_L16s” in the same location as the resource group.

Step 2: Deploy the ARM Template
To deploy the ARM template, we’ll use the Azure Portal. Navigate to the resource group where you want to provision the Azure Data Explorer cluster and click on “Deploy a custom template”.

Select the “Build your own template in the editor” option and paste the ARM template code into the editor. Click “Save” and then “Review + Create”.

Review the template parameters and click “Create” to deploy the Azure Data Explorer cluster.

Step 3: Verify the Deployment
After the deployment is complete, you can verify that the Azure Data Explorer cluster was provisioned successfully. You can check the status of the cluster in the Azure Portal or use the Azure CLI to run the following command:

az kusto cluster show --name adx-cluster --resource-group MyResourceGroup

This command will return information about the Azure Data Explorer cluster, including the status and SKU.

So there you have it – yes ARM templates are “relatively” easy, but they can be horrid..
Use bicep as a first go to or my favourite – terraform.

Either way – just use infrastructure as code to do any provisioning – OK??!!

(Also Scott – when I am in Seattle next – you own me a lot of red wine)

Yip.

In late March 2021 Azure Data Explorer Engine v3 was release and it is the latest version of the engine that powers Azure Data Explorer. It is designed to provide a number of key features and benefits for data analytics, including:

1. Scalability: The Azure Data Explorer Engine v3 is designed to be highly scalable, allowing you to process and analyze large amounts of data quickly and efficiently. The engine can be scaled up or down as needed to handle varying data volumes and workloads.
   
2. Performance: The Azure Data Explorer Engine v3 is highly optimized for performance, allowing you to analyze data in near real-time. The engine can perform complex queries and aggregations on large data sets quickly and efficiently.
   
3. Flexibility: The Azure Data Explorer Engine v3 is designed to be highly flexible, allowing you to work with a wide range of data types and formats. The engine can handle structured, semi-structured, and unstructured data, as well as data in a variety of formats, including CSV, JSON, and more.
   
4. Reliability: The Azure Data Explorer Engine v3 is designed to be highly reliable, ensuring that your data is always available and secure. The engine is built on top of Azure, providing built-in redundancy and failover capabilities to ensure high availability.

Key features of the Azure Data Explorer Engine v3 The Azure Data Explorer Engine v3 provides a number of key features that make it a powerful and flexible platform for data analytics, including:

1. Columnstore Indexes: The Azure Data Explorer Engine v3 uses columnstore indexes to store and query data. Columnstore indexes provide high compression rates and fast query performance, allowing you to quickly analyze large amounts of data.
   
2. Time Series: The Azure Data Explorer Engine v3 includes built-in support for time series data, allowing you to analyze time-based data quickly and efficiently. The engine can handle large volumes of time-series data and provide real-time insights into trends and patterns.
   
3. Query Language: The Azure Data Explorer Engine v3 supports a powerful query language called Kusto Query Language (KQL). KQL provides a simple and intuitive syntax for querying data, allowing you to quickly and easily analyze large data sets. But you know this from these blog posts right?
   
4. Security: The Azure Data Explorer Engine v3 provides built-in security features, including role-based access control (RBAC) and encryption, to ensure that your data is always secure.

You can read more about V3 of the engine here:
https://learn.microsoft.com/en-us/azure/data-explorer/engine-v3

If you have read any of my last few posts on provisioning Azure Data Explorer then you will probably be wondering…..

Will he write about Powershell?

Ok, I will.

Step 1: Install and configure Azure PowerShell To use PowerShell to provision Azure Data Explorer, you first need to install the Azure PowerShell module. You can install it using the following command:

Install-Module -Name Az -AllowClobber -Scope CurrentUser

Once the module is installed, you need to log in to Azure using the Connect-AzAccount command. This command will prompt you to enter your Azure credentials to authenticate with the Azure portal.

Step 2: Create an Azure Data Explorer cluster To create an ADX cluster, you can use the New-AzKustoCluster cmdlet.

Here’s an example command that creates an ADX cluster named “myadxcluster” in the “East US” region with a “D13_v2” SKU and two nodes:

New-AzKustoCluster -ResourceGroupName myResourceGroup -Name myadxcluster -Location EastUS -SkuName D13_v2 -Capacity 2

This command will create an ADX cluster with the specified name, location, SKU, and node capacity. You can customize these settings to fit your needs.

Step 3: Create an Azure Data Explorer database After creating an ADX cluster, you can create a database within the cluster using the New-AzKustoDatabase cmdlet. Here’s an example command that creates a database named “myadxdatabase” within the “myadxcluster” cluster:

New-AzKustoDatabase -ClusterName myadxcluster -Name myadxdatabase

This command will create a new database with the specified name within the ADX cluster.

Step 4: Configure data ingestion Once you have created a database, you can configure data ingestion using the Azure Data Explorer management portal or PowerShell. To use PowerShell, you can create a data ingestion rule using the New-AzKustoDataIngestionMapping cmdlet.

Here’s an example command that creates a data ingestion rule for a CSV file – we will use a file that Microsoft has provided for learning KQL and also Azure Data Explorer ( https://kustosamples.blob.core.windows.net/samplefiles/StormEvents.csv ):

$mapping = @"
col1:string
col2:int
col3:datetime
"@

New-AzKustoDataIngestionMapping -ClusterName myadxcluster -DatabaseName myadxdatabase -Name mydataingestionrule -Format csv -IgnoreFirstRecord $true -FlushImmediately $true -Mapping $mapping -DataSource @"
https://kustosamples.blob.core.windows.net/samplefiles/StormEvents.csv
"@

This command will create a data ingestion rule named “mydataingestionrule” for a CSV file named “StormEvents.csv” within the specified ADX cluster and database. The data ingestion rule specifies the file format, data mapping, and ingestion behavior.

Step 5: Verify your deployment.
Once you have completed the above steps, you can verify your Azure Data Explorer deployment by running queries and analyzing data in the ADX cluster. You can use tools like Azure Data Studio, which provides a graphical user interface for querying and analyzing data in ADX.

Using Powershell is fairly easy – personally I prefer terraform – but that’s only because I am a fully cloud person on all the clouds…

But that could be another blog post

Yip.

I can’t really write about provisioning anything in Azure without mentioning Azure CLI.
My last two posts were about using

terraform and bicep

Here we will be using the Azure CLI.

Step 1: Install and configure the Azure CLI To use the Azure CLI, you first need to install it on your local machine. The Azure CLI can be installed on Windows, macOS, or Linux, and detailed instructions can be found in the Azure documentation.

Once the Azure CLI is installed, you need to log in using the az login command. This command will prompt you to enter your Azure credentials to authenticate with the Azure portal.

Step 2: Create an Azure Data Explorer cluster To create an ADX cluster, you can use the az kusto cluster create command. Here’s an example command that creates an ADX cluster named “myadxcluster” in the “East US” region with a “D13_v2” SKU and two nodes:

az kusto cluster create --name myadxcluster --location eastus --sku D13_v2 --capacity 2

This command will create an ADX cluster with the specified name, location, SKU, and node capacity. You can customize these settings to fit your needs.

Step 3: Create an Azure Data Explorer database After creating an ADX cluster, you can create a database within the cluster using the az kusto database create command.

Here’s an example command that creates a database named “myadxdatabase” within the “myadxcluster” cluster:

az kusto database create –cluster-name myadxcluster –name myadxdatabase

This command will create a new database with the specified name within the ADX cluster.

Step 4: Configure data ingestion Once you have created a database, you can configure data ingestion using the Azure Data Explorer management portal or the Azure CLI. To use the Azure CLI, you can create a data ingestion rule using the az kusto data-ingestion-rule create command.

Here’s an example command that creates a data ingestion rule for a CSV file:

az kusto data-ingestion-rule create --cluster-name myadxcluster --database-name myadxdatabase --name mydataingestionrule --data-source @'./mydata.csv' --format csv --ignore-first-record --flush-immediately --mapping 'col1:string,col2:int,col3:datetime'

This command will create a data ingestion rule named “mydataingestionrule” for a CSV file named “mydata.csv” within the specified ADX cluster and database. The data ingestion rule specifies the file format, data mapping, and ingestion behavior.

Step 5: Verify your deployment Once you have completed the above steps, you can verify your Azure Data Explorer deployment by running queries and analyzing data in the ADX cluster. You can use tools like Azure Data Studio, which provides a graphical user interface for querying and analyzing data in ADX.

Provisioning Azure Data Explorer using the Azure CLI is a pretty simple and straightforward process.

Yip.

So in my last post I wrote about how to provision Azure Data Explorer using terraform .

In this post I will use bicep to provision Azure Data Explorer.

Steps:

Step 1: Set up your environment Before you can begin using Bicep to provision Azure Data Explorer, you need to set up your environment. This involves installing the Azure CLI and Bicep. You’ll also need to create an Azure account and set up authentication.

Step 2: Define your infrastructure as code Once you have your environment set up, you can begin defining your infrastructure as code using Bicep. This involves writing code that defines the resources you want to provision, such as Azure Data Explorer clusters, databases, and data ingestion rules.

Here’s an example of a Bicep file that provisions an Azure Data Explorer cluster and database:

param resourceGroupName string
param location string
param clusterName string
param capacity int

resource cluster 'Microsoft.Kusto/clusters@2022-01-01-preview' = {
  name: clusterName
  location: location
  sku: {
    name: 'D13_v2'
    capacity: capacity
  }
}

resource db 'Microsoft.Kusto/clusters/databases@2022-01-01-preview' = {
  name: 'my-kusto-database'
  parent: cluster
  dependsOn: [cluster]
}

In this code, we declare four parameters: the resource group name, the location, the cluster name, and the cluster capacity. We then define an Azure Data Explorer cluster using the Microsoft.Kusto/clusters resource, specifying the name, location, SKU, and capacity. Finally, we define a database using the Microsoft.Kusto/clusters/databases resource, specifying the name and the parent cluster.

Step 3: Deploy your infrastructure. Now that you have defined your infrastructure as code, you can deploy it using the Azure CLI. First, run the az login command to authenticate with Azure. Then, run the following commands to create a new resource group, build the Bicep file, and deploy the infrastructure:

az group create --name my-resource-group --location westus2
az deployment group create --resource-group my-resource-group --template-file main.bicep --parameters resourceGroupName=my-resource-group location=westus2 clusterName=my-kusto-cluster capacity=2

This will create a new resource group, build the Bicep file, and deploy the Azure Data Explorer cluster and database.

Step 4: Test and monitor your deployment Once your infrastructure is deployed, you should test and monitor it to ensure it is working as expected. This may involve running queries on your Azure Data Explorer cluster, monitoring data ingestion rates, and analyzing performance metrics.

Using Bicep to provision Azure Data Explorer offers many benefits, including faster deployment times, greater reliability, and improved scalability. By automating the provisioning process, you can focus on more important tasks, such as analyzing your data and gaining insights into your business.

Bicep is a powerful tool that can simplify the process of provisioning Azure Data Explorer. By following the steps outlined in this blog post, you can quickly and easily set up a Azure Data Explorer.

Yip.