Azure Platform Series – renaming your YAML pipeline in Azure DevOps

This blog post is about YAML pipelines in Azure DevOps.

I had a repo called InfrastructureAsCode for a client. I have been transitioning them to use YAML for their build pipelines in Azure DevOps – because now we can source control the build itself.

If you do not know what a YAML pipeline is in Azure DevOps check it out here:

https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema

You can try this for yourself by using the awesome AzureDevOpsLabs:

https://azuredevopslabs.com/labs/azuredevops/yaml/

I had created a pipeline using YAML – which was called InfrastructureAsCode as the YAMP file was in the root directory.

However I wanted to move it into a folder .\InfrastructureAsCode\pipelines\… and run the YAML file from there – as I would have a non-prod and PROD version of them (as the schedule was different for each).

That went well – but I didn’t like the name InfrastructureAsCode as my pipeline name. And it got worse when I added the PROD version of the YAML file – I now had InfrastuctureAsCode(1).
That’s even uglier.

So I needed to rename the pipeline – but how do you do this with YAML??

Well – click your pipeline and edit it and then up in the right hand corner is the 3 dots – click that and choose Triggers:

Click on YAML tab:

And now you have renamed your pipeline!!

Yip.

Azure Platform Series: cloudshell code – your (brilliant) online editor

This post is about code which is an online editor you can use with cloudshell.

I live in the Azure platform all day (almost) every day. What this means is I need to be able to run scripts from just about any device – there are times where I will be scaling a Hyperscale database at 10:30pm from my phone….

So I use cloudshell a LOT in Azure. I wrote a post about setting up cloudshell here:
Azure Platform Series: How to setup Azure Cloud Shell

The best part of cloudshell is the online editor – which you can access within cloudshell by clicking on the {} in the toolbar:

Or you can access it by typing code . in the commandline:

which brings up the online editor:

You can now browse the directory structure and click on a script that you might have written before. In the example below I have clicked on scalePREPROD.sh to edit it:

You can read more about the editor here:

https://azure.microsoft.com/en-us/blog/cloudshelleditor/

For me – it is a great way to edit and store those scripts I use most in Azure.

Yip.

Bash: ^^}: bad substitution – upgrade your Bash version (macOS)

This blog post is about a situation where you are capitalising an argument in bash and you get the following error:

bash: ${state^^}: bad substitution

In this example I was capitalising a variable called state to STATE. After some quick googling I found that the reason I am getting this error is not because of my bad coding – but because I might not be using a version of bash greater than 4.

When I checked the version I am using on my mac it was:

$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
Copyright (C) 2007 Free Software Foundation, Inc.

and running the same command in Visual Studio showed the same – that I was running 3.2.57 (??!!??)

Which seemed quite old – that copyright of 2007 gives it away. I am running Big Sur – version 11.1 of macOS and from some more googling it appears there is some history around Apple and Bash and GPL licenses and <yawn>…… here are some links http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/ and

And I realised that many macOS users probably don’t know is that they are using an outdated version of Bash shell.

How to upgrade Bash on macOS.

For macOS you should use homebrew to install things:

https://brew.sh/

$ brew install bash

To verify the installation, you can check that you now have two versions of Bash on your system:

$ which -a bash
/usr/local/bin/bash
/bin/bash

The first one is the new version, and the second one is the old version:

$ /usr/local/bin/bash --version
GNU bash, version 5.1.4(1)-release (x86_64-apple-darwin20.2.0)
Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ /bin/bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20.2.0)
Copyright (C) 2007 Free Software Foundation, Inc.

Since the directory of the new version (/usr/local/bin) comes by default before the directory of the old version (/bin) in the PATH variable, the version used when you just type bash is the new one:

$ bash –version
GNU bash, version 5.1.4(1)-release (x86_64-apple-darwin20.2.0)
Copyright (C) 2020 Free Software Foundation, Inc.

We now need to whitelist our shell as Unix systems restrict this, so to use the newly installed Bash shell as the default shell, it must be able to act as a login shell. So we will add it to the /etc/shells file. We will edit this file as the root user:

$ sudo vim /etc/shells

and enter in your admin password.

You will now be in vim editor:

Use the i key to go into INSERT mode, paste in /usr/local/bin/bash down the bottom then hit th ESC key and type

:wq!

using the vim editor

Which will write and save our changes.

Set Default Shell

Because /bin/bash is still set as the default shell – you will still be using version 3.2.57. To change this to the new shell, execute the following command:

$ chsh -s /usr/local/bin/bash

The default shell for your user is now set to the new version of Bash. If you close and reopen the terminal window, you should now be using the new version already:

 $ echo "$BASH_VERSION"
 5.1.4(1)-release 

The chsh command changes the default shell only for the user who executes the command. If you want to change the default shell for other users too, so run the following (which will ask for your admin password):

$ sudo chsh -s /usr/local/bin/bash

All done – right?
So I went back into Visual Code and ran the original script – but still had the issue!!

Updating Visual Code to use the new Bash version

I found you need to go to Preferences and change Settings:

In the search field type in bash

click on “Edit in settings.json” for Osx and enter in “/usr/local/bin/bash” as shown below

Exit the Terminal and a new bash session will have

echo "$BASH_VERSION"
5.1.4(1)-release

I ran my script and it worked.

Yip.

Azure Platform Series “You have exceeded the maximum amount of scale changes within the past hour”

This blog post relates to where you might be doing scale operations of your app services or VMs in Azure and get the following error after doing quite a few scale up and down operations.

The exact error I received was:

Failed to update configuration for 'Api'. {"Code":"Conflict","Message":"You have exceeded the maximum amount of scale changes within the past hour(56 changes and limit is 40). Please retry later.","Target":null,"Details":[{"Message":"You have exceeded the maximum amount of scale changes within the past hour(56 changes and limit is 40). Please retry later."},{"Code":"Conflict"},{"ErrorEntity":{"ExtendedCode":"03024","MessageTemplate":"You have exceeded the maximum amount of scale changes within the past hour({0} changes and limit is {1}). Please retry later.","Parameters":["56","40"],"Code":"Conflict","Message":"You have exceeded the maximum amount of scale changes within the past hour(56 changes and limit is 40). Please retry later."}}],"Innererror":null}

Where I was scaling my app service plan called Api and went past some arbitrary limit.

It is hard to find the exact limits within Azure for app scale operations but by a process of elimination I have determined:

SKULimit
Premium/PremiumV2/Isolated40
Standard20
Basic15

Unfortunately there isn’t much we can do here – you just have to wait until your hour is up….

Bash: “command not found” in IF statement

This blog post is about a situation where writing an IF THEN ELSE statement with variables fails with something like

./scalePREPROD.sh: line 26: [[1: command not found

This was in a script called scalePREPROD that scaled app services if they were not a certain value – so I am checking to see IF they are a value of 15 and if they are not then scale up to 15.

TL;DR

You need a space in the IF statement when variables are used.

The code looked something like this:

for i in "${appServiceArray[@]}"; do
appcapacity=<bash query to check app services>
if
[[$appcapacity -ne $scaleAPPWorkersUp]];
then
<stuff to do>
else
<echo something back that we're all good>
fi
done;

So in the example above – I wanted to see if for my array of 6 app services ${appServiceArray[@]} any were not equal to a value of $scaleAPPWorkersUp.

To fix it I just needed to put a space so:

[[$appcapacity -ne $scaleAPPWorkersUp]];

becomes

[[ $appcapacity -ne $scaleAPPWorkersUp ]];

Pretty easy fix.

Yip.

Open Letter to the PASS Membership,

When I was appointed to serve as a Director at Large for the PASS Organization a year ago, it was with great passion, excitement and enthusiasm. I had some great ideas about how I could help globalize, diversify and improve our wonderful community that I’ve been a member of since 2013.

The PASS Community has provided me with friends and the opportunity to connect, share and learn.  I wanted to serve our community but have realised sadly that I cannot. 

It is with a heavy heart that I am delivering this message today. On Monday 7th December 2020 (Pacific Time) I resigned from the PASS Board. 

I had considered resigning from the board back in July this year, but wanted to continue and try and serve our community by continuing on the board. I wanted to bring change – I campaigned on it when running for the board this year. Ultimately I could not bring the change that I feel we needed and so this is the reason I am stepping down. To let others bring what they feel our community needs.

I want to thank the community for the support they have given me. This was a very hard decision to make, but I’m a give it a 100% person and I tried.

#KiaKaha PASS

What the future of PASS needs – and how I can help

At the recent PASS Virtual Summit I and the other candidates took part in a Ask Me Anything (AMA) session – where the community posed us questions.

It was a great experience for me – of the 3 days it ran I had time to participate in 2 of them. Current board activities and client requirements didn’t allow me to take part in the final session.

It was on reflection of the AMA session that I thought some key things need to be looked at.

Namely:

— how the community can be engaged in the future of PASS

— how we can align as a global team

— what the event(s) on 2021 might look like

  1. Community Engagement
    The Board needs to look to the leaders within our community. We need their help – because to go forward we all need to be on the same page. We need to work cohesively to save the community platform that is PASS. This cannot be a top down approach. We need your help.
    I have worked in many situations where people have said “it can’t be done” or people are siloed in their approach. One of the reasons I got into DevOps was I realised early on that we needed a goal that could align disparate groups of people. It worked wonders in my corporate career and I’ve excelled in it whilst running a consultancy company.
    In short – we need to listen to our community, we need to work with that community and take the positive actions and measure the not-so positive actions. Measure them against the goal of PASS – a platform that helps the community learn, grow and connect.
  2. Alignment
    The alignment of a global team is related to the above action item. We need to work actively with our Regional Mentors – because they represent our regions. They should be actively working with their community leaders and actively telling us the board what works and doesn’t. We need this alignment, we need this communication and we need people who are passionate about making a difference for their community. I appreciate these are volunteer positions, that it can take a bit of personal time – however – time can be allocated if something is important enough and I found helping a community to be somewhere people felt safe, could learn and become a better person. I was recognised by the community in 2019 for being relentless when it came to helping others:

    https://www.pass.org/About-PASS/PASS-News/2019-passion-award-winner

    In my life I have a pragmatic approach to disparity – one of the my early mantras was “make stuff go” and when placed in situations where groups of people wouldn’t agree on action items – I’d measure our opinions on whether it made stuff go or not. It is a very simple thing to do, the hard part is actually listening and distilling down to the fundamental problems at hand. Not up-talking or complicating things for our own ego…. That is what is needed for both items 1 & 2 above. I can help here, I have a proven corporate record in doing that.
  3. Basically this is about running out of $$…
    We need to take a pragmatic approach to the survival of PASS – and events may be smaller and more regional in 2021 – purely because of the covid situation we are experiencing right now. In saying that the Virtual Summit that we’ve just had was better than I personally thought it would be. I’m interested to see what the number/percentages look like for attendee experience. Certainly compared to other events I’ve been on it was better, yes there are improvements to be made/had. There were aspects that I didn’t like – but we have a voice and constructive feedback is far better than just saying something is bad.
    It will be interesting to see what the balance sheet looks like when all things are done – I feel that some hard decisions have to be made to ensure the longevity of the platform known as PASS. Some of those decisions will involve how we engage with the community, how we manage events as well as what strategic decisions we can make around content delivery. Not easy at all, but my passion for our community means I’m ready to stand up and fight for what PASS really needs to be,

PASS has changed my life – dramatically. I used to be a mediocre DBA. I thought locally rather than globally and the personal growth I have experienced by meeting passionate mentors has made me a far better person. It has resulted in my own mentoring of people and also having a wider sense of community – not just a data platform community – but helping others:

https://techcommunity.microsoft.com/t5/humans-of-it-blog/guest-blog-how-mvps-can-be-champions-for-diversity-and-inclusion/ba-p/824013

This is why I want PASS to continue – things have to change and I want to be a positive participant of that journey.

Please vote here:

https://pass.simplyvoting.com/?user=pass&language=en&og=112896

My other blog post around why I am running is here:
Why I am running for the PASS Board in 2020

#Yip.

Why I am running for the PASS Board in 2020

This blog post is about why I am running for a 2 year term on the PASS Board. The PASS Community is one that I care deeply about, it is one that has helped me grow as a person and allowed me to contribute and help other people grow.

I served for one year on the board – it has been an insightful year in terms of how content and services are delivered, how decisions are made and what level of influence Directors have.

I am running to serve a 2 year term as we need to continue the great things that PASS does for the community but how it operates needs to change.

Speaking of change – 2020 has certainly been a year about change….

The past 8 months have upended so many things in society and this is evident with PASS as well. We’re about to go into our first ever Virtual Summit – and we’re currently running at a $1.6 million deficit in this fiscal financial year.

In the past the annual Summit has been the sole source of revenue for running the community – it was a flawed business model and this year has accentuated the flaws in that model.

Now is the time to have a positive change in terms of how PASS delivers content and services to the Community. We simply cannot continue on the path we have. This does not mean I want PASS to not exist, it does not mean I want to throw out the current management company.

What I want to do is introduce ways that we can work smarter – not harder…

The size and scale of what PASS is – requires management and operational oversight in how services and content is delivered. PASS requires professional interactions and liaisons with sponsors/vendors as well as our main partner Microsoft.

In terms of how we operate some of the IT systems – that definitely needs to change. We have a community of data professionals who know how to run events – can we build on that knowledge and perhaps use utilities like Sessionize and other platforms to perform some of the current costly operations more efficiently?

Over my career I have worked with companies to move towards more efficient operations, to remain sustainable and it has meant that some difficult but transparent/open conversations occur around reducing overheads. 

We need to have those same discussions around the future of PASS.

I do not want PASS to die – I want it to continue to serve the community, to be a vehicle that will help people grow. Therefore we need to engage more with that community around what change may look like. Collaboration is vital for the future of PASS.

The PASS Board and management company alike need to engage with the community, to listen to the community’s needs and collaborate on those needs in a positive constructive manner.

PASS needs openness and transparency in how it operates and needs to rebuild the connection with the community.

PASS has to change or it will not survive.

I want to serve our community by leading that change.
That is why I am running for the board.

You can find out about all the candidates in this years election here:

https://www.pass.org/Governance/Elections

We are all passionate people, I’m really glad that we have people who will put themselves forward for the future of the community.  There is a scoring mechanism that can help you the voter decide who are the best candidates to help bring about positive change to ensure the future of PASS.

If you have any questions at all for me please reach out to me @theHybridDBA on twitter.

Update – I have written a follow post here:
https://hybriddbablog.com/2020/11/16/what-the-future-of-pass-needs-and-how-i-can-help/

We need change – but we need positive change for survival & growth.

Vote here:

Azure Platform Series: Creating an Azure Container Registry – ‘registry_name’ must conform to the following pattern

This is another blog post in the series of how to #MakeStuffGo in Azure.

Background:

I am working on a client project that requires containerised applications – so we will be using Azure Container Registry.

I am using bash scripts to create the resources in Azure – so am using variables for all the things.

How to #MakeStuffGo with Azure Container Registry

Initially I had a Resource Group called AKS-RG as the client wanted their resource groups to have the function of what would be in there (in this case Azure Kubernetes Services) and -RG so that it was obvious it was a resource group.

So initially the script for the Azure Container Registry was:

az acr create \
--resource-group $RESOURCE_GROUP \
--location $REGION_NAME \
--name $ACR_NAME \
--sku Standard

Where I had the variables $RESOURCE_GROUP, $REGION_NAME and $ACR_NAME already defined further up in the script.

The only problem was – we had defined

ACR_NAME=acr-$RESOURCE_GROUP
which then failed in Azure Cloud Shell with:
validation error: Parameter 'registry_name' must conform to the following pattern: '^[a-zA-Z0-9]*$'.

The fix was pretty simple – we just changed ACR_NAME variable to be:

ACR_NAME=acr$CLIENTNAME

Which to be honest was a better standard – as we already had the clientname defined as a variable.

It is also the same in the UI (sometimes it is good to try this out as sometimes there are inconsistencies in Azure……)

Screen Shot 2020-06-07 at 17.54.22

So there you go – in some places you can use dashes in Azure and in some places you cannot. But that’s why we have prototypes – to find out all this fun stuff that keeps us on our toes..

 

Yip.

Azure Platform Series: Push to GitHub repo with 2FA from a Mac using Personal Access Token

This is another blog post in the series of how to #MakeStuffGo in Azure.

Background:

I have some code on my Mac. I need to put it into an empty GitHub repo for later deployment in Azure and I have to use a Personal Access Token (PAT). I was provided the URL of the repo and given the PAT so stored that in the keychain on my Mac.

#MakeStuffGo with PAT, GitHub and a Mac

  1. Add the PAT into Keychain:
    Screen Shot 2020-06-07 at 17.25.40
  2. Create the Project Folder on the Mac:
    heymish$ mkdir devops-aks-middleware-api
  3. Initialise the local git repo
    heymish$ cd devops-aks-middleware-api
    heymish$ git init
  4. Add the remote git repository – I have called mine origin – it could be anything
    heymish$ git remote add origin https://github.com/<clientrepo>/devops-aks-middleware-api.git
  5. Create a local branch
    heymish$ git checkout -b master
  6. Do some stuff
    For me I was just copying in code that we had worked on so far – so copied that into the directory and did:

    git status
    <list of files that I've added>
    "nothing added to commit but untracked files present (use "git add" to track)"
  7. Now we want to add those files to be added to our initial commit:
    heymish$ git add .
    heymish$ git commit -m "initial code commit"
  8. Now comes the hard part – getting the 2FA stuff going with GitHub.
    a) we need to setup the GitHub email address we’ll use
    b) we need to setup the GitHub username we’ll use

    heymish$ git config --global user.email "hamishwatson59@clientx.com"
    heymish$ git config --global user.name "TheHamishWatson"
  9. Now we will push our local changes to the remote branch.
    The -u flag indicates that we are pushing local changes upstream to origin

    git push -u origin master
  10. This will now ask me for credentials – which is the PAT I stored in the keychain before:
    Screen Shot 2020-06-07 at 17.19.36
    I type in the password for my keychain and I only click Allow – I’m ok to type in the password every time I need to auth to GitHub – I could click Always Allow – it’s just a personal thing.
  11. Git will now push my local changes up to the repository and I can now start to edit code and collaborate with my clients developers.

So there you go – not too hard and things are nice and secure. There are other ways to use GitHub – ssh vs https which I’ve done here. But for https the above method using a Personal Access Token is pretty straightforward.

 

Yip.