Issue occurred while running a planned upgrade Kubernetes using the Azure provided method of upgrading the clusters. Process was carefully tested initially on both our Dev and STG environments which revealed some issues we resolved so they don’t occur once upgrading Production.

During the Production cluster upgrades the backend upgrades were taking a considerate amount of time on the Azure end. We engaged Microsoft and monitored their work trying to expedite the process. During the course of the backend nodes being upgraded to some services went into a degraded state. In this case the Inbound load balancer was having some deployment issues that automatically resolved once the cluster upgrade completed.