We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Infrastructure as Python Code: Run your Services on Microsoft Azure

00:00

Formal Metadata

Title
Infrastructure as Python Code: Run your Services on Microsoft Azure
Title of Series
Number of Parts
160
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Infrastructure as Python Code: Run your Services on Microsoft Azure [EuroPython 2017 - Talk - 2017-07-11 - Anfiteatro 1] [Rimini, Italy] Using Infrastructure-as-Code principles with configuration through machine processable definition files in combination with the adoption of cloud computing provides faster feedback cycles in development/testing and less risk in deployment to production. The Microsoft Azure Cloud (https://azure.microsoft.com/) allows different ways to provision, deploy and run your python service: The Azure Resource Manger Templates (https://azure.microsoft.com/en-us/resources/templates/) allows you to provision your application using a declarative template. With parameters, variables and Azure template functions, the same template can be used to deploy your application in different stages (dev, test, production) and environments for different customers. We open sourced the tropo library (https://pypi.python.org/pypi/tropo/) to create Azure Resource Templates from python. Azure SDK for Python (http://azure-sdk-for-python.readthedocs.io) for a low level access to manage resources in the Azure Cloud. An Azure Ansible Module (https://docs.ansible.com/ansible/guide_azure.html) based on the Azure SDK to automate software provisioning, configuration management, and application deployment in a single environment. Each of the alternatives has different strengths and drawbacks. Presenting our learnings from migrating our infrastructure into the Azrue Cloud will help to avoid common pitfalls and show deployment patterns that will ease the live of devops
95
Thumbnail
1:04:08
102
119
Thumbnail
1:00:51
Service (economics)SoftwareIntelVirtual machineAlgorithmDecision theoryImplementationData storage deviceSoftware developerLecture/Conference
SoftwareCodeMotion blurShift operatorHuman migrationAsynchronous Transfer ModeArmComputer configurationDeclarative programmingPower (physics)Computer wormInternet service providerGastropod shellWeb serviceLocal GroupGroup actionContinuous trackMenu (computing)Open sourceMachine learningGroup actionIntegrated development environmentSoftware testingLibrary (computing)Revision controlVirtual machineArmDifferent (Kate Ryan album)Web serviceResultantNeuroinformatikDatabaseSocial classBitData miningInternet service providerLevel (video gaming)Data storage devicePhase transitionDeclarative programmingArchitectureMeasurementProcess (computing)Content (media)Client (computing)Computer configurationSampling (statistics)Repository (publishing)SoftwareCellular automatonSoftware developerRepresentational state transferCloud computingDesign by contractProgramming languageData managementPlanningSystem callMultiplication signSlide ruleLink (knot theory)Shift operatorPoint cloudVideo gameDigitizingNumberComputer wormCycle (graph theory)Bit rateWindowEvent horizonStorage area networkOperator (mathematics)Template (C++)Server (computing)LastteilungHuman migrationAsynchronous Transfer ModeProduct (business)Java appletSet (mathematics)Interface (computing)Meeting/InterviewXML
ZugriffskontrolleData storage deviceLocal GroupComplete metric spaceTemplate (C++)Rollback (data management)ArmNumbering schemeFunction (mathematics)BackupPrice indexString (computer science)Variable (mathematics)Rollback (data management)Group actionDialectArmType theoryCASE <Informatik>Data storage deviceRevision controlLatent heatString (computer science)NumberDigitizingParameter (computer programming)Computer configurationServer (computing)Numbering schemeComputer fileMereologyMultiplication signPoisson-KlammerPositional notationClient (computing)Functional (mathematics)ResultantWeb serviceNamespaceSoftware testingUniqueness quantificationImage resolutionUniform resource locatorLengthCalculationTemplate (C++)Subject indexingPattern languageNeuroinformatikDifferent (Kate Ryan album)Variable (mathematics)Function (mathematics)Complex (psychology)Operator (mathematics)Inheritance (object-oriented programming)Error messageProjective planeDescriptive statisticsRollenbasierte ZugriffskontrolleData managementHuman migrationState of matterSampling (statistics)Single-precision floating-point formatNumeral (linguistics)Game controllerDrop (liquid)Linear regressionPhysical systemSummierbarkeitMaxima and minimaTerm (mathematics)Open sourceXML
Template (C++)ArmParameter (computer programming)String (computer science)Type theoryObject (grammar)Function (mathematics)Information retrievalComputer fileComplex (psychology)Content (media)Variable (mathematics)Formal languageCommon Language InfrastructureTemplate (C++)Function (mathematics)Representational state transferType theoryObject (grammar)Different (Kate Ryan album)Information retrievalFunctional (mathematics)AreaSensitivity analysisRepository (publishing)Information securityComputer configurationLibrary (computing)Visualization (computer graphics)Parameter (computer programming)Level (video gaming)Product (business)Key (cryptography)Computer fileExpressionVariable (mathematics)String (computer science)DialectFormal languageSoftware testingInternet service providerRevision controlAdditionFiber (mathematics)Link (knot theory)Moment (mathematics)MeasurementMultiplication signJSONXMLComputer animation
Common Language InfrastructureInteractive televisionQuery languageDifferent (Kate Ryan album)Function (mathematics)File formatInternet service providerConfiguration spaceComputer-generated imageryLoginRevision controlClient (computing)Open sourceDifferent (Kate Ryan album)Parameter (computer programming)Cellular automatonSocial classComplete metric spaceMereologyFigurate numberEstimatorSystem administratorRepresentational state transferServer (computing)Function (mathematics)File format
SineFunction (mathematics)Table (information)Software engineeringElectronic mailing listQuery languageTemplate (C++)ArmAerodynamicsScripting languageBridging (networking)Server (computing)Parameter (computer programming)Data storage deviceLocal ringTask (computing)Software testingData managementModul <Datentyp>Revision controlPointer (computer programming)Computer-generated imageryFront and back endsClient (computing)Complex (psychology)CodeSoftwareWeb serviceClient (computing)Data storage deviceGroup actionFormal languageFunction (mathematics)File formatDynamical systemScripting languageTemplate (C++)Table (information)Server (computing)Library (computing)Greatest elementUniform resource locatorInformationChainMereologySoftwareRange (statistics)Asynchronous Transfer ModeComputer fileNeuroinformatikData managementKey (cryptography)Distribution (mathematics)Parameter (computer programming)Revision controlLevel (video gaming)Point (geometry)BitImplementationReal numberMoment (mathematics)Computer configurationRepository (publishing)Integrated development environmentOpen sourceVirtual machineRepresentational state transferConfiguration managementBranch (computer science)Software testingSynchronizationDirectory serviceSoftware developerProduct (business)Module (mathematics)Task (computing)Complex (psychology)Configuration spaceDifferent (Kate Ryan album)Query languageBackupPhysical lawCore dumpFrequencyMehrplatzsystemMaß <Mathematik>Multiplication signMeasurementCellular automatonFormal grammarLine (geometry)1 (number)CASE <Informatik>Endliche ModelltheoriePreprocessorDifferenz <Mathematik>Hybrid computerSet (mathematics)PlanningDatabaseState of matterSelectivity (electronic)Game controllerWordCartesian coordinate systemIterationXML
Transcript: English(auto-generated)
Okay, so my name is Peter Hoffman. I'm a software developer at Blue Yonder We develop machine learning algorithms for retail So for example, we calculate optimized price decisions for online retailers Or we calculate future demand for retailers and implement
Replenishment on top of it so that the stores of the retailers neither go out of stock or have too much waste So that's yeah, we do this with machine learning and most of our stack is Python You can contact me via Twitter or in the conference or on the social event if you have any questions
And the slides will be available on github under our company account I'll show the link to the slides again at the end of the presentation. So yeah, you can take your notes So Maybe before we start who thinks Microsoft is a really cool company that knows about open source
Embraces Python and is really fun to work with Okay, nobody one one one hand so probably that's the same I thought I've been an open source guy for all my life and we have been self hosting our infrastructure at Blue Yonder for all the time We have a three-digit number of servers and sometime our CTO came to us and said, okay
We want to move to the cloud and we started Evaluating AWS and everything went fine and we thought okay. Let's go to AWS it's the cool cloud provider every cool company is going to AWS and Then some months later he came again and said, okay now we are going to the cloud, but it will be Microsoft Azure
So at first I thought oh no and now I have to administer a Windows Server I have to learn PowerShell and I Yeah, I have I lose all my open source knowledge and all the stuff But it turned out to be quite different
I've learned a lot in the last half year. We have started to migrate all of our infrastructure to to to Azure We are following shift and lift approach. So in the first place we are just using the basic resources of Microsoft Azure that storage networking and computation as virtual machines and
Once we are done with the initial migration We are moving up the stack and are using much more Sophisticated services like managed database services HDFS services or even the managed Hadoop stack So I've quite learned a lot about Microsoft in the last half year. I think they've changed a lot
Probably they are still not the coolest open source company but they really embrace Python they treat Python as a first-class citizens in their stack and I'll show you in the next 20 or 30 minutes a little bit how we deploy to Azure and what we've learned from our deployments
So what is Microsoft Azure? Microsoft Azure is a cloud infrastructure provider from from MS and it's basically Infrastructure as a service so you give them money and you get infrastructure It's a little bit the same with our ops guys guys
We gave them money and they build us the infrastructure But the turnaround cycle with our ops team was about I don't know two to three months Until we get we have to buy the servers You have to put them in the rack You have to put the cables in there and get all the software on it on it So if really long cycles if you just want to try something out or want to scale up
With infrastructure the service you probably have an API and you can get servers with one click get lots of them and Once you are done with testing thing out, you can throw them away. So you are much faster And which yeah really helps you to grow as a company
You See here the Azure dashboard that's their UI how you can work with Azure you can click servers virtual machines all the stuff but Once you are beyond the initial phase with trying out stuff. You don't want to deploy your infrastructure. We are UI
You want to use tools in the best way a declarative declaration of the infrastructure? That you always can deploy it again and probably can deploy the same version of your infrastructure In a testing environment and so you want to all to automate all this stuff So now I'm going to show you how we automate it and how we learned what we learned about the tools that Microsoft provides
Before we start I'll tell you a little bit about the architecture of Microsoft Azure, so The basic concept in Azure is a resource and resource providers for example virtual machines are resources and
The resource providers takes care that you can call API and did you get the resources that you requests? So for each different resource, you have a different resource provider. So for virtual machines computation storage or higher level services And they all fulfill the resource provider contract that's a standard API how you can interact with Azure
and on top of the resource manager you have different tools like the portal like CLI command or a Python client library or even the plain rest calls how you can
provision your infrastructure in Azure Mostly we have used three different deployment options the one that we don't use is the Microsoft PowerShell The next one is ARM templates. That's a Declarative deployment option and then you have on top of the REST API
Microsoft ships you Python client libraries and you can talk to them. We are the Python API or we are the ASET command-line client or we are Ansible Really nice thing nice thing is that they use swagger
For all their API definitions and JSON schemas for the content of their API's or for the payload, so Beside Python they support Java C sharp I think PHP and they all generate the client libraries from the same swagger source definitions
So you will always have the same version of the client libraries in the different programming languages That's really nice because with this approach a pison is always up-to-date and on par with the PowerShell library Another basic concept of Microsoft Azure are the resource groups
You can you can group your resources into a resource group and you should do this You could of course put your whole infrastructure in one resource group But it's better to put your resources in different resource groups Based on your deployment lifecycle because normally we'll always deploy a whole resource group in a complete mode
And you want to not if you just want for example to update your storage accounts You don't want to update the whole infrastructure, but just the resource groups that you're interested in On the left side you see a sample definition of a DEVPI server
That's a PI PI compatible internal repository we use So we have an availability set and a load balancer and then we have some network Interfaces again storage account and virtual machines and that defines the whole service Internally and you could can use this resource group or the definition of the resources in this group
To deploy the service in production in development and in testing What's important and what's a really complex topic in Azure is the role-based access Control so for each operation you can do we are the command line or the portal
You can have a role-based access control So you when you first create the Azure account you are kind of super user and you should really as soon as he has this account start adding other accounts and drop your privileges and give the Deployment accounts just the privileges the need to deploy their resource groups
You could even say some users are only allowed to view the resource groups to see what's what's inside your Infrastructures and others are able to deploy it That you should really take care of this because it helps to prevent errors That's for example what happened to me I
Just wanted in the beginning of our migration project I wanted to deploy a resource group, and I missed that there was still a storage account in it So I deleted it and all the data was gone In principle I shouldn't have been able to do this because this storage account was not my business
It was not my service, but we weren't that far with role-based access management So I could delete it even if I haven't shouldn't have to do it So what we are using as the primary deployment option is ARM templates ARM templates are a declarative JSON based description of the desired deployment state
It's a JSON document and you submit it to the as a resource manager and the resource manager Takes care of the parallel provision of the resources the rollback and the deployment
At the bottom I've shown you with a simple command line Interface how you can deploy a resource group you always have to tell Azure which resource group you want to deploy the name And the template where I define all the chase and stuff You can deploy a template to different regions
Different services and you can define multiple resources in one template But each unique resource only can live in one template You have two different deployment options It's always either complete or incremental if you do an incremental Deployment the Azure resource manager will only add new resources that are in the new template, but will not delete stuff
That's okay for trying things out, but on the normal way you want to do the complete Deployment where the Azure resource manager also takes care that it deletes resources that are not defined in your template
Because you always want to have the declarative state that is in your template to be the one that is deployed on Azure So what does a minimal template look like You always have to link to a JSON schema of your template You have to give the content versions and then you can specify resources
Parameters variables and outputs Microsoft has open-sourced all the JSON schema definitions for the different templates And you can go to get up and we'll see what what kinds of values are allowed in your template for which resources
So let's define a sample storage account. That's a single resource Again, you have to tell the resource manager which type you want to deploy in this in this case It's Microsoft storage storage account You have to give him the API version you want to talk to so for One resource type there are always different API versions. You can use with different parameters
You have to tell in which region West Europe US Asia you want to deploy? Your resource and then some specifics about the resource in this case It's just the type of storage account you want to deploy
locally redundant And one thing we learned pretty soon and use very extensively is tagging of resources so for each resource you can apply a number of tags, I think up to 250 and you can later on use the tags for grouping for example in billing or in monitoring and
once you have You're going to have a larger infrastructure with I don't know sweet digit number of servers It really helps to see which service is responsible for which which which costs or in monitoring to see which service fails, so
my advice from the beginning think about a tagging scheme and really apply tags to all your resources and The ARM template is not simply a JSON Template or JSON file, but you can use within the JSON file in the value parts
You can use the ARM template functions. So each time you use the bracket notation. You basically call a function and During deployment the result of this function will be replaced in the template for the rollout
for example The storage accounts in Azure they share one big namespace through all customers So probably if you take the name test for your storage account You can't deploy it because it's already used by some other guy and you can here use this unique string function Together with a group resource group ID and it will generate you a unique ID
That only you use and with this you have the unique name in the global namespace another one is you Don't you already specify in the command line client to which region you want to deploy?
And with the lower with the resolution resource group location, you can get this value and don't have to type it again and again and Now you also can use this template to deploy it in different regions without changing the value of the location You have lots of different functions at hand you have array functions
We can use the first or the last value of an array You can get an index of an array the length of an array You have numeric functions to do basic calculations and you have some string functions that you can use in the templates
Another pattern you can use in your templates is the use of variables So as soon as you need to use one One variable more than once in a template you can define a variable with the storage account and then can use it throughout your template in this case, we are defining a storage account with a name and then we can use the variable in the
in the computation to get to attach the storage account to a server and The third thing you can use in templates are the outputs So for each variable you generate inside the template you can define an output
So that once you run your template on Azure, you can see what the actual value once it is evaluated is in your template to use Resource template in different stages in our example
It's we always have the test area then a staging area and then the production area You want to inject into your template external values and you can do this with parameters So you define the parameters you want to use in your in your template? You can use it in your resources and once you want to deploy it
You can specify an additional template and parameters file where you actually provide the real value so you have one template and different parameters file and with this you can deploy it in different regions or in different staging areas of your infrastructure
What we've also learned pretty fast is don't put sensitive data into templates So Microsoft Azure provides ways To inject sensitive data into your templates without having them in your Git repository and in plain text, so they have to secure string and secure option type objects
They have front-end retrievals within the template function for the secure type objects and you can also reference the key value secrets So you can in Azure Generate a key value vault with secrets and then you can use it into it in your templates and In production always turn off debugging and logging because it could also dump out your secrets
For rather simple deployments the Azure resource Is pretty okay, but for complex ones, it's
Just a JSON file where you define your Resources, but you have also content versioning for different resources You have parameters and variables you have inline template expression language. You can also link template templates together so it's pretty fast gets pretty hard to
To edit all these templates by hand. So we have a pretty soon come to Visual studio which supports the Azure resource templates syntax and you have auto intelligence as IntelliSense and highlighting and it makes it much more easier to really edit these templates
we have also tried to Put some Python libraries around the templates and generate the stuff But that didn't work out that well So at the moment, we still edit the templates by hand, but with a powerful tool like Visual Studio So, how do you actually talk to the to the
REST API from Microsoft Azure Azure provider Microsoft provides a command-line interface The command-line interface version 1 was built in node.js But as the Microsoft guys told us that didn't work that well Because with the command-line client they want to target Linux admins and users from the open source
Community and the node.js clients just didn't behave Like the tools like you expected in this Linux server server community So they developed the CLD version 2 and they developed it in Python
It's a really nice command-line interfaces with auto Completion and nice documentation and different output formats It has support for searching via GMS path and It's also fully generated from the swagger definitions also REST API
So they're always up to date with this command-line client and it really helps you to work with the REST API yeah, for example, you can tell the the a say command-line client To list you all the storage accounts inside a resource group and then you get back a JSON document with all the information
That's fine. If you want to use it to Pre-process it in Python or some language that understands JSON or even pipe it into a JQ and then select just some values But you can also use different output formats So for example, you can dump it as a table the command-line interface uses the tabulate library
That's a common library in Python to dump table your formats or you can dump it as top separated values and then Call your org script To get the data out of it. So it really fits well into the command-line chain on Linux
If I said earlier can also use gems parts the query language. So for example here, I just want to list all storage accounts that are a standard RGS From the name and you just want to have the name and the endpoint of the blob
So that's really nice to interact with the twister API We use Ansible exclusively in configuration management to a provision and deploy our server and services and we also
Thought it was a good way to use it to provision our stuff on Azure because with this ansatz We don't have to have duplicate Configuration for example we have network ranges or host names and you have to define them in the Azure template and we also had defined them in our Ansible scripts, so
There's an Azure RM module for for Ansible and we are using this to interact with Azure through Ansible So there are three ways how we use it We use it to deploy the ARM templates with Ansible. There is the possibility to generate
resources directly via the REST API and we use it as a dynamic inventory script To bridge the server and services into our other Ansible scripts simple Azure deployment It's pretty easy. It's not that different to the command-line client
So at the bottom you define your resource template You can also define some parameters that you want to use in the resource template You tell Azure which resource group you want to deploy in which location and you have the same Deployment modes as I said earlier the complete and the incremental deployment mode. So that's
pretty and the same than the command-line client, but What it really helps us using the parameters from within Ansible that we can use the same parameters that we use elsewhere Instead of providing a template file. You could also use the Azure
RM mode for Ansible to specify the resources inline Ansible this works quite okay for for simple resources like computation storage and Networking But for the more sophisticated ones, there's still no support
from for in Ansible for the Azure modules And that's what I said. There's only support for this for for for services So we'll still stick to the resource templates So this is a little more complex example. So we are using the resource
Manager with Ansible to deploy virtual machine Again, you have some parameters where you tell it which size of machine you want to deploy which third account you want to use
The initial SSH key set up and in which network you want to deploy the machine Microsoft offers quite a range of Linux distributions which you can use so in this case, we just say okay Give us the latest Debian 8 and you'll always get an up-to-date version
If you want if you deploy lots of machines it probably helps If you tag them right and then you can use the Ansible Dynamic inventories feature to pull the actual information from your deployed
Infrastructure and feed it in your Ansible scripts and reuse the groups that are definitely defined there for all your other Actions you want to do later in the provisioning stage So Why we quite use Ansible with Azure in production. We are not
All of it. There are some points where we are not that happy about it It does not work with the latest client libraries. So it's always a little bit behind Microsoft did the initial implementation of the Azure Ansible module But there's not real open source community around it. So
Until now, I'm not sure if let's see how it will further involve Using Defining resources within Ansible is okay for Tim simple tasks for complex tasks It's better to switch back to the resource templates and just call the template. We are in the book
the dynamic inventory was really helpful for us and we pretty much use it and it works and And so now we kind of have a hybrid approach where we use the chase and templates to define all our infrastructure And inject the parameters we are via Ansible
So that's a rough overview of the deployment how we use Microsoft Azure As I said earlier if you have any questions, you can ask them now or just meet me at the conference. Thanks
You said you were planning to shift to Azure and then start shifting up into the high-level services
I wonder if you've got an idea yet of what those high-level services you're thinking of shifting to are For example until now we use only the simple blob storage, but we want to migrate to the HDFS services that Azure provides
At the moment we are deploying our Postgres databases at our own Microsoft Azure has on demand service for Postgres services where if your automatic backup and automatic upgrades and all this stuff So that of two examples where we want to to go up in the stack and even something like they have a full managed
Hadoop service We don't have to care about Deploying the stuff that's also an option All right, thanks for your talk I was wondering how do you authenticate your tools like
Ansible's or like any other script you have against the Azure API Yeah, basically we have a GitHub repository with all your our deployments Installed there. We have a requirement txt file and that's what everybody uses and once we want to migrate to a newer version
We do a branch and then update the versions deployed to the test developer Environment and see if everything works and then roll it out to the other stages Yeah, but you mentioned
Now we have we have a sync tool we have an internal active directory or it's an open held up server and we've written a sync
He sings the permissions that are in or internal the groups and the developers to Azure and then we have some Labels attached to each group or developer and that's the way we manage the world best access
No, we have that's a separate tool we just use the API to sync our users inside Okay, then thank you