Infrastructure as Python Code: Run your Services on Microsoft Azure
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33783 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 201742 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Service (economics)SoftwareIntelVirtual machineAlgorithmDecision theoryImplementationData storage deviceSoftware developerLecture/Conference
00:25
SoftwareCodeMotion blurShift operatorHuman migrationAsynchronous Transfer ModeArmComputer configurationDeclarative programmingPower (physics)Computer wormInternet service providerGastropod shellWeb serviceLocal GroupGroup actionContinuous trackMenu (computing)Open sourceMachine learningGroup actionIntegrated development environmentSoftware testingLibrary (computing)Revision controlVirtual machineArmDifferent (Kate Ryan album)Web serviceResultantNeuroinformatikDatabaseSocial classBitData miningInternet service providerLevel (video gaming)Data storage devicePhase transitionDeclarative programmingArchitectureMeasurementProcess (computing)Content (media)Client (computing)Computer configurationSampling (statistics)Repository (publishing)SoftwareCellular automatonSoftware developerRepresentational state transferCloud computingDesign by contractProgramming languageData managementPlanningSystem callMultiplication signSlide ruleLink (knot theory)Shift operatorPoint cloudVideo gameDigitizingNumberComputer wormCycle (graph theory)Bit rateWindowEvent horizonStorage area networkOperator (mathematics)Template (C++)Server (computing)LastteilungHuman migrationAsynchronous Transfer ModeProduct (business)Java appletSet (mathematics)Interface (computing)Meeting/InterviewXML
08:06
ZugriffskontrolleData storage deviceLocal GroupComplete metric spaceTemplate (C++)Rollback (data management)ArmNumbering schemeFunction (mathematics)BackupPrice indexString (computer science)Variable (mathematics)Rollback (data management)Group actionDialectArmType theoryCASE <Informatik>Data storage deviceRevision controlLatent heatString (computer science)NumberDigitizingParameter (computer programming)Computer configurationServer (computing)Numbering schemeComputer fileMereologyMultiplication signPoisson-KlammerPositional notationClient (computing)Functional (mathematics)ResultantWeb serviceNamespaceSoftware testingUniqueness quantificationImage resolutionUniform resource locatorLengthCalculationTemplate (C++)Subject indexingPattern languageNeuroinformatikDifferent (Kate Ryan album)Variable (mathematics)Function (mathematics)Complex (psychology)Operator (mathematics)Inheritance (object-oriented programming)Error messageProjective planeDescriptive statisticsRollenbasierte ZugriffskontrolleData managementHuman migrationState of matterSampling (statistics)Single-precision floating-point formatNumeral (linguistics)Game controllerDrop (liquid)Linear regressionPhysical systemSummierbarkeitMaxima and minimaTerm (mathematics)Open sourceXML
15:39
Template (C++)ArmParameter (computer programming)String (computer science)Type theoryObject (grammar)Function (mathematics)Information retrievalComputer fileComplex (psychology)Content (media)Variable (mathematics)Formal languageCommon Language InfrastructureTemplate (C++)Function (mathematics)Representational state transferType theoryObject (grammar)Different (Kate Ryan album)Information retrievalFunctional (mathematics)AreaSensitivity analysisRepository (publishing)Information securityComputer configurationLibrary (computing)Visualization (computer graphics)Parameter (computer programming)Level (video gaming)Product (business)Key (cryptography)Computer fileExpressionVariable (mathematics)String (computer science)DialectFormal languageSoftware testingInternet service providerRevision controlAdditionFiber (mathematics)Link (knot theory)Moment (mathematics)MeasurementMultiplication signJSONXMLComputer animation
18:51
Common Language InfrastructureInteractive televisionQuery languageDifferent (Kate Ryan album)Function (mathematics)File formatInternet service providerConfiguration spaceComputer-generated imageryLoginRevision controlClient (computing)Open sourceDifferent (Kate Ryan album)Parameter (computer programming)Cellular automatonSocial classComplete metric spaceMereologyFigurate numberEstimatorSystem administratorRepresentational state transferServer (computing)Function (mathematics)File format
19:48
SineFunction (mathematics)Table (information)Software engineeringElectronic mailing listQuery languageTemplate (C++)ArmAerodynamicsScripting languageBridging (networking)Server (computing)Parameter (computer programming)Data storage deviceLocal ringTask (computing)Software testingData managementModul <Datentyp>Revision controlPointer (computer programming)Computer-generated imageryFront and back endsClient (computing)Complex (psychology)CodeSoftwareWeb serviceClient (computing)Data storage deviceGroup actionFormal languageFunction (mathematics)File formatDynamical systemScripting languageTemplate (C++)Table (information)Server (computing)Library (computing)Greatest elementUniform resource locatorInformationChainMereologySoftwareRange (statistics)Asynchronous Transfer ModeComputer fileNeuroinformatikData managementKey (cryptography)Distribution (mathematics)Parameter (computer programming)Revision controlLevel (video gaming)Point (geometry)BitImplementationReal numberMoment (mathematics)Computer configurationRepository (publishing)Integrated development environmentOpen sourceVirtual machineRepresentational state transferConfiguration managementBranch (computer science)Software testingSynchronizationDirectory serviceSoftware developerProduct (business)Module (mathematics)Task (computing)Complex (psychology)Configuration spaceDifferent (Kate Ryan album)Query languageBackupPhysical lawCore dumpFrequencyMehrplatzsystemMaß <Mathematik>Multiplication signMeasurementCellular automatonFormal grammarLine (geometry)1 (number)CASE <Informatik>Endliche ModelltheoriePreprocessorDifferenz <Mathematik>Hybrid computerSet (mathematics)PlanningDatabaseState of matterSelectivity (electronic)Game controllerWordCartesian coordinate systemIterationXML
Transcript: English(auto-generated)
00:05
Okay, so my name is Peter Hoffman. I'm a software developer at Blue Yonder We develop machine learning algorithms for retail So for example, we calculate optimized price decisions for online retailers Or we calculate future demand for retailers and implement
00:24
Replenishment on top of it so that the stores of the retailers neither go out of stock or have too much waste So that's yeah, we do this with machine learning and most of our stack is Python You can contact me via Twitter or in the conference or on the social event if you have any questions
00:42
And the slides will be available on github under our company account I'll show the link to the slides again at the end of the presentation. So yeah, you can take your notes So Maybe before we start who thinks Microsoft is a really cool company that knows about open source
01:01
Embraces Python and is really fun to work with Okay, nobody one one one hand so probably that's the same I thought I've been an open source guy for all my life and we have been self hosting our infrastructure at Blue Yonder for all the time We have a three-digit number of servers and sometime our CTO came to us and said, okay
01:26
We want to move to the cloud and we started Evaluating AWS and everything went fine and we thought okay. Let's go to AWS it's the cool cloud provider every cool company is going to AWS and Then some months later he came again and said, okay now we are going to the cloud, but it will be Microsoft Azure
01:45
So at first I thought oh no and now I have to administer a Windows Server I have to learn PowerShell and I Yeah, I have I lose all my open source knowledge and all the stuff But it turned out to be quite different
02:01
I've learned a lot in the last half year. We have started to migrate all of our infrastructure to to to Azure We are following shift and lift approach. So in the first place we are just using the basic resources of Microsoft Azure that storage networking and computation as virtual machines and
02:22
Once we are done with the initial migration We are moving up the stack and are using much more Sophisticated services like managed database services HDFS services or even the managed Hadoop stack So I've quite learned a lot about Microsoft in the last half year. I think they've changed a lot
02:42
Probably they are still not the coolest open source company but they really embrace Python they treat Python as a first-class citizens in their stack and I'll show you in the next 20 or 30 minutes a little bit how we deploy to Azure and what we've learned from our deployments
03:02
So what is Microsoft Azure? Microsoft Azure is a cloud infrastructure provider from from MS and it's basically Infrastructure as a service so you give them money and you get infrastructure It's a little bit the same with our ops guys guys
03:20
We gave them money and they build us the infrastructure But the turnaround cycle with our ops team was about I don't know two to three months Until we get we have to buy the servers You have to put them in the rack You have to put the cables in there and get all the software on it on it So if really long cycles if you just want to try something out or want to scale up
03:42
With infrastructure the service you probably have an API and you can get servers with one click get lots of them and Once you are done with testing thing out, you can throw them away. So you are much faster And which yeah really helps you to grow as a company
04:03
You See here the Azure dashboard that's their UI how you can work with Azure you can click servers virtual machines all the stuff but Once you are beyond the initial phase with trying out stuff. You don't want to deploy your infrastructure. We are UI
04:21
You want to use tools in the best way a declarative declaration of the infrastructure? That you always can deploy it again and probably can deploy the same version of your infrastructure In a testing environment and so you want to all to automate all this stuff So now I'm going to show you how we automate it and how we learned what we learned about the tools that Microsoft provides
04:48
Before we start I'll tell you a little bit about the architecture of Microsoft Azure, so The basic concept in Azure is a resource and resource providers for example virtual machines are resources and
05:03
The resource providers takes care that you can call API and did you get the resources that you requests? So for each different resource, you have a different resource provider. So for virtual machines computation storage or higher level services And they all fulfill the resource provider contract that's a standard API how you can interact with Azure
05:28
and on top of the resource manager you have different tools like the portal like CLI command or a Python client library or even the plain rest calls how you can
05:42
provision your infrastructure in Azure Mostly we have used three different deployment options the one that we don't use is the Microsoft PowerShell The next one is ARM templates. That's a Declarative deployment option and then you have on top of the REST API
06:06
Microsoft ships you Python client libraries and you can talk to them. We are the Python API or we are the ASET command-line client or we are Ansible Really nice thing nice thing is that they use swagger
06:20
For all their API definitions and JSON schemas for the content of their API's or for the payload, so Beside Python they support Java C sharp I think PHP and they all generate the client libraries from the same swagger source definitions
06:41
So you will always have the same version of the client libraries in the different programming languages That's really nice because with this approach a pison is always up-to-date and on par with the PowerShell library Another basic concept of Microsoft Azure are the resource groups
07:02
You can you can group your resources into a resource group and you should do this You could of course put your whole infrastructure in one resource group But it's better to put your resources in different resource groups Based on your deployment lifecycle because normally we'll always deploy a whole resource group in a complete mode
07:25
And you want to not if you just want for example to update your storage accounts You don't want to update the whole infrastructure, but just the resource groups that you're interested in On the left side you see a sample definition of a DEVPI server
07:41
That's a PI PI compatible internal repository we use So we have an availability set and a load balancer and then we have some network Interfaces again storage account and virtual machines and that defines the whole service Internally and you could can use this resource group or the definition of the resources in this group
08:00
To deploy the service in production in development and in testing What's important and what's a really complex topic in Azure is the role-based access Control so for each operation you can do we are the command line or the portal
08:21
You can have a role-based access control So you when you first create the Azure account you are kind of super user and you should really as soon as he has this account start adding other accounts and drop your privileges and give the Deployment accounts just the privileges the need to deploy their resource groups
08:42
You could even say some users are only allowed to view the resource groups to see what's what's inside your Infrastructures and others are able to deploy it That you should really take care of this because it helps to prevent errors That's for example what happened to me I
09:02
Just wanted in the beginning of our migration project I wanted to deploy a resource group, and I missed that there was still a storage account in it So I deleted it and all the data was gone In principle I shouldn't have been able to do this because this storage account was not my business
09:21
It was not my service, but we weren't that far with role-based access management So I could delete it even if I haven't shouldn't have to do it So what we are using as the primary deployment option is ARM templates ARM templates are a declarative JSON based description of the desired deployment state
09:46
It's a JSON document and you submit it to the as a resource manager and the resource manager Takes care of the parallel provision of the resources the rollback and the deployment
10:00
At the bottom I've shown you with a simple command line Interface how you can deploy a resource group you always have to tell Azure which resource group you want to deploy the name And the template where I define all the chase and stuff You can deploy a template to different regions
10:21
Different services and you can define multiple resources in one template But each unique resource only can live in one template You have two different deployment options It's always either complete or incremental if you do an incremental Deployment the Azure resource manager will only add new resources that are in the new template, but will not delete stuff
10:49
That's okay for trying things out, but on the normal way you want to do the complete Deployment where the Azure resource manager also takes care that it deletes resources that are not defined in your template
11:01
Because you always want to have the declarative state that is in your template to be the one that is deployed on Azure So what does a minimal template look like You always have to link to a JSON schema of your template You have to give the content versions and then you can specify resources
11:22
Parameters variables and outputs Microsoft has open-sourced all the JSON schema definitions for the different templates And you can go to get up and we'll see what what kinds of values are allowed in your template for which resources
11:40
So let's define a sample storage account. That's a single resource Again, you have to tell the resource manager which type you want to deploy in this in this case It's Microsoft storage storage account You have to give him the API version you want to talk to so for One resource type there are always different API versions. You can use with different parameters
12:05
You have to tell in which region West Europe US Asia you want to deploy? Your resource and then some specifics about the resource in this case It's just the type of storage account you want to deploy
12:20
locally redundant And one thing we learned pretty soon and use very extensively is tagging of resources so for each resource you can apply a number of tags, I think up to 250 and you can later on use the tags for grouping for example in billing or in monitoring and
12:44
once you have You're going to have a larger infrastructure with I don't know sweet digit number of servers It really helps to see which service is responsible for which which which costs or in monitoring to see which service fails, so
13:02
my advice from the beginning think about a tagging scheme and really apply tags to all your resources and The ARM template is not simply a JSON Template or JSON file, but you can use within the JSON file in the value parts
13:25
You can use the ARM template functions. So each time you use the bracket notation. You basically call a function and During deployment the result of this function will be replaced in the template for the rollout
13:40
for example The storage accounts in Azure they share one big namespace through all customers So probably if you take the name test for your storage account You can't deploy it because it's already used by some other guy and you can here use this unique string function Together with a group resource group ID and it will generate you a unique ID
14:06
That only you use and with this you have the unique name in the global namespace another one is you Don't you already specify in the command line client to which region you want to deploy?
14:21
And with the lower with the resolution resource group location, you can get this value and don't have to type it again and again and Now you also can use this template to deploy it in different regions without changing the value of the location You have lots of different functions at hand you have array functions
14:44
We can use the first or the last value of an array You can get an index of an array the length of an array You have numeric functions to do basic calculations and you have some string functions that you can use in the templates
15:01
Another pattern you can use in your templates is the use of variables So as soon as you need to use one One variable more than once in a template you can define a variable with the storage account and then can use it throughout your template in this case, we are defining a storage account with a name and then we can use the variable in the
15:26
in the computation to get to attach the storage account to a server and The third thing you can use in templates are the outputs So for each variable you generate inside the template you can define an output
15:44
So that once you run your template on Azure, you can see what the actual value once it is evaluated is in your template to use Resource template in different stages in our example
16:01
It's we always have the test area then a staging area and then the production area You want to inject into your template external values and you can do this with parameters So you define the parameters you want to use in your in your template? You can use it in your resources and once you want to deploy it
16:22
You can specify an additional template and parameters file where you actually provide the real value so you have one template and different parameters file and with this you can deploy it in different regions or in different staging areas of your infrastructure
16:41
What we've also learned pretty fast is don't put sensitive data into templates So Microsoft Azure provides ways To inject sensitive data into your templates without having them in your Git repository and in plain text, so they have to secure string and secure option type objects
17:03
They have front-end retrievals within the template function for the secure type objects and you can also reference the key value secrets So you can in Azure Generate a key value vault with secrets and then you can use it into it in your templates and In production always turn off debugging and logging because it could also dump out your secrets
17:29
For rather simple deployments the Azure resource Is pretty okay, but for complex ones, it's
17:44
Just a JSON file where you define your Resources, but you have also content versioning for different resources You have parameters and variables you have inline template expression language. You can also link template templates together so it's pretty fast gets pretty hard to
18:03
To edit all these templates by hand. So we have a pretty soon come to Visual studio which supports the Azure resource templates syntax and you have auto intelligence as IntelliSense and highlighting and it makes it much more easier to really edit these templates
18:21
we have also tried to Put some Python libraries around the templates and generate the stuff But that didn't work out that well So at the moment, we still edit the templates by hand, but with a powerful tool like Visual Studio So, how do you actually talk to the to the
18:43
REST API from Microsoft Azure Azure provider Microsoft provides a command-line interface The command-line interface version 1 was built in node.js But as the Microsoft guys told us that didn't work that well Because with the command-line client they want to target Linux admins and users from the open source
19:05
Community and the node.js clients just didn't behave Like the tools like you expected in this Linux server server community So they developed the CLD version 2 and they developed it in Python
19:21
It's a really nice command-line interfaces with auto Completion and nice documentation and different output formats It has support for searching via GMS path and It's also fully generated from the swagger definitions also REST API
19:40
So they're always up to date with this command-line client and it really helps you to work with the REST API yeah, for example, you can tell the the a say command-line client To list you all the storage accounts inside a resource group and then you get back a JSON document with all the information
20:05
That's fine. If you want to use it to Pre-process it in Python or some language that understands JSON or even pipe it into a JQ and then select just some values But you can also use different output formats So for example, you can dump it as a table the command-line interface uses the tabulate library
20:26
That's a common library in Python to dump table your formats or you can dump it as top separated values and then Call your org script To get the data out of it. So it really fits well into the command-line chain on Linux
20:44
If I said earlier can also use gems parts the query language. So for example here, I just want to list all storage accounts that are a standard RGS From the name and you just want to have the name and the endpoint of the blob
21:02
So that's really nice to interact with the twister API We use Ansible exclusively in configuration management to a provision and deploy our server and services and we also
21:20
Thought it was a good way to use it to provision our stuff on Azure because with this ansatz We don't have to have duplicate Configuration for example we have network ranges or host names and you have to define them in the Azure template and we also had defined them in our Ansible scripts, so
21:42
There's an Azure RM module for for Ansible and we are using this to interact with Azure through Ansible So there are three ways how we use it We use it to deploy the ARM templates with Ansible. There is the possibility to generate
22:03
resources directly via the REST API and we use it as a dynamic inventory script To bridge the server and services into our other Ansible scripts simple Azure deployment It's pretty easy. It's not that different to the command-line client
22:23
So at the bottom you define your resource template You can also define some parameters that you want to use in the resource template You tell Azure which resource group you want to deploy in which location and you have the same Deployment modes as I said earlier the complete and the incremental deployment mode. So that's
22:45
pretty and the same than the command-line client, but What it really helps us using the parameters from within Ansible that we can use the same parameters that we use elsewhere Instead of providing a template file. You could also use the Azure
23:02
RM mode for Ansible to specify the resources inline Ansible this works quite okay for for simple resources like computation storage and Networking But for the more sophisticated ones, there's still no support
23:23
from for in Ansible for the Azure modules And that's what I said. There's only support for this for for for services So we'll still stick to the resource templates So this is a little more complex example. So we are using the resource
23:52
Manager with Ansible to deploy virtual machine Again, you have some parameters where you tell it which size of machine you want to deploy which third account you want to use
24:04
The initial SSH key set up and in which network you want to deploy the machine Microsoft offers quite a range of Linux distributions which you can use so in this case, we just say okay Give us the latest Debian 8 and you'll always get an up-to-date version
24:26
If you want if you deploy lots of machines it probably helps If you tag them right and then you can use the Ansible Dynamic inventories feature to pull the actual information from your deployed
24:42
Infrastructure and feed it in your Ansible scripts and reuse the groups that are definitely defined there for all your other Actions you want to do later in the provisioning stage So Why we quite use Ansible with Azure in production. We are not
25:05
All of it. There are some points where we are not that happy about it It does not work with the latest client libraries. So it's always a little bit behind Microsoft did the initial implementation of the Azure Ansible module But there's not real open source community around it. So
25:25
Until now, I'm not sure if let's see how it will further involve Using Defining resources within Ansible is okay for Tim simple tasks for complex tasks It's better to switch back to the resource templates and just call the template. We are in the book
25:44
the dynamic inventory was really helpful for us and we pretty much use it and it works and And so now we kind of have a hybrid approach where we use the chase and templates to define all our infrastructure And inject the parameters we are via Ansible
26:04
So that's a rough overview of the deployment how we use Microsoft Azure As I said earlier if you have any questions, you can ask them now or just meet me at the conference. Thanks
26:37
You said you were planning to shift to Azure and then start shifting up into the high-level services
26:43
I wonder if you've got an idea yet of what those high-level services you're thinking of shifting to are For example until now we use only the simple blob storage, but we want to migrate to the HDFS services that Azure provides
27:01
At the moment we are deploying our Postgres databases at our own Microsoft Azure has on demand service for Postgres services where if your automatic backup and automatic upgrades and all this stuff So that of two examples where we want to to go up in the stack and even something like they have a full managed
27:23
Hadoop service We don't have to care about Deploying the stuff that's also an option All right, thanks for your talk I was wondering how do you authenticate your tools like
27:49
Ansible's or like any other script you have against the Azure API Yeah, basically we have a GitHub repository with all your our deployments Installed there. We have a requirement txt file and that's what everybody uses and once we want to migrate to a newer version
28:07
We do a branch and then update the versions deployed to the test developer Environment and see if everything works and then roll it out to the other stages Yeah, but you mentioned
28:34
Now we have we have a sync tool we have an internal active directory or it's an open held up server and we've written a sync
28:42
He sings the permissions that are in or internal the groups and the developers to Azure and then we have some Labels attached to each group or developer and that's the way we manage the world best access
29:01
No, we have that's a separate tool we just use the API to sync our users inside Okay, then thank you