Network visualization and automation
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33681 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017141 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Visualization (computer graphics)Computer networkSoftwareIntelComa BerenicesDemonDenial-of-service attackRouter (computing)OpticsFiber (mathematics)Computer networkCrash (computing)PlanningDistanceLink (knot theory)Type theoryData transmissionInternet service providerGraph drawingOpticsTelecommunicationPlastikkarteFacebook1 (number)Table (information)InternetworkingService (economics)Right angleComputer networkInterface (computing)Moment <Mathematik>Computer animation
01:53
LengthRule of inferenceTable (information)Streaming mediaFiber (mathematics)MereologyVideo gameMatter waveAssociative propertyIP addressSystem callDecision theoryInterface (computing)Division (mathematics)ApproximationContent (media)Order (biology)Band matrixChannel capacitySlide ruleClosed setDiagram
03:54
Computer networkData managementVisualization (computer graphics)Physical systemComputer networkRingnetzAlgorithmGraphical user interfaceSoftware frameworkComputer programmingGUI widgetPolygonShape (magazine)Module (mathematics)MultiplicationPoint (geometry)Computer fileLibrary (computing)Order (biology)Projective planeGUI widgetPixelPolygonComputer fileShape (magazine)Coordinate systemData conversionSphereComputer programInformationDistortion (mathematics)Process (computing)Software frameworkRandomizationLoop (music)Functional (mathematics)Computer configurationPoint (geometry)Type theoryMereologyAngleInterface (computing)Set (mathematics)Electronic mailing listObject (grammar)MultiplicationPosition operatorLink (knot theory)Library (computing)outputComputer networkSoftwareLevel (video gaming)Module (mathematics)Web applicationNetzwerkverwaltungTheoremRingnetzGraph (mathematics)Bus (computing)MappingDegree (graph theory)Distribution (mathematics)Default (computer science)Uniform resource locatorMoving averageAlgorithmDistanceElectronic visual displayPlanningPredictabilityInstance (computer science)File formatSlide rulePhysical systemAreaString (computer science)RectangleCircleComputer animation
11:21
FingerprintLaptopOpen setAttribute grammarCodeWindowComputer animationSource codeXMLProgram flowchart
12:19
Inclusion mapMaxima and minimaInflection pointShape (magazine)Chemical polarityPolygonMetreInversion (music)Drum memorySoftware testingMenu (computing)Color managementQuiltPolygonShape (magazine)Functional (mathematics)Reverse engineeringOrder (biology)Projective planeEuler anglesBitBus (computing)Thermal conductivityRadiusOperator (mathematics)Object (grammar)State of matterPublic domainGraphical user interfaceLoop (music)WindowSound effectCodeComputer fileCodeCausalityPredictabilityCoordinate systemData conversionCircleHazard (2005 film)Computer animation
15:40
Lemma (mathematics)PlastikkarteAsynchronous Transfer ModeEmulationDampingDuality (mathematics)Line (geometry)ResultantComputer networkComputer fileOrder (biology)Projective planeReal numberComputer animation
16:47
Drum memoryAddress spaceAddress spaceConfiguration spaceLevel (video gaming)Physical systemInstance (computer science)Process (computing)Connected spaceMereologyComputer networkCASE <Informatik>Water vaporSlide ruleEndliche ModelltheorieSoftwareInternetworkingRight angleNeuroinformatikDrop (liquid)Type theorySoftware frameworkRadical (chemistry)Category of beingIP addressDrag (physics)Router (computing)Message passingAsynchronous Transfer Mode
20:28
Maxima and minimaPolygonLimit (category theory)Wechselseitige InformationComputer networkTelecommunicationTask (computing)Library (computing)DampingCommunications protocolConfiguration spaceMultiplicationComputer networkData managementTelnetConic sectionComa BerenicesOperations support systemLatent heatSlide ruleOpen sourceMultiplication signComputer networkProjective planeOrder (biology)View (database)1 (number)Particle systemComputer networkConfiguration spaceCommunications protocolLibrary (computing)Color confinementModeling languageDefault (computer science)Social classConfiguration managementMomentumConnected spaceDifferent (Kate Ryan album)CodeComputer animationProgram flowchart
23:03
PasswordConvex hullSpherical capMathematicsInformationConfiguration spaceComputer fileScripting languageoutputOrder (biology)Line (geometry)Object (grammar)IP addressResultantVariable (mathematics)Descriptive statisticsRouter (computing)Module (mathematics)Parameter (computer programming)Thread (computing)Connected spaceData dictionaryInterface (computing)CodeFunction (mathematics)Multiplication signMathematicsElectronic mailing listLoop (music)Maxima and minimaWater vapor2 (number)WordAddress spaceInstance (computer science)Execution unitGastropod shellTraffic reportingData storage deviceComputer animation
28:58
Video gameExpert systemInstance (computer science)MereologyScripting languageFinite-state machineLoginLecture/Conference
29:47
Representation (politics)Shape (magazine)Presentation of a groupPosition operatorQuery languageComputer networkSoftwareRingnetzOrder (biology)Multiplication signLecture/Conference
30:48
Graphical user interfaceFlow separationLecture/Conference
31:33
Grand Unified TheoryScripting languageMathematicsMereologyComputer networkRadical (chemistry)Category of beingConfiguration spaceComputer fileRight angleSoftwareComputer animation
32:35
IP addressIntegrated development environmentGraphical user interfaceSheaf (mathematics)Computer fileReal numberCategory of beingScripting languageComputer networkGraph (mathematics)MereologyMappingData storage deviceInterface (computing)
Transcript: English(auto-generated)
00:05
So, the plan is very simple, I will start with network visualization and I will end with network automation. But before I start, I want to make a short introduction into networking, like a crash course in networking, for those of you who don't know well what it means.
00:25
So when I say network, what I mean is a telecommunication network. It's a network which nodes are networking devices, like routers, switches and antennas, and a network which links are transmission links, usually ethernet cables and optical fibers.
00:47
In the network of an internet service provider, we find three main types of nodes, router, optical switch and antenna. As you can see, these are very big devices, like with lots of cards and lots of interfaces,
01:03
so this has nothing to do with the type of device that you would have at home. These are corporate devices that we would find in the network of a big company like Facebook or Google, or in the network of an internet service provider like Vodafone,
01:21
Telefonica and so on. Same for the transmission links, these are links like those ones on the left, an ethernet cable with 10G capacity, a 10G connector, and on the right, an optical fiber.
01:41
So ethernet cables are used for short distance transmission, and optical fibers can be used for both short distance transmission and long distance transmissions. Well, there are a few approximations in this slide because it's like to make it simple
02:02
for you to understand, but if we imagine that you are in Remini and you want to call someone in France, like with your mobile phone, with a 4G, what would happen is that like your mobile phone would send a microwave signal to an antenna that would be located nearby, like close to this Pala Congresi, and then the signal will travel through a
02:26
bunch of antennas. It will be a microwave signal, and this microwave part with antennas, we call it the microwave back hole. Eventually it will reach a rotor. A rotor is a device that will take the forwarding decision. What we can say about
02:45
them, it's like the brain of the rotor. They know how to reach any part of the world, and they will make the forwarding decisions. They will know which path the traffic has to take. In order to do that, the rotor has what we call a routing table, which
03:02
is a table that associates an outgoing interface to an IP address. And finally, since we want to go to France, which is very far, we need the signal to go very fast. So the rotor will send a signal to what we call an optical switch, and this
03:22
optical switch will perform an electrical to optical conversion, which means the signal is electrical at first, and it will become a wavelength, which is a stream of light. And on top of that, in a single fiber, we can have usually up to 88 wavelengths,
03:42
so 88 stream of lights, and this is a technology we call Wavelength Division Multiplexing, which is used to have very high bandwidth capacity. When we buy network devices from a vendor like Cisco, Nokia, Huawei, they usually provide
04:02
what we call a network management system. It's like a software or a web application that can be used for network visualization, inventory, provisioning, supervision, and sometimes automation as well. It's a network that has like an overview of the network of all
04:22
devices. If we want to visualize our network, the input file that we have is usually a network topology. It could be like a CSV file, Excel file, text file, basically it's a file that will contain the list of all nodes and the list of all links and the position
04:44
of the nodes as well. So if we want to draw this network, as you can see, if we draw like nodes at random positions, this doesn't yield a very good display, so we have to find other ways. We have basically two options. The first one is to use, the first one
05:05
is to draw the nodes at their physical location, where they are like physically located by using GPS coordinates. To do that, we use a type of software that we call GIS software,
05:21
which stands for geographic information system, and that allows us to, the idea is to draw maps and draw our network devices on this map. And the other option is to use a graph drawing algorithm. I will make a demonstration of both if it keeps working.
05:42
So what I'm going to show now is how we can create a simple GIS software in Python, so a software to draw maps. The first thing that we need is a grid programming framework. We have a lot of such framework in Python. We have PyGDK, PyQT, WX Python, PySide,
06:03
but the one I'm going to use is Teqinter. It's Python interface to TCL, and it says the only grid framework that is like part of the standard Python distributions, which means you don't have to pip install anything. It's by default, it's a built-in module.
06:23
Inside that framework, we will need a widget for drawing. All widgets have one. In Teqinter, it's called a canvas. It's basically a widget that allows you to draw circles, rectangles, but most importantly polygons. And we will use inside that widget a method to draw
06:44
a polygon. The reason for that is that a map is actually just a set of polygons. If you look at Italy on this slide, Italy is just like a polygon. It can be considered a polygon. So to draw these polygons, we need to have coordinates, and we will use
07:03
a special type of file, which is called a shapefile. A shapefile is a file that contains shapes, and the shapefile that I'm going to use contain two types of shapes, polygons and multi-polygons. These shapes are described like as a list of coordinates
07:26
as a list of points on the earth, and a point on the earth is defined as a longitude and a latitude. So the first library that we will use is called PySHP. It's simply a library
07:41
that allows us to draw, to extract the shapes from the shapefile. It contains a reader object that we can use to read the file. We can draw polygons with the create polygon method in Takeinter, but as for multi-polygons, we cannot draw them. We can
08:04
only draw polygons. So what we will need to do is to convert all the, a multi-polygon is actually just a set of polygons, and we will need to convert a multi-polygon into multiple polygons in order to be able to draw them. And to do that, we will use another
08:23
library called Shapely. When you use Shapely and you convert a shapefile multi-polygon into a Shapely multi-polygon, it will actually make the multi-polygon iterable,
08:40
which means it will have this double underscore iter function so that you can loop over the multi-polygon to yield all the polygons it contains. As I said, shapefiles contains coordinates defined as a longitude and a latitude, and if you look on this slide,
09:02
you will see that these are angles, but when we draw polygons in a software, in a pixel coordinates. So we will have to make some kind of conversion from angles to pixels.
09:23
This is actually a conversion from 3D to 2D because we have a point on the earth, and we need to have a point on a plan, and this is what we call a projection. So there is a mathematical theorem by Gauss that basically says that if you make such
09:43
a projection, like you have a sphere and you want to project this sphere on a map, it's called the remarkable theorem, then you will lose some information in the process. There will be what we call a distortion. It could be a distortion of distance, of
10:01
angles, of shapes, any distortion, but we will lose some information. For instance, if we consider the Mercator projection, it preserves angles, but it doesn't preserve areas. In order to make that projection, to convert 3D coordinates to 2D coordinates,
10:29
we will use another library called Pyproj. It's a library where you can choose a type of projection. It could be Mercator or it could be azimuthal orthographic, like you see on this slide, and we will convert a longitude and a latitude into pixel coordinates.
10:50
So if I sum up, we need a grid programming framework. We will use Teckinter. Inside this framework, we will need to use a widget to draw polygons, Teckinter's canvas. We
11:02
need a file that contains the coordinates of the map, a shapefile. We'll use PySHP to read the shapefile. We'll use Shapely to convert multiple polygons into polygons, and we'll use Pyproj to convert geographic coordinates into projected coordinates.
11:22
So okay, okay. Well, I will open Jupyter notebook to show you how it looks like. So okay, so well, this first code shows you how to initialize Teckinter main window
12:28
and how to initialize a canvas, and finally, how to use a create polygon method in order to create the polygons. If I start this code, I will get this, which is like a graphical
12:42
interface with a polygon. Finally, I have this snippet to show you how to use Pyproj. We need to initialize a projection. This EPSG code, like 33-1995, stands for the Mercator
13:00
projection. Then I have the longitude and latitude of Remini, and I can use the Mercator projection to convert longitude and latitude into projected coordinates. And I can use this inverse equal true to make the reverse operation, which is convert projected coordinates
13:21
into geographic coordinates. And if I run this code, I print the coordinates, we can see that we have the XY and longitude, latitude. Then I show you how to use the shapefile reader and how to import shapelyes. So I have this local path to a shapefile,
13:46
and I will start with a reader object. I will extract all the shapes from the shapefile, and when I loop over the shapes, I will convert all of them into shapelyes shapes so that multiple polygons are iterable, like this. You see that I have this hazard shape
14:14
magic iter, and it returns true, so all multiple are iterable. And finally, this
14:24
is like, well, the final code that shows you everything, how to start the kinter, how to give the local path to the shapefile. I have this two projected coordinates function in order to convert longitude and latitude into projected coordinates. I will read the
14:44
file, extract all its shapes, I will loop over the polygons, convert them into shapelye polygons, and finally I will use the create polygon function. If I execute this code, I get this. It's very simple, but it's working. And I can improve it a little
15:08
bit by, I can change the projection here to have an azimuthal autographic projection. I will create a circle with which radius is the radius of the earth, and I will use
15:27
the kinter file dialogue so that when you run the code, kinter will actually ask you to choose a file, and it will then draw it. So if I run this code, it asks me for a file, and if I draw it, I get this. So this is very easy to do in Python. As
15:48
you can see, it only takes like 30 lines of Python to get this result. So what I want to show now, if it works, no, it doesn't work. Okay. So I will
16:31
actually use this method in order to import a real network. I will import project. This network file contains a real network. It's actually one of the main French backbone.
16:46
You will see in a few minutes. Okay. So if I import this file, okay. So this is one
17:04
of the main French network. It is made of about 350 optical switches. In case you are interested, these optical switches are Alcatel-Lusend Photonic Service Switch or PSS1830.
17:20
So basically if you are in France and you are trying to use internet with your mobile phone or with your computer, well, there is a chance that the signal you send will actually be transported over this network. Of course, if I wanted to do just visualization, I could use frameworks like I could use a Google Map API, I could use GeoDjango, or
17:47
I could use D3 also if I wanted to do it in JavaScript. But the reason why I want to use an actual software is that I can select some nodes, I can move them, I can delete
18:06
them, I can create new nodes with a drag and drop system so I can like actually create a network, and I can look at the property of the node. For instance, if I put the
18:26
IP address here, okay, well, I can say like this node has this IP address and once this
18:40
is done, I can hopefully start a message connection to the device like this. Okay, so the device I'm using, you don't see it from there, but it's actually just here on the ground. It's a Cisco 1841. Well, it's back. Okay, so I'm connected to the router.
19:25
There is a few things that I want to show you. The name of the router, it's called router 2. It's what you see on the left part right here. So this is what we call the host name. If I want to change it, what I need to do is to enter the configuration
19:43
mode by writing configure terminal and then type host name and the host name I want, for instance, Remini. And when I do that, you see that the host name, the left part, is changed to Remini. So what I'm going to do now, well, the idea is that if you
20:03
have like 5,000 devices and you want to change the host name on all 5,000 devices, you cannot possibly log into all the devices like by hand and change the host name yourself. So you need to automate this process and we can do this by using Python. So I will
20:26
come back to my slide. Okay, so this is a slide, this is an overview of the network
21:03
automation landscape, which is all the methods you can use in order to automate the network. The first one is the one that network engineers have been using for many years. It's simply to connect to the device with a library such as Paramiko. Paramiko is a different
21:20
library in Python to start SSH connections, but Paramiko is very complicated to use. So network engineers have built other libraries on top of Paramiko in order to make it easier for network engineers to automate. The two most famous libraries are Netmiko
21:41
and XScript. Finally, you have also the NetConf and Yang solution. NetConf is a network configuration protocol and Yang is a modeling language that we use to shape the data that are sent over NetConf. This is standardized by the IETF. I don't want to spend too much
22:03
time on this one, but just so you know, there was a talk at the last PyCon US in Portland about how to use NetConf and Yang. Finally, you can use OSS protocols like TL1 and SNMP with Python library to SNMP, which is called PySNMP, but SNMP is usually very
22:25
complicated to use. You can also use configuration management tools. These tools originate from the DevOps world, but they can actually be used for network devices as well. So you have Puppet, Chef, SaltStack, HPNA, but you have also Ansible, which is the one
22:45
that is gaining momentum at the time, and it's an open source Python project. If it works, I will make a demonstration of how to use Netmiko, XScript, and Ansible in order to change the hostname. So I will come back to the code. I will just explain
23:08
what I would have done. What I wanted to do was to use XScript, Netmiko, and Ansible in order to show you how to do basic automation. I'm not sure I can still do it.
23:22
Okay, I'm still connected to the Cisco device. You can see the hostname is Remini, and if I use XScript, so the idea is that I must create an account object in order to give the credentials to connect with the device. I must create a host object to give him the IP address of the device, and finally, I will send all the commands required
23:45
to make that automation. So if I say hostname France, and I run this code in a new shell,
24:05
you see the hostname was changed to France without having to log into the device. So if we had, like, a lot of devices, what we would use is to multi-thread all the SSH connection in order to configure all the router at the same time. XScript
24:23
allows you to do this by, if you loop over the IP address and you create host for each IP address, then you append them to a host list that will contain all the device that you want to run the script on. Then XScript will actually use multiple
24:42
threads so that all SSH connections are done in parallel, and you have this max thread parameter here when you can say, I want at most five threads. You can do exactly the same thing with Netmiko. You have this connection parameters
25:00
dictionary that contains the credentials, like username, password, as well as the IP address of the device. You would use this connect handler to log into the device, and you will send all the commands you need to send. So, for instance, what I'm doing here is that I'm adding a description to the interface fasteternet0.0, and I'm
25:27
also storing the result of show running config, which is a configuration of the router, inside a config variable. So if I run this script, it will take a few seconds, okay.
25:52
If I print config, I see that config contains the configuration of the router, and if I
26:01
look at the interface, the configuration of the interface, I see that it has this description OSPF interface. That configuration was pushed with Netmiko. I can also do the same thing with Ansible by, I will start Cygwin. So the idea with
26:29
Ansible is that you have a file called inventory. If I look at this file, you see it contains the IP address of the device you want to send the script to, and you have
26:41
what we call playbooks. These are these file .YML files. You can look at one example here in order to do the same thing, which is changing the hostname. So I will first give the credential to Ansible, and I will then use the iOS config module,
27:02
and this lines hostname Paris will change the hostname to Paris. So I can do that using Cygwin by writing Ansible playbook. I tell him that the inventory file is inventory, and I change the hostname, okay. As you can see, it worked. The Ansible script
27:38
was sent to the device, and the hostname was changed.
27:40
What I can also do is use the iOS command Ansible module in order to send the show running config command, and then I will use copy to store this configuration in a file output.txt. I will also use Cygwin to send this one. So change hostname, save
28:05
running config, okay. And if I look at the folder, you can see that I have this new
28:24
output.txt file, and if I look at what's inside, it contains the configuration of my device. So, well, I had more things to show, but I think we can stop here. Thank you.
28:46
Thanks, Antoine, for the talk. Questions? When you want to use commands that will require confirmation, like, I don't know, reload
29:03
the router, probably will. Yes. Actually, with XScript, you should look at the documentation. I don't know if I have internet, but if you look at XScript documentation, you can tell when XScript should expect a prompt that it does not recognize.
29:23
For instance, if you connect to an Alcatel devices, sometimes it will ask you for yes or no, say yes to continue the login part, and you can tell XScript that if it sees yes, then it should understand that this is, that the device is expecting something, and that works perfectly. Hi. Maybe you have told us where have you
29:49
got the data for the geographic representation from? Where I took the shapefiles. Yeah, I mean the raw data, where did you
30:00
get it from? The shapefiles, you can find them online. I just googled shapefiles countries or shapefile continents about the data of the network. That's because, well, I work at the company that runs this network, so I
30:22
did an XML query to the SAM5620, which is a software that supervises a network in order to retrieve the topology. Hi.
30:42
What are the best practices to map a lot of devices?
31:16
Actually, the tool I made, I wanted to show it, but because of the technical issues,
31:21
I couldn't, can do that. That was my next demonstration. If you have several devices like this, what you could do is, what I did is providing a graphical interface to XScript so that you don't have to cut anything. It's all graphical. I can go to script creation,
31:41
and I can change the, like, make a script change host name, and then if I write configure terminal and then host name, you would like this part to be a variable, right? Is that right? So using this software, what you could do is like doing host name, which
32:06
means replace this part with the host name value of the property of the device, and you can save this script. Then if you look at the properties, it will look for the host name properties, and it will replace the value. And actually, the host name properties
32:25
does not exist yet. So what I did is a way for network engineers to create properties on the fly, even if it does not exist. So I have this, I have a file called change
32:41
host name, and it contains the value of the host name, and if I import this file like this, import host name, you can see that the property, well, it doesn't work here, but you can see that you have your new host name properties, and it actually
33:01
contains the value that is in the excel file, and then what you would do is select all three devices like this, and then you click on send a script, you will choose a script, and for each devices, the host name part would be replaced with the actual value of host name for the node, you know? This is using x script.
33:25
X script, but you could do the same using Netmiko, and you could do the same using Ansible as well. And if you wanted to do this like in pure Python, you would just need to have a dictionary that maps, well, the IP address of the device to the value of host name
33:42
or whatever you want, and that would work too. The question was if you use some bolt or some secret storage, something like that, it's actually just credentialed in customer environment, are they, you say, important? Yeah, no, I don't use any secret storage because it was just for a simple demonstration,
34:02
and well, it's not like, but yeah, you would need to encrypt the data in a real network. Okay, thanks. More questions? So is there an Ansible interface for PyNMS?
34:22
Is it what? Can you run Ansible scripts with like selecting a few hosts and right-click run script? Actually, I want to add that as a new feature, but right now, I'm just working on Ansible, trying to make useful Ansible scripts, and then when I get something that's working, I want to make PyNMS like as a graphical interface for Ansible so that you don't need
34:45
to actually write the YAML file, you can do it graphically as well. More questions? Okay, no questions? So let's thank Antoine again.
35:03
Thanks.