Reach for the Clouds With OpenSIPS 3.0
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47329 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Point cloudPresentation of a groupComputer animation
00:29
Core dumpSoftware developerSession Initiation ProtocolServer (computing)Configuration spaceProgramming languageModul <Datentyp>Computer architectureService (economics)Configuration spaceCursor (computers)Programming languageSession Initiation ProtocolModule (mathematics)TwitterWordServer (computing)BuildingInternettelefonieComputer animation
01:43
Modul <Datentyp>Presentation of a groupComputer animation
02:08
Human migrationPoint cloudSoftware testingMereologyHuman migrationInternet service providerSoftware developerSoftware testingMarginal distributionInstance (computer science)TwitterCycle (graph theory)Service (economics)Computer animation
03:02
Scripting languageSummierbarkeitSoftware developerOperator (mathematics)Goodness of fitBitProgramming languageProduct (business)Configuration spaceComputer animation
03:53
BitRoutingComputer animation
04:23
Scaling (geometry)Scale (map)Operator (mathematics)Scaling (geometry)HookingWorkloadSet (mathematics)Graph (mathematics)Computer hardwareNumberProcess (computing)Matching (graph theory)Computing platformThresholding (image processing)Line (geometry)Computer animation
05:41
Read-only memoryShared memoryBitStructural loadData structureFeedbackNumberPortable communications deviceMappingComputer animation
06:30
Scripting languageScripting languageSoftware developerComputer animation
07:05
Programming language
07:28
PreprocessorPhysical systemScripting languageBinary fileUDP <Protokoll>Computer fileError messageLine (geometry)ParsingDialectTraffic reportingComputer fileMathematicsLine (geometry)Sheaf (mathematics)Scripting languagePreprocessorComputer animation
08:53
Module (mathematics)Function (mathematics)ConsistencyBitIntegerFunctional (mathematics)ConsistencyModule (mathematics)String (computer science)Variable (mathematics)Computer animation
09:56
Read-only memorySoftware testingResource allocationCompilerMathematicsSession Initiation ProtocolResource allocationOpen setSemiconductor memorySoftware testingComputer hardwareIntegrated development environmentBitComputer animation
10:52
Information managementInterface (computing)Control flowInformationParameter (computer programming)StatisticsScripting languageSession Initiation ProtocolVideo game consoleInterface (computing)Open setDatabaseUser interfaceInformation managementInformationComputer animation
11:57
Scripting languageInformation managementInterface (computing)Control flowInformationParameter (computer programming)StatisticsCommon Language InfrastructureVideo game consoleFunction (mathematics)Open sourceLine (geometry)Interactive televisionOpen setSession Initiation ProtocolModule (mathematics)Flow separationCartesian coordinate systemEntire functionInstance (computer science)Software frameworkQuicksortInterface (computing)Computer animation
13:12
Information managementComplete metric spaceComputer configurationDifferent (Kate Ryan album)Function (mathematics)File formatDatabaseGroup actionSession Initiation ProtocolOpen setLine (geometry)Function (mathematics)Instance (computer science)Configuration spaceComputer animation
14:12
Configuration spaceBackupInstance (computer science)Interface (computing)Information managementInformationUtility softwarePublic key certificateTransport Layer SecurityHuman migrationDatabaseModul <Datentyp>Interface (computing)BackupComputer fileModule (mathematics)Instance (computer science)Session Initiation ProtocolScripting languageConfiguration spaceVirtual machineOpen setConnected spaceComputer animation
15:34
Channel capacityInformationQuery languageRead-only memoryProcess (computing)Session Initiation ProtocolCommon Language InfrastructureModule (mathematics)Thresholding (image processing)Time domainSynchronizationParameter (computer programming)Uniform resource locatorDifferent (Kate Ryan album)Structural loadModule (mathematics)Asynchronous Transfer ModeDirect numerical simulationProcess (computing)Server (computing)Multiplication signSession Initiation ProtocolQuery languageMathematical analysisTable (information)InformationComputing platformOpen setSemiconductor memoryTracing (software)Computer animation
17:28
Session Initiation ProtocolVideo trackingPlastikkarteEntire functionInformationMessage passingBlogSystem callBitReal-time operating systemSession Initiation ProtocolComputer animation
17:56
ArchitectureSession Initiation ProtocolSystem callLetterpress printingVideo game consoleStructural loadLatent heatComputer animation
19:06
Computer animation
19:28
Computer animationLecture/Conference
19:50
MaizeRevision controlNetwork socketFunctional (mathematics)Analytic continuationImplementationMereologyConfiguration spaceComputer animationLecture/Conference
21:11
SynchronizationCodeFunctional (mathematics)SynchronizationComputer fileModule (mathematics)Control flowField (computer science)Descriptive statistics
22:06
SynchronizationCodeParameter (computer programming)System callSession Initiation ProtocolFunctional (mathematics)Computer filePointer (computer programming)Scripting language
23:28
SynchronizationParameter (computer programming)Functional (mathematics)LogicSource code
24:03
Variable (mathematics)Validity (statistics)Computer fileScripting language
24:48
Reading (process)BlogError messageConfiguration spaceMessage passingScripting languageMultiplication signEndliche ModelltheorieSemiconductor memoryPointer (computer programming)
26:25
Software testingNumberScripting languageMultiplication signComputer animation
27:43
Point cloudOpen source
Transcript: English(auto-generated)
00:05
Thank you, Saul, and thank you everyone for staying up until this last presentation. As Saul said, we are going to be actually three speakers and Mia Razvan will be talking about what are the latest features in the stable 3.0 release and Peter will be giving us some practical insights about the
00:25
asynchronous API. If you don't know us, my name is Liviu Kirkou. This is Razvan Krajna. Together we have like 18 years of experience in working with OpenSIPs and
00:42
you can reach us out at this is an interesting thing, so my cursor is moving, okay, at those tags over there on Twitter and IRC and from looking at you, how many of you know about OpenSIPs or what it is and
01:03
Okay, so that's pretty much everyone, but for those of you at home that don't know much about it, a couple words. It's a highly customizable SIP server. It's very performant. It has a multi-process architecture and a custom configuration language
01:25
which allows you to quickly build SIP and VoIP services and a ton of modules that back it up. So you can do a lot of cool stuff with it. Actually, there's a and it's of course all of it is open-sourced and free to use.
01:44
A lot of it is also available in this nice presentation that Bogdan, the OpenSIPs founder, gave at Astricon a couple of years ago. Just search on YouTube OpenSIPs Swiss Army Knife and that's all. You'll learn everything in just 40 minutes.
02:02
So back to today's topic about the philosophy and features behind 3.0, we noticed that there are some trends in the RTC industry such as migration to cloud-based deployments, whereas the service provider want to optimize their costs and
02:27
improve their margins. Everyone wants to have this rapid development cycle and testing of course. They have the CICD pipelines and whatnot. Fully automated deployment, of course. You don't SSH into instances anymore. I mean, I guess you sometimes do but
02:48
ideally you'd want this to be fully automated. So you can do the last part that I've highlighted there such that you ensure better SLAs and have a better reputation
03:00
and be a better player. So what does all of this sum up into? The development side, the ensuring good quality, and of course operating this whole thing, DevOps. And that was what we decided to focus on in 3.0. And I'm going to, we did a little bit of each. So if we talk about operating OpenSIPs,
03:26
a few notable features will be, so the first one would be the ability to reload the configuration language. So it's finally here. This was something that was waiting, I guess, for too many years and we decided to take some time and do it and
03:43
stop doing cool SIP features and focus a bit on productivity and making things easier for our users. So you pretty much, this is all you have to do. Just send in my reload command and you should be good to go.
04:00
Just a small mention here, it will only reload the routes. So if you have like to load a new module or change a module parameter, you will still need to restart it. But we feel that just reloading the routes should cover up for 90% of needs to restart. For example, when you need to change some logging line or
04:20
change a bit of route here and there, a new operating related feature that we introduced is the ability for it to auto scale. So there was always this problem that you didn't know how many processes you should configure it with. And there was no really good answer to it because it depends on what kind of workloads you have, what hardware you have, whatnot.
04:46
And so now you can just specify like a top threshold and a bottom threshold. So for example here, it will grow up to 25 processes. So it will keep on forking, but it also knows how to un-fork them. So it
05:01
cuts down on your costs, maybe during the night or when you have less workload on your platform. You just hook it into the timer settings or the UDP listener and that's pretty much it. Here's a graph of how it works. For example, you can see here with a blue line that we threw like 400 CPS at it with CIP and
05:26
the matching number of, or the matching graph shows the number of processes going up and towards the end we cut down on the traffic. So the process is un-forked. Okay.
05:41
Another, this is a bit of an experiment. We don't, at least Rezvan says that it works well. We haven't yet played with it a lot. The persistent memory, which is a really nice way of reloading and restarting your instances, even if you may have like hundreds of millions of data. For example here,
06:03
let's say you have like a number portability database, right, of 16 gigabytes and you want to load it from MySQL. Well, that thing will take up like 60, 15 minutes or something. So that's what we used to have, but now with the magic that Rezvan did with mapping this to a file,
06:21
everything loads instantly and the data structures are getting rebuilt right on after the restart. So yeah, play with that and let us know how it worked for you. I'm sure Rezvan would love some feedback on it. Moving on to script development and how we try to improve the experience over there for our users.
06:47
I would like to ask a quick question. How many of you have heard of at least one of these tools? Ansible, Puppet, Chef, or Salt? Okay, so that's like more than half the room. So we, you know, got up to speed with those as well and
07:03
because we had this idea that, okay, we have templating, right? These commits were done in 2013. We have M4, everyone should be happy, right? But it turns out that not everyone uses M4 anymore. People evolve and the people started using, you know, maybe embedded Ruby or Jinja or other templating languages that
07:25
are easier for them to manage and previously we were pretty much, they were stuck with M4. They had no package of support if they wanted to use their custom preprocessors and let alone the init scripts and
07:42
the small problem was also that you had to manage two files, the M4 file and the CFG file. So it happened, I'm pretty sure it happened to you as well where you would make a change in the CFG, you restart and then it gets regenerated and overwritten. So you lose the change because you had to write in the other file instead
08:01
and so that problem is gone as well now. And also there was this idea that it reported in one file so that if you had a typo or some error, right, at line 2227 in the CFG that would actually end, it would look like this. So some line, right? When you go back to the generator file, the template file, that would be in a whole different place because it would get the
08:25
it wouldn't have the ability to match the proper line. So even this is fixed now. So now all of the, if you use the new dash P option, the preprocessor, the generic preprocessor, now all the lines will point to the template file. So this is pretty cool.
08:46
There's a lot of stuff on the website on this. There's a new section templating CFG files. You can read more about those and this one was a bit, a bit of a quirk that we took care of.
09:01
Actually if you, if you look at what we had before, we had this strange problem that some of the module functions accepted both strings and variables whereas some, like NAT-VAC test, didn't. They only accepted static strings. But notice how I, even I call them strings, but these are actually integers.
09:24
So another question is why the hell do you quote integers? That's another thing that nobody dared to ask up until now. Okay, we took care of all of this. So take a look at how things look like now, right? An integer is an integer, no quotes. And any function accepts any kind of input,
09:40
consistency. So we feel that this will speed up the development and make it easier for new people to pick up open sips and get up to speed with it and start building services with it. This is another example of function that we normalized and going into the QA side of things, we also made some changes there. If you're familiar with how open sips works,
10:04
are you, do you know the three allocators of open sips or you do? Okay, so not that many people. No, but I'm cool. So we basically have three ways of you to configure open sips to manage its memory. One of it, the default, it's fast and good.
10:21
The other one is for QA and testing and the other one is for multi-core hardware and to be used in that kind of environment. And those are all at, those are all built in to the binary now. You don't need to recompile with each one. So you, we have that now where we've added these switches and you can just pick between
10:42
between these at startup. Really convenient. So I'm going to let Rusvan talk a bit about how you can interact with open sips. Thank you, Livio. Okay, so there are a few ways of interacting with open sips and probably all of them rely on the fact that open sips provides a management interface where you can query or you can trigger
11:08
commands to it. And there are several, there were several tools that could use that management interface and make things more fancier. One of them was, is open sips control panel, which is a PHP web interface
11:25
that can be used to both provision data into database and trigger reloads to open sips or fetch information from open sips. Another one was open sips CTL, which are basically some bash scripts, which allow you to do more or less the same thing.
11:42
And there was another project, not sure how many of you ever used it. I haven't. The open sips console, which is, which was a Perl console that you could use to interact with open sips. What we've done was to remove these two
12:01
and replace them with the new open sips CLI. Open sips CLI is a Python module and a command that can be used to do basically the same thing. It, it aims to replace the old open sips CTL and open sips console and provides a single package that can be used to,
12:21
to control and interact with several open sips instances from a single application. It is written in Python and it provides the open sips CLI module. This means that you can use an, you can only use, write your own Python application and import the entire open CCLI
12:45
framework by using, by writing use open sips CLI. And from that point, you can trigger MI commands and all sorts of, all sorts of interesting commands to open sips. So it's very, very easy to use and to extend. It has a modular interface. Again, all you can do is write the file,
13:05
define the method, and you will have your own commands in open sips CLI that can, can trigger open sips different, different actions. It is highly configurable. As I said,
13:21
has multiple instances. You can use a tab auto-completion to find out all the commands that open sips supports. It has commands history. It has a lot of documentation. And a nice thing that, or a problem that we had with open sips CTL was that the output was not structured. This means that it was somehow very hard to parse the output of open sips CTL
13:46
because it was always line oriented and you had to, to split based on different characters that can be duplicated and so forth and so on. Now we have a standardized output that can also be line oriented if you're just interested in dumping something at the console,
14:04
but you can get them as a JSON or a YAML that you can easily parse in your application and do automated, automated jobs with it. The configuration is very, very simple. It's basically a YAML configuration file and there are several paths that open sips automatically searches
14:24
for the config. When it finds one, it uses that configuration. Instances can be specified in those configuration files. So for, in this example, we have, we have here two different instances, one, a primary machine and the backup machine. Each of them have a different URL prompt
14:45
so that you can know on which instance you run. And basically all you have to do is specify open sips CLI minus I primary, and it will start a connection to the primary interface. Then you can switch to the backup interface and then you'll have a new prompt with the,
15:02
for the backup machine. So basically this is how you switch between different interfaces. Of course you can have a script that, to run commands on all those interfaces. It's quite easy to do. There are several modules already built in. Most of them are, were already in OpenSIP CTL, so I'm not going to talk that much about them. Two of them that are,
15:25
sorry, that are of interest for us are these two, the diagnose and the trace, which have been added in OpenSIP 3.0. The self-diagnosed tool basically is used to, to debug OpenSIPs. So as you all can, as you all know, probably OpenSIPs has different
15:46
subsystems like UDP processing, TLS workers, shared memory, DNS queries, SQL. All of these might, might be working or not, depending on different parameters or, or depending on different stuff.
16:01
What this diagnose tool does basically helps you pinpoint whenever there's a problem in your platform. So here, for example, we run the diagnose module and find out that workers are OK. However, shared memory is in critical mode. So if you want to find out more information or extended information, you'll have to run diagnose memory to get more detailed information
16:25
about this specific subsystem that is not working OK. Same for private. C processing is OK. However, DNS queries are, are a bit, are critical. And when you go and do diagnose DNS, you basically see what are the slowest queries, what are constantly slow queries. So depending on,
16:45
on your, on the servers that you need to resolve, how many, how many, how much time do you, do you lose for each query and how many times do you run it? So it's basically an extended analysis of how your C processing works. The same thing is for SQL where you can have
17:05
different commands and diagnose those. So for example, you can find out how much time you lose querying open, open sees that dialogue, the dialogue table from open seats or location or whatever. Diagnose memory as well. Here you can find out how memory is going, diagnose load
17:28
and so forth. A similar, a nice, very nice tool that we added was the SIP tracing. Basically, it's an end grip or a SIP grip for real time traffic. It's basically, we call it a smart filtering because for example, if you want to track down a particular call, you'll have to do
17:45
an end grip and then put out the calls that you are not interested. And if you have like thousands of calls, that might be a bit hard to do. However, with the tracing, all you have to do is specify a filter. How it works, basically in OpenSIP CLI,
18:03
it will declare a HEP listener because everything relies on HEP and instruct OpenSIPs subscribe for the tracing command. And then it will get all the data and print it at the console. So basically you will find something similar to SIP grip, but only for a particular
18:23
call. And you do this for very high, very loaded instances, OpenSIPs instances. So it's very easy to track down only specific calls or calls from a specific IP. As you can see,
18:43
usage is very simple. What we are doing here is trace all the calls from Alice or trace all the calls from Bob that come from this specific IP. So that's what you will see at your console. These are the resources for the new OpenSIPs CLI and all the stuff that
19:06
me and Liviu have talked about. If you want to find out more, just come to the OpenSIP Summit. That's going to be held in between 5th and 8th of May. So probably I'll see most of you there, I'm pretty sure. Now I will pass, Peter, a few. I know we are a bit behind schedule, but
19:24
we're the last one, so hopefully we'll be able to. Peter is going to present his experience with OpenSIPs 3.0. So go ahead. Hello all, my name is Peter Lemingoff. First of all, thanks for letting me participate. Yeah, now I can, but okay. So this is actually not really
19:46
a new feature, but I feel like it's really misrepresented. It desires a lot more coverage when people think. So what is it? Since maybe version 2.1, there is a reactor within the OpenSIPs kernel,
20:04
which actually you may push socket in, and since this socket has data, it will kick in and call some given function. It's really not, as we feel, it's not really widely used,
20:25
and not quite well documented. And although it's available since 2.1, I will talk about version 3.0 in continuation with two previous speakers. I really don't go into details, so if you want
20:46
to go into details, come to the OpenSIPs Summit, if I may, and you will hear much more about that from mechanic. So as I said, it's not really widely used, so if you want to try it,
21:02
so first you have to implement it. Implementation contains in two parts. One is in the module, and another one is in the config script. So in the module, you just need to fill up an extra field in the module description when you describe the function. I strongly advise to use
21:22
the same names for synchronous and asynchronous functions, just for simplifying things later. And you have to define a function in the same way as your typical synchronous functions,
21:42
just with one smaller addition, a control structure which has to be filled right before the function returns. And it's quite easy, you have just to pass a callback function, and the file descriptor you have to, you want to read from, you want to wait for the data to
22:06
read from. So that's basically it, basically all, but it should be, the callback function is also quite straightforward. It just reads, it requires three arguments. One is file descriptor,
22:20
another one is SIP message, which is processed currently, and the third one is arbitrary structure, pointer to arbitrary structure, you have to fill it up. In the config, everything is much more interesting. First of all, it might be counterintuitive and not really easy to
22:45
understand, but the execution of script, well, first of all, how to implement it? Since I advised you to use the same names as in synchronous and asynchronous functions, so all you need is just to
23:00
add the next, put it into the, put the name of your previous asynchronous functions as an argument to the async call and provide the next argument as a road, I'll talk about later. And as soon as this async call is called, all executions of the current script
23:21
will be stopped, and as well as execution of all other roads, which call this particular script. Again, this might not be, this might be counterintuitive, so this could be a problem. And the second argument is a road which will be called right after the callback function will be executed.
23:43
So this is where all you, you will put all the data which, all the logic which was skipped right after the calling this asynchronous road. And that's all part. That was the advertising part. This is how it's supposed to work. However, it's not that really well
24:04
documented, so during the implementation, we are facing some strange, some strangely looking bugs. Well, it's not bugs, but maybe discoveries. We have to go back to the files dating back from
24:22
2003 maybe, some global functions, global variables and that stuff. Turn it out, well, it is, it's quite obvious what asynchronous things are quite different from synchronous.
24:40
They're quite more demanding. And in this case, it's very, in this case, it's very much demanding in, to the quality of your script, to the overall validity of the script. For example, we've got a message which nobody really was inclined to fix about,
25:02
like, although it was written with big capital letters like scripting bug, everything worked even with this message, so why should we fix it? Maybe next time. And this next time was, it was really, was really for a while. As soon as we implemented this asynchronous function,
25:23
which, by the way, didn't took a long implementation, the model took about maybe eight hours, maybe four hours we spent on debugging, like memory leaks, no pointer and all that stuff. And the rest, the rest of the time was
25:43
quite a lot. I'm too shy to say that because it was really a horrible story. It was spent on the config debugging. I must admit that OpenSips has unique features to debug execution of the
26:06
interpreter, and it's really helped. However, it turned out that you have really to remove all warnings, all errors from the scripting, from your script. Otherwise, you will face
26:21
very strange looking things. Basically all. To sum up things, I would say that if you want to use it, really go and use it because it's really, really interesting. You don't have numbers right now. It's very alive, very hot feature. There are some of
26:44
the things we did just yesterday, so it's, we don't have really something to share related to numbers. But it looks very, very promising. So go and use it, but be very, very careful with implementation and with the scripting because it's really demanding, let's say that. So
27:06
that's basically all. Thank you guys. I guess there's no time for questions. We already took too much. No, there is. Are there any questions then? This is our last talk, so
27:26
we have all the time for questions. Does anybody want to ask anything? And then see you in Amsterdam. Bye everybody. Thanks for coming.