Flow-based programming for heterogeneous systems
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32525 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201488 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
Internet der DingeIntuitionistische LogikMereologySystem programmingComputer programmingSinc functionDemo (music)Goodness of fitLecture/ConferenceMeeting/Interview
01:10
Graphische ProgrammierungMathematicsConnected spaceFundamental theorem of algebraSystem programmingDifferent (Kate Ryan album)Vertex (graph theory)DataflowUser interfaceComputerType theoryProgramming languageStapeldateiComponent-based software engineeringInstance (computer science)Game controllerSoftware developerMathematical analysisServer (computing)MereologyCASE <Informatik>State of matterParameter (computer programming)Problemorientierte ProgrammierspracheJava appletObject-oriented programmingElement (mathematics)Process (computing)Constraint (mathematics)Matching (graph theory)Field (computer science)MicrocontrollerComputer programmingCodeRun time (program lifecycle phase)Scripting languageStandard deviationGraph (mathematics)Demo (music)Integrated development environmentFunctional programmingElectric generatorImplementationNeuroinformatikBitSource code
06:43
Data analysisCASE <Informatik>SoftwareServer (computing)Web 2.0BitRouter (computing)System programmingNeuroinformatikCombinational logicMicrocontrollerDegree (graph theory)Element (mathematics)Volume (thermodynamics)Message passingGoodness of fitMathematical analysisPRINCE2Game controllerSource code
09:23
Client (computing)Matching (graph theory)Digital electronicsWeb 2.0Software bugMessage passingConnected spaceMereologyNichtlineares GleichungssystemServer (computing)TelecommunicationFunctional programmingTime zoneSoftwareCASE <Informatik>Sinc functionMathematicsComputer iconSoftware frameworkConfiguration spaceIntrusion detection systemComputer programming1 (number)Program flowchart
12:41
Connected spaceBitObject-oriented programmingDifferent (Kate Ryan album)Line (geometry)Database normalizationComputer programmingDescriptive statisticsHystereseMicrocontrollerComponent-based software engineeringDigitizingMessage passingGame controllerInverter (logic gate)Function (mathematics)Server (computing)Boolean algebraComputer fileLogic programmingTelecommunicationMereology2 (number)Run time (program lifecycle phase)CASE <Informatik>Term (mathematics)Stress (mechanics)Graph (mathematics)Lecture/Conference
16:45
Game controllerDemo (music)Client (computing)Server (computing)CASE <Informatik>Web 2.0Sound effectVisualization (computer graphics)Computer programmingoutputInstance (computer science)Right angleMicrocontrollerDigitizingComputer clusterComponent-based software engineeringProgram flowchart
19:09
Server (computing)System programmingWeb browserFunction (mathematics)CuboidComputer programmingLevel (video gaming)MicrocontrollerProof theoryComponent-based software engineeringParameter (computer programming)Right angleJSON
20:32
Server (computing)System programmingFunction (mathematics)Component-based software engineeringSoftware maintenanceVirtual machineMultiplication signGame controllerFluid staticsCategory of beingComputer programmingTime zoneMathematicsVideoconferencingTerm (mathematics)Client (computing)Service (economics)Serial portoutputRun time (program lifecycle phase)State observerRepresentation (politics)MicrocontrollerInstance (computer science)Process (computing)Intrusion detection systemProper mapOpen setDemo (music)Different (Kate Ryan album)CASE <Informatik>Machine visionSource codeJSON
22:37
Virtual machineGame controllerCASE <Informatik>Computing platformProcess (computing)VideoconferencingVideo gameSystem programmingServer (computing)BitSoftware frameworkComputer programmingMultiplication signCellular automatonTwitterService (economics)CodeGaussian eliminationRun time (program lifecycle phase)Total S.A.Existential quantificationConfiguration spaceInternet forumFocus (optics)Level (video gaming)Communications protocolDataflowCondition numberSerial portNeuroinformatikInteractive televisionArithmetic meanMicrocontrollerType theoryVisualization (computer graphics)Integrated development environmentRing (mathematics)Software testingPower (physics)Reading (process)Instance (computer science)CodeRange (statistics)Graph coloringControl flowComponent-based software engineeringSemiconductor memoryCartesian coordinate systemTelecommunicationTransportation theory (mathematics)Object-oriented programmingNormal (geometry)Fiber bundleRouter (computing)Client (computing)Vertex (graph theory)Texture mappingBootingProgrammer (hardware)Web browserMobile appSpeech synthesisSoftwareFunctional programmingLecture/Conference
Transcript: English(auto-generated)
00:05
So those of you who joined us in the last few minutes, my name's Peter, I'll be the host for most of the day. This is the Dev Room for Internet of Things. Thank you very much for coming. Thanks to the team for organizing this, helping with everything.
00:20
Our first speaker is Jon Nodby from Norway, who will talk about this and show this, and we will have 25 minutes for every session. Questions as part of that, and if you have stuff you want to talk about, think about it much later. This afternoon, we have an open session, two hours where we can talk about different things.
00:42
So I'd like to give Jon, since you're the first speaker, extra two minutes, and we can start. Thank you very much. Everyone, a big hand of applause for Jon. Thank you very much. Thank you, Peter. Good morning. So I'm going to try to do a demo a bit later, but let's start with the introductions first. My name is Jon Nodby.
01:01
I'm from Norway, as I said, and I'm here to talk about flow-based programming in heterogeneous systems or for heterogeneous systems, and I'm lucky. So first, I'll introduce flow-based programming. Does anyone have a concept
01:20
and idea what flow-based programming is? I've heard about it before. There are a few hands, but not many. So flow-based programming is a data flow-oriented way of programming, where the key things you have are components, which are isolated pieces of functionality,
01:46
perhaps stateless, perhaps not, that you connect with other components through ports. So it can look something like this. This is a NoFlow Jekyll, which is a port of Jekyll to flow-based programming environment,
02:01
and this is running then on Node.js JavaScript. Important thing to note with flow-based programming, at least the NoFlow and Microflow implementations is that there is no code generation. This is not something that you put in something, and you'd get out something else, and you have to maintain that something else, and no way of going back and forth. It is a proper runtime,
02:21
even on the microcontroller. So you can introspect it. There is APIs for it, and you can do interesting stuff. So another thing, flow-based programming
02:40
is not necessarily visual programming. It lends itself very nicely to visual programming, but you can program it in other ways. This is a fpp or .fpp, which is a domain-specific language, which basically describes the component.
03:06
Yeah. So this describes the graph of nodes and connections using domain-specific language. So it's just quite natural, fairly humanly readable. There are two key things here.
03:21
The first you see in the corner where you have an instance name and a component name. That's the instantiation of a node, and then the out dash trigger that is a connection. Those are the two fundamental things. There's also literals, so-called IIPs, which are static data.
03:41
That data is not very different from a connection. It's just that it doesn't change. You can, for any of those, also connect real data and the changes. Both Microflow and NoFlow have embedding APIs. So you can use it for part of your program if you so choose. Maybe you have something that you feel is very, would fit Dataflow very well,
04:01
the other stuff you don't think is necessarily so. You can embed one of those and use them. So what is the combination? Why? So nowadays, we have a lot of different computing systems.
04:23
Maybe especially in the inner of things, we have different types of computing systems that we want to integrate together. So the case here is very trivial. It's just a heating system which you have a UI for and has a server for doing logging and potentially analysis and so on,
04:42
and there's a microcontroller actually controlling a heating element. Typically, you would program the user interface. If it's native, you would do like Objective-C plus Java, plus C-sharp or something.
05:00
Maybe three different technologies already there. Or you could use JavaScript and HTML, which might give you cross-platform solutions for the UI bits. On the server side, people use things like Python, Lua, Ruby, which has different languages, the different tools, use different IDEs, and some of the ways of thinking are different as well.
05:24
On the microcontroller, it's even more different. You typically use C or C++. You don't really have the opportunity to use JavaScript or so on and probably never will on a really small microcontroller. So how can we bridge these things together? If you want to have small development team,
05:41
maybe you're just one person, maybe it's just you. You only want to learn that and that, and then also take care of all the different constraints. I mean, you want the fluent UI, which is very nice and smooth and looks pretty. On the other hand, you want to secure your server systems and you
06:01
want maybe real-time sensitive microcontroller. So the idea here is to use flow-based programming as a fairly universal metaphor, because the components can be backed by anything. In the case of microflow, which runs on the Arbinos here,
06:21
the components are C++. In the case of stuff you're running in the UI, the components are JavaScript, HTML, standard DOM, inflation stuff. But there's a common way of accessing them. Yeah. So I think I'll just jump into
06:41
the demo and hope that things are still running. So just the UI in this case is,
07:04
I've broken the demo already. Anyway, I have a piece of UI which just shows the temperature of my heating system. That's just the most trivial bit that you would ever have. So let me try to heat the thing. It's sensing. Hopefully, it has changed.
07:24
Yes. If I heat a bit more. Well, then the heater turned off because well, apparently 28 degrees is a good indoors temperature. You don't need to heat more than that. So this is then a combination of a Web UI,
07:44
and then there is a, show a terminal, the beauty.
08:10
So let me just pop up. So the server in this case only does logging. So but the server could, which is on my computer in this case, would maybe more be on a router or
08:21
a Raspberry Pi or something in a realistic setup. That's something that I mean, it can do the things that a microcontroller cannot. A microcontroller can easily control the heating elements and so on. For instance, you're going to do data analysis of the stuff that you've collected, you might want to have a server. If you just want to store the stuff,
08:40
you might want to have server as well. So the server I'll just show you.
09:05
So it's just putting out a log of all those data. So it's just really trivial just to have all the pieces involved that you might want to have in a more realistic case. So now for the software where things get interesting.
09:23
Put away the magic. So the UI software. There we go. So this is what makes the UI. This is a full base programming of this trivial thing that
09:42
you saw that just lists the temperature. So it communicates to the server via WebSocket. So I'm just specifying the URL to connect to. I connect and I listen for messages. I don't really do anything with
10:00
the same message because I just respond to the temperature changes. On temperature change, I use a selector for getting the DOM element, which I want to manipulate, and I just write that. So that is how the value is of it.
10:21
So this is the UI, and here you can do crazy stuff. The IDE that you see is written using the same tools and the same framework. So you can do quite a few things. This is the server, and there's a bug so there is no icons. So this one has a WebSocket server configured here.
10:46
So you see this is the port that matches what I have on the client side, and it just listens for connections. These are for requests. We don't do anything with HTTP requests, so we just drop them on the floor.
11:00
The connections, we again listen to messages too, and this is just for debug. The same connection, so the client connects, I get that in here, and then I pass that to send message so that I can send the message back using the same connection.
11:21
Here is the five things. So this is the ones you see up here. Those are part of the same functionality. So together, they form the handling of WebSocket communication.
11:45
So the server acts as the actual server, but it gets the configuration from kick, which just starts the whole thing, and sends its configuration on, and then the server will give you connections. When one client comes along, you get one connection. That goes on to listen to messages.
12:13
I have constructed a web server using them. The question was, are these five pieces,
12:25
like are they a web server already, or have I made a web server with these things? If I understand the question correctly. So they handle separate parts of the same problem. For instance, if I wanted to handle different clients differently, I would need to do something else than just
12:42
pass all the connections on. I might want to say, okay, if you're on this host or that host, then I'll do something different with that connection object that gets passed along. In this case, I'm just treating everything the same. That's why it looks a bit redundant because it's just a straight line.
13:05
The other part here is communication with a microcontroller. This is where things get a bit more interesting. This component allows me to take a graph, which is a description of the program that's running on the microcontroller.
13:22
This is what's running on the microcontroller. I can put that in here so that the server actually programs the microcontroller as it should be, in this case, just be a thermostat, and then it will give me its output here. So that's how I am retrieving the temperature data from the microcontroller.
13:42
That is then, it's sent to the UI through WebSocket, just dumping it to the terminal, and I'm appending it to the file. So that's the log. So the last piece is what's actually running on the microcontroller. It's same principles again.
14:03
This is maybe a simpler program to explain because it's very simple logically. I have a timer component which fires a message every whatever it's set to, in this case, 1,000 milliseconds, one second. When that happens or when that sends a message on,
14:21
that triggers the next component. It reads the temperature from this digital temperature sensor. It goes through a hysteresis latch so that I can have a different threshold for when to activate the cooler than to drop it, so that if the temperature fluctates around something and it won't go on-off, on-off, on-off like crazy.
14:42
Because I wanted the output to be on when it's too cold, I had to put an invert Boolean in there, and then I just write that status to the output which is, in this case, the LED. That would normally be the relay for the output.
15:00
So how am I doing on time? So I'll just show a brief thing.
16:06
So this is the program I was going on on my controller. Did you see the values change? This is me observing the values of the data that's going through the program inside the microcontroller. So there's a runtime so I can introspect it. I can say, okay, what are the values of
16:21
this connection between the temperature and the latch? So I want to know if it's working correctly or not. These down here are the outputs that are here. So I can determine, okay, is this address is working correctly or not? I can introspect this visually,
16:40
which is quite a powerful tool.
17:02
How can you make back to send the data from the web server to the client side? So this component here,
17:22
that represents the microcontroller program in the server. Yeah. So the question is, if the Arduino residents are now the port, right now, there's a hack.
17:43
This, I knew the program and I'm using the introspection facilities that I showed before to retrieve the value. It will work differently quite soon. Go. Yes. So what I did.
18:21
Yes. That is possible to do. Sorry. The question was, the digital write, for instance, how it has an input and it has a side effect of being actually triggering the LED and then it passes its output on.
18:41
In this case, in the server, I'm hooking into this value, the temperature value. So this will work slightly differently soon. This is just for the demo. I'll talk about that now.
19:18
So this is very much a proof of concept right now.
19:21
It's very rough. I'm glad it worked at all. This concept of combining stuff that's running in the browser, on a server, on a microcontroller, and then thinking about systems that include all of them together is very much in the early stages for
19:43
for flow-based programming and Microflow and Noflow. So what we want to do now is to finish the Microflow components. So there was a way of getting a Microflow program that's running here and represented on the server. That should be done automatically. Right now, as I said, there was a hack.
20:01
And then we will allow you to expose ports and that will be a UI feature. So when you have this program, you should be able to say, okay, I'll draw a box on all of them and I want to expose, for instance, that temperature out to the outside world and maybe the output here
20:22
just to be able to verify it from a server side. And the components should automatically reflect the changes that you make in the program
20:42
that the component represents. That's currently not handle at all. It's very static. And this component that I have here, this, when I blow that up, I get that.
21:03
I should just be able to double-click and go there. Right now, you see there are different tabs in my IDE. They're actually different. They're actually different IDs. This is the same IDE but different instances. That should be more integrated.
21:20
Henry has made a public promise that this will work next week. And we also want to do the same kind of stuff for the bridging between servers and the client side. So now I did all this WebSocket handling myself, the filling and stuff. I didn't have to care that this was serial over USB to the microcontroller.
21:40
It should be the same between the client and the server. So I'll just be able to say, okay, I have a server. Let's communicate with it somehow. It has this program. This program has certain inputs and certain outputs. And I just want to be able to hook into those. And that's the vision for that. And here, this is very much a hierarchical system.
22:03
The UI's on top, and there's a server, and then there's a microcontroller. And, but we want to also think about having, for instance, a Python runtime, JavaScript runtime, bringing the same machine and collaborating as peers. That there are more equals and not just one is a slave over the other.
22:23
And that also ties in with specialized domains, for instance, audio-video processing and so on. And we want to do some proper case studies, not just silly demos. So that's what I have. I'm open for questions. And there will be stickers for those who give.
22:40
Yeah, go. That's our you. Let's get that. How do you deal with very slowly non-responding sensors to have a timeout value which you can catch and then retry or do whatever? So right now, there's no watchdog functionality or anything like that, but it's definitely
23:00
a natural thing to bake in the framework. You have a server which knows it's communicating a microcontroller, and it has even the knowledge of the software it's supposed to run on there. It should verify that it's responding as it should. And if not, it should kill it and restart it or whatever is serious as a fallback. Questions?
23:24
Could you tell me the footprint of the current prototype on the microcontroller? So the current microcontroller here is Arduino Nano, which is an Atmel Atmega328, the most common one. It's running with 500 bytes of SRAM
23:46
and something like 16K of program memory for the standard image. It can run on ATtiny, which has a total of 700 bytes
24:01
of SRAM or something like that. So it's pretty low. Questions? So is there a certain protocol you are using to talk to your Arduino or is it just low level
24:22
or you are reading just the serial data and transfer? So the communication with the Arduino here is serial over USB, but the protocol that I have on it is like on a separate level. So I consider this to be just one transport. For instance, I also have a simulator,
24:40
a runtime that can run microcontroller programs on my computer, and there I just have a transport which is just between JavaScript objects. So I will also have transports like UDP and probably other stuff, whatever makes sense. Bluetooth.
25:04
How open is NoFlow? Because I heard there was a Kickstarter for it, so is there a closed component to it or something? I'll just say one thing. All the code that you saw here is open. It's on GitHub now.
25:21
Yeah, so NoFlow is all MIT licensed, both the server side and the client side and the UI. And mainly the license, what we talked about in the Kickstarter is more for the convenience of not having to install it yourself. So you will get it as a service. But if you want, you can just install everything
25:41
on your own machinery and run it. So the host of service will be Flow Hub, which is. So any more questions? Come on, there's stickers. I have many. A little more.
26:01
Question two. Definitely worth exploring the range of use cases that you see this being applicable for. Because obviously there's microcontroller, but are there any other, you mentioned video and audio processing type stuff. So that's definitely more of a desktop type application. I'm kind of interested in mobile applications as well. Yeah, I mean, this was not a microcontroller.
26:23
This was a microcontroller, an embedded Linux machine, or well, my ordinary Linux machine. And you could run a Raspberry Pi or router. And browser. Those are the three platforms that we enable right now. So whatever you can do on your normal machine with Node.js, if you have a Node.js API for it,
26:43
you can put that in trivially. Places that this would be natural. The use cases that I see and I focus on are home automation, where the user might want
27:01
to configure some things themselves. So that programming is not something that only the programmer does. But there's a fluent kind of, you can move between looking at the UI to tweaking a bit, to hacking the system, to really making that thing.
27:23
That's home automation. The other things is things like interactive art. I'm working with some, some guys for that. So doing stuff that reacts in the environment, audiovisuals, that react to physical sensors. That maps very nicely to data flow, usually.
27:41
And then there's more the traditional embedded type things where you have a microcontroller that might control the power-up, or the boot-up of your real embedded Linux system. And I'm working there on trying to get much better automated testing. So the runtime I said about running my computer, that allows me to write tests
28:01
for the microcontroller program in JavaScript using mocha. Big hand of applause for John. Very nice. We actually have live video streaming of this whole thing.
28:24
We have a five minute break now and then we'll have our next speech. Thank you very much.