Brainwaves for Hackers 3.0
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 133 | |
| Anzahl der Teile | 169 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/21096 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
EuroPython 2016133 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
ComputerInterface <Schaltung>Verhandlungs-InformationssystemWellenlehreCOMStichprobeIndexberechnungProgrammbibliothekAnalysisStreaming <Kommunikationstechnik>Notebook-ComputerChi-Quadrat-VerteilungSummierbarkeitArchitektur <Informatik>ServerKernel <Informatik>BrowserKlasse <Mathematik>DatensichtgerätE-FunktionKorrelationZahlenbereichProzess <Informatik>OvalDickeDatensatzStatechartArithmetisches MittelJensen-MaßBinärdatenBetrag <Mathematik>Große VereinheitlichungBetafunktionLokales MinimumRechenwerkURNKorrelationsfunktionCoxeter-GruppeCodeSoftware Development KitStichprobenumfangDickeBildschirmfensterProgrammbibliothekVektorpotenzialFlächeninhaltWelleFormation <Mathematik>NeuroinformatikCliquenweiteInformationsqualitätAnaloge SignalverarbeitungDatenanalyseServerZweiProzess <Informatik>Notebook-ComputerKernel <Informatik>Kartesische KoordinatenBitComputerarchitekturStreaming <Kommunikationstechnik>TypentheorieBrowserDifferenteInformationKorrelationsfunktionAdressraumIterationCodeSchreib-Lese-KopfZellularer AutomatRechenschieberFunktionalFolge <Mathematik>Klasse <Mathematik>WidgetMultiplikationsoperatorProjektive EbeneStandardabweichungStrömungsrichtungFrequenzQuick-SortOpen SourceKonfiguration <Informatik>FirmwareZeitreihenanalyseDatensatzAnalysisJensen-MaßHumanoider RoboterApp <Programm>SummierbarkeitProtokoll <Datenverarbeitungssystem>CASE <Informatik>UmwandlungsenthalpieSoftwareMereologieNP-hartes ProblemAggregatzustandVariableGruppenoperationTermNummernsystemBetafunktionWellenpaketExogene VariableTouchscreenGüte der AnpassungSynchronisierungBitrateCharakteristisches PolynomHackerComputerunterstützte ÜbersetzungArithmetisches MittelZeitstempelIntegralNormalvektorPunktMathematische LogikEinflussgrößeHeuristikAbstandRechter WinkelPersonal Area NetworkInterface <Schaltung>FluidHilfesystemElektronische PublikationLeistung <Physik>InstantiierungDatensichtgerätDateiformatRahmenproblemCoxeter-GruppeVersionsverwaltungAuflösung <Mathematik>RauschenGamecontrollerTwitter <Softwareplattform>SchnittmengeDynamisches SystemPhysikalische TheorieRobotikPhysikalisches SystemMinimumWhiteboardSpieltheorieTabelleWarteschlangeGrenzschichtablösungDigitaltechnikCase-ModdingKonzentrizitätFehlermeldungNebenbedingungSoftwarewartungVerzeichnisdienstSchlussregelCodierungObjekt <Kategorie>DatenverwaltungAutorisierungProgrammierungIndexberechnungKontrollstrukturSkriptspracheBildgebendes VerfahrenDrahtloses lokales NetzFormale SpracheAutomatische IndexierungEndliche ModelltheorieMusterspracheDivergente ReihePhasenumwandlungBeobachtungsstudieMomentenproblemFunktion <Mathematik>EllipsePaarvergleichBenchmarkWechselsprungTopologieQuellcodeOffice-PaketEreignishorizontResultanteKoordinatenPartikelsystemZählenGraphSensitivitätsanalyseBridge <Kommunikationstechnik>EchtzeitsystemMathematikWorkstation <Musikinstrument>KongruenzuntergruppeComputersimulationTeilbarkeitMultiplikationt-TestSystemaufrufMechanismus-Design-TheorieSoftwaretestBildverstehenDatenstrukturTurnier <Mathematik>PlotterZeichenketteTreiber <Programm>Einfache GenauigkeitGeschwindigkeitElektronischer ProgrammführerSoftwareentwicklerVerkehrsinformationBildschirmmaskeRechenwerkWellenlehreVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:01
Hola, buenas dias, my name is Andreas Klosterman and this talk is called Brainways for Hackers. It's about using our Python tools to understand our brain a little closer. It's also in the sense of hackers as people who build stuff and not as hackers who try to hack your brain or something.
00:26
Well, please don't take anything of this as medical advice. This is just mainly a hobby project for me. It's no academic endeavor or something. I've given two talks previously like this.
00:44
If you have seen them, you probably don't. If you have seen them, I'll still show you something new, but if you haven't, you probably can follow quite nicely without it. First of all, what are brainwaves?
01:02
Brainwaves are created by your brain cells. You probably know your brain is composed of billions of brain cells and these communicate with each other via electrical signals. These signals sort of travel outwards through your skull and through your skin.
01:26
If you have a sensitive sensor, you can measure the waves at these locations. As this also already implies, these signals are very weak and they are very noisy.
01:43
You can only measure the summation of billions of cells at once, so you can't listen in on a specific brain cell or specific group of brain cells. That makes the signal analysis very challenging and it limits what we can get in terms of information.
02:09
So, what can we do with brainwave technology at all, especially as hobbyists? We can do neurofeedback training and I think that is a very big and very good area for devices, for consumer-grade devices.
02:30
It is about training your brain to attain certain mental states. The way it works is that we use brainwave data to assess, for example, how concentrated a user is.
02:44
Then, on the screen or via headphones, you give him a feedback about this mental state and he tries to increase it. By trying to increase it, he learns how to consciously control this brain state.
03:05
This kind of training has been proven quite successful in, for example, ADHD and epilepsy. If you want to use neurofeedback in a medical indication, please consult a doctor.
03:27
Now, the second thing you can do are simple neurologic experiments. I've done some correlation things and tried to figure out how especially the Muse brain band works.
03:43
There is another thing that I got to work with the newest guy, which is auditory evoked response potential. Since I didn't get it to work here without some serious yak shaving, I didn't bring it today and I won't even explain what it is.
04:06
Then there is this area of brain-computer interfaces, these BCI types of applications. They are a bit like the neurofeedback thing, but the goal is actually to control the computer or to control a robot or something just with your brain.
04:24
But it's very difficult and it's not really my main interest because it doesn't really work very reliably. You have to find a way to have some kind of button, for example, a button that you can press when you want to press it and not press it when you don't want to press it.
04:45
And that's kind of hard, especially with the devices I have. Some researchers had more success with implanting electrodes under the skull, not necessarily for this reason but rather for epilepsy.
05:02
And then they did BCI experiments and it worked very nicely, but please don't do that at home. Theoretically, you could use EEG devices like this for diagnostic purposes, but I would not believe that
05:20
it is legal to do this with these devices and probably they aren't really suited for this. Now the device I presented in my last few talks was the NeurSky MindWave headset. It's a Bluetooth connected device and it has one sensor in the front.
05:44
I have this device here and it has a sensor and you can wear it like this and it has a reference electrode and one channel. Inside this headset is a technology called an ASIC, an application specific integrated circuit and this can be used in other devices too.
06:11
For example, there are board games where you can control a ball moving up and down through your brain waves. There's also a headset that has cat ears mounted on servos and then these ears move according to how relaxed or concentrated you are.
06:31
Well, and there's also another headset which is a bit more expensive but is more comfortable to wear.
06:40
But it uses always the same protocol and the same characteristics. The second device is the Interaxon Muse. The Interaxon Muse is a bit more expensive, about 300 euros and it has four channels.
07:02
It has a very good app for Android and iOS, also connected through Bluetooth and the app teaches you to meditate and gives you feedback how calm you are and how you progress over time. It uses lots of gamification and I think this app is really very well designed.
07:28
Overall, I'd say the Muse is also a lot more comfortable, so it's just this headband and it's a bit flexible so you can adjust it to your head.
07:41
And in the direct comparison, the Muse has four channels and the mindwave has one channel. So in theory, the Muse will give you better quality data.
08:02
The Muse also has an accelerometer channel, two accelerometer channels, so this measures how the head is moving. And that's quite useful because if you know that the head is moving, you can already say the brainwaves are probably not very useful but you will probably have artefacts in there.
08:26
They have different sample rates, so the mindwave uses 512 samples per second because of bit reasons, so it's two times 256.
08:41
And the Muse has a little trouble with the p-sets, so you can't change the options directly but you have to set certain p-sets and then there are consumer p-sets and research p-sets. And what the documentation didn't say is that the firmware that was distributed with the Android
09:06
app does not give you the research p-sets and then you only have 200 hairs. Also the protocol is somehow a bit compressed, so they have a bit variable bit length scheme to encode the packets in the consumer things.
09:27
And well, this year I switched to a newer firmware in the Android app and then I got the research p-sets to work and that did work nicely.
09:40
But the newer versions of this headset, so I have the 2014 one and probably the 2016 one, doesn't do research, it only has these 200 hairs. Very important is that both these devices have dry sensors, so you don't need to
10:01
put any gel or fluid on there to make it work like the medical devices. The mindwave also has pre-computed measures, I'd say, so there's a value from 0 to 100 for attention and for meditation, what they are calling it.
10:21
And this is already completely analyzed from the brainwave data and the only thing you have to do is to react to this one byte. And this means that, for example, an Arduino or a micro-bit would be able to pass this protocol and do something useful with it.
10:46
With the Muse it's not possible because it only gives you raw data and you have to process this raw data and that takes more processing power. But on the other hand, the Muse data is less processed so you can have more control over how to actually process it.
11:08
So that's why I say the hackability of the mindwave is excellent and from the Muse it's only good. The Muse still has documentation, I found it a bit confusing or even wrong, but it still has this documentation, it's accessible over Bluetooth, but
11:28
the mindwave really is a bit easier to work with and the data you get from it, also the raw data, is easier to process.
11:41
But these devices are also very different in price, so the mindwave costs about 130 euros, the Muse costs about 300 euros. Last time I checked, a bit more I think. So the third kind of device, which I don't own however, is the OpenBCI.
12:02
They have two versions, one for 100 dollars and one for 400 dollars or 500, I don't remember quite correctly. But anyway, the one has four channels, the other has eight channels, which is more than the Muse even. Well, the advantage of this OpenBCI technology is that it's extremely flexible, so you can
12:27
even program the firmware yourself if you want and you can even reprogram the amplifiers, so you can even measure heart signals or other electrical heart signals or electrical muscle signals.
12:43
The big disadvantage of this device is that it's very do-it-yourself, it doesn't even have an enclosure and you have to attach your own electrodes and these electrodes probably have to be wet and all in all it takes a lot more effort to get this set up.
13:02
But anyway, it probably produces very good data and probably is very useful for custom mods and as the name OpenBCI implies, it's probably mainly geared towards brain control interfaces, brain computer interfaces.
13:25
Now, what I've done with this stuff is I've built a little library called
13:56
breathing rates or anything else you can think of and they are transmitted at different frequencies,
14:04
of course, and I'm using the panda's time series functionality to make this a little less painless. First of all, it currently supports MindFave and Muse, I'm also working on a Bitolino driver, but that doesn't
14:23
really work that well if you know what a Bitolino is, but it's also something for ECG and stuff. It supports especially multiple and irregular time series, so you don't really know when the data is coming and maybe it has a varying rate or whatever and panda's really supports this really well.
14:49
Another main feature is integration into the Jupyter notebook and I tried to make this relatively painless.
15:01
Now I can try to show you how this works. I have to adjust the headband to my head. Now it's streaming but it's not quite right yet, so this device is a bit more finicky, that looks a bit
15:49
noisier than it should, but in any case the problem with this kind of brainwave, with this kind of electrophysiological measurement is that virtually anything in your face or in your head will have a higher or louder signal than your brain.
16:08
For example, I can move my eyes, I can clench my teeth and this time I'm displaying here two channels, the blue
16:28
one is the left one and the red one is the right one and I can just try to move my left face. It doesn't really work currently, so now I can move my left face and my right face
17:28
and as you can see the strength of the signal is slightly different because it's a different distance. So the architecture of how this works is that there's an asyncio server that is a different process and I did that to isolate all this Bluetooth
17:52
handling stuff from the notebook and it communicates with the Bluetooth headset to get the data and then it can send it onwards to the iPison kernel.
18:11
So inside the iPison kernel I have some code that manages an experiment, it can record the data and it can then update for example a time series or something.
18:29
So inside the notebook there's also, on the browser side there's also some software. Currently I'm not using any custom JavaScript, earlier I had to but Bokeh now has this push notebook and that also works very nicely.
18:46
But it's important to know that the browser is sort of a different network endpoints and the actual server or the kernel and these are different moving paths.
19:03
So what Pisiology is mainly about is doing experiments and experiments can be anything, mostly some recordings or maybe you do some feedback or some sound experiments or whatever.
19:24
I have an experiment class so these are declared like this and in this case I would be declaring two devices, the Muse and the MindWave to have these different MAC addresses and then the server is contacted and tries to reach these devices and streams our data.
19:44
This is a more practical example with only one device. In the second cell there is first of all an HTML widget from the iPi widgets
20:00
library and this is later updated to contain some information and I instantiate the experiment class with a file name where all the data is stored. Now the experiment class instance is used as a decorator on a handler function and this handler function is called every time some new data arrives.
20:29
Then it checks if the experiment already has data on the AF7 time series which is one of the channels and then there is this display value thing which updates the HTML dynamically.
20:47
In this case just with how long the time series currently is. So that is the problem, it jumped to 30 seconds
21:16
because it has had data in the queue and that's not, I still have some problems with timing and synchronization and stuff.
21:26
It could be a bit smoother but it works quite nicely. So with relatively simple means I can already have some kind of feedback synchronization.
21:48
These experiment classes contain time series as I said and for example in this case from the Muse the AF7 and this shows the first few samples.
22:02
Every sample has a time. As a timestamp, this timestamp isn't really the real timestamp but rather I try to figure out inside the server at which point this was measured by some simple heuristics.
22:21
Unfortunately neither the Muse nor the MindWave contain any logic to tell you when they sample something. Now to analyze it we need to, often we need to window this data. So a particular sample of BrainWave is relatively useless. What is more useful is for
22:47
example different frequencies that you are seeing or different signal strengths over a certain time period. So for example you would window the data like with a window width of 3 seconds and a step size of 1 second
23:06
that means that the first window is at 0 seconds to 3 seconds and 1 to 4, 2 to 5 and so on. And this is turned into an iterator and then we can do more processing on it.
23:23
So the analysis class then can, so here on the button I pass this processing function over the windows and then into a data frame and this processing function generates several features for every window.
23:45
Especially important is the standard deviation because this is a nice way to find if there is noise in the system or any bigger artifacts. On the bottom here I try to, or I do calculate some frequency measurements, not really important
24:09
to understand it right now but there are several frequency bands so when you have alpha waves you should be more relaxed and better waves mean more concentration and so that's important to know.
24:27
Of course this data frame now is just a standard panda data frame with columns and now that we have this data frame information we can actually do normal data science stuff for example correlations.
24:43
Now as I said the Muse has 4 channels and 2 of these are on the front of the head and these are AF7 and AF8 and I am plotting these here together. The first, the alpha strength on the left, the alpha strength correlates very nicely just as a better one strength
25:08
and this also means that even if you have more channels than one it's still relatively similar information but you can also imagine that having 2 channels who measure more or less the same thing is more reliable or more precise.
25:31
Now the second thing I tried is to correlate Muse and MindWave so I did what I showed earlier, I put 2 devices into an experiment and recorded it. I had to wear these devices at the same time which I want to now because it looks silly.
25:47
But it doesn't really work as well. It could mean that I didn't put much work in it and the signal could be horrible, that could be a reason why this is not such a good correlation.
26:07
It could also mean that they have very different latencies. So that's basically everything I have to tell today.
26:24
I am trying to do more experiments but most of the work I have done so far is in integrating these devices and trying to make them work. So I haven't really progressed into doing much experimentation but anyway the code is also
26:43
on GitHub. I have to publish it later because I have to clean something up. The slides will also be on GitHub. This whole notebook presentation and some supporting code will be there and I will treat it later.
27:06
Now I'd like to talk about some other stuff I've been up to. I've been working on Poor Kid Python which is a library I created to deal with Oxford Nanopore data which is a DNA sequencer.
27:20
So the Oxford Nanopore minion is a DNA sequencer which is about the size of a computer mouse and it can sequence DNA and it has a particular data format and I try to work with that and maybe next year I'll present genome sequencing for hackers.
27:43
I'm also working on a workshop called Presenting with Jupyter which contains several tricks that I'm also using here. So you probably noticed that I don't have the help thing and the x thing and
28:01
various other ugly artifacts and I changed the transition styles and adjusted the size a bit. That's all nice to know and it's not so common knowledge. A lot of presenters had problems here with the screen resolutions. Anyway, I also did something called Jupyter Flight Gear which is a silly afternoon hack so
28:25
I used Flight Gear. Flight Gear is a flight simulation software open source and I managed to stream data from the Flight Gear directly into the Jupyter notebook and have some graphics or something. In theory you could do your own autopilot but I just want to mention it here because it's sort of cool but I have to abandon it because of time.
28:52
Now I'd like to thank you for your attention. My Twitter handle is beijianhorse. I have a GitHub account with Pyzeology.
29:01
Especially it doesn't have the news code yet but it will today. Another project I've been working on is Table Cleaner which cleans up data frame like data and turns the errors into data itself.
29:25
The notebook assets stuff is old. I should have deleted it. We now have time for questions and I'd like to start with that. Anyone questions?
29:48
We should have a microphone but we don't even have a chair.
30:18
Yes, I mean it depends. Sorry, let's repeat the question. He's talking about headphones. Which name?
30:30
Cocool. From Kickstarter they have an EEG sensor somehow and maybe it's an interesting device to add to that.
30:42
Yes, I'll try. I see if you want to do a pull request or if you need help I can certainly help you.
31:16
Can you please repeat this with the ability? You mean the data quality?
31:30
Yes. So the question was which has the better data quality and I'd say the Muse clearly has more data and you probably can get more information out of it.
31:43
It depends. For the kind of application you want, I mean the Mindwave, the new Sky stuff does very much signal processing already inside the device that can have advantages and disadvantages. I'd say the Muse produces certainly more data and probably more information, especially of different parts of the brain.
32:16
Can you please be louder? Hardly. You can tell something about the mental state but even that isn't very useful without knowing.
32:36
I have to repeat the question. The question was if you have the recording can you tell what the person was doing?
32:47
No. So the question was can you tell what someone is doing just from the brainwave data? No, you can't really because it
33:04
very much depends on the person and it's very vague information and you can maybe tell if he is very concentrated or very relaxed. But even that depends on how his brainwaves usually are distributed and their interpersonal differences.
33:27
Yes, it's possible to recognise feelings if you set up the experiments right. I guess you could do that.
34:05
The question was that I only showed two channels and you asked about if I can analyse different kinds of wavebands inside the brainwaves.
34:32
So the data analysis works like you can do a Fourier transform analysis and then you can detect the strengths at different frequencies and of course you can calculate that.
34:45
In my example I only calculated alpha and beta one and I could of course calculate everything else and it should be more precise with the muse and with the mindwave.
35:10
Okay, if there's no further questions then I'd like to thank you again and have a nice day.