Cool. So yeah, as Matt said, I'm going to talk about my attempts to see inside HTMs. It's using comportx-vis that Marcus has already

helpfully introduced. And I have to say, as an aside, he was very misleading in his comments there. He actually did a lot of substantial work on comportx-vis there and the integration. And it's fantastic to have comportx running

on a server on the JVM, which has a lot more capacity than running in the browser. So I'm looking forward to using that as well. If you're interested in comportx-vis, it's on GitHub.

And it's able to run HTM in the browser and do some visualizations of it. And why don't I just demonstrate that really quickly. So if I run this, this is a sensory motor data stream.

Wait, OK. So to just get you oriented in this space. On the left, we have a representation

of what the input is. In this case, it's just like a sequence of letters, A, B, C, D, E, F, G, H, A, J, K, not H, A, J. And it's currently focusing on one letter, so represented by the black line. So basically, this letter, C, is just the input through a category encoder.

And because this is a sensory motor case study, it has an additional bit of input, which is the direction that we're going to move in the next time step, the saccade, you might call it. And that's coming from this region. Again, it's like a category about the direction

and the amount of the movement. These fields here are the activated columns. And in this case, there's two layers, layer four and layer three. Layer four receives the sensory input on the proximal feed

forward segment. And it also receives the motor input as a distal input to the distal segments. So that's just to get you oriented

about what all these dots are. But it's not really what I want to talk about today, because what I worked on for my hack was the temporal pooling algorithm. I've been trying to implement the temporal pooling algorithm. I'm not going to explain it here. If you don't know what the temporal pooling is,

you can look it up on the Numenta Wiki, some good stuff. Basically, the idea is that to form stable but distinct representations in higher level regions or higher level layers.

And it does that by keeping cells active over time while the underlying sequence is predictable. So the problem I ran into, I've been playing around with implementing this, but as you go through the process,

you realize that it's not clear how to do it exactly and what should be the relative weight of all these parameters that you're faced with creating. And what I ended up realizing I needed was an ability to visualize what

the different influences are on the cells that are becoming active. And because it's a sparse representation, that's feasible. There's only a relatively small number of active cells on each time step. The number is usually like 2% of the number of columns.

So in this example, what I'm showing here on the right, the individual bars here are single cells that are becoming active. So there's a total of, I'd say, 12 or 15 cells

that are becoming active. And in each one, the color represents the amount of influence that's causing it to become active. So red is just the proximal excitation as represented by these lines here. It's the proximal excitation that's driving that.

The yellow, I don't know if you can see the yellow, that's the boosting factor, which is part of the algorithm that's to do with encouraging columns to become active that haven't become active recently. So it surprised me, actually, when I saw this how much effect the boosting is having.

Obviously, I'd set that value, but I'd forgotten about it. And it was invisible until now. So this is at the beginning. It's just proximal forcing. But if we run this for a little while, just 100 times steps

or so, it will start to be able to predict. So the first layer will start to be able to predict what's happening. And we can see that in the time series plot. So in this plot, although it's the same kind of structure, it's showing a completely different thing.

In this case, it's showing the distribution of the states of the active columns, so red being active and unpredicted, active but not predicted to become active. And purple is correctly predicted to become active. And you can see over time, a higher proportion of the columns

were predicted, as you'd expect. And we're just moving constantly around this one sequence of letters. So of course, it's been predicted. OK, I'm going to stop that there. Also, incidentally, the light blue represents columns which were predicted but did not become active.

So if we look now at what's happening, I'm just going to turn off these. OK, so you can see on the lower level layer,

there's cells which are predicted. But the red, these represent the active cells, but the red indicates that the feed forward input that's activating those was not predicted

and can't be predicted because it's coming from the input data. This is actually not really relevant there. But if you look at the layer three, which is the next layer in the stack, the purple here is indicating that what's forcing these cells to become active is input. Hang on, I'll just look at mine.

So the purple lines to these synapses representing that the proximal input that's turning those cells on is coming from cells which were predicted below. So it means the underlying sequence is predicted. And in that case, it adds to a persistent level

of excitation called the temporal pooling level, which on this plot is the green line. So it's that which is responsible for forcing some of these cells to become active in the higher level

region. And I'm just stepping forward in time here. So you have a situation where this set of cells has become active. And because it's from the predicted input, then in the following time step, you've got a large amount of persistent excitation, which

is the green lines there. And that just decays over time until those cells turn off. And if you step forward, then you've got red coming in. That means something has been unpredicted, not predicted, and then lay it below. And then the temporal pooling tends to turn off in that case.

So this has helped me to just calibrate or check that nothing crazy is going on. And they were just crazy things like the number of active synapses coming in

and activating cells was way too low. It was like five synapses. So it's just good to be able to check that sanity check and to start to get a feel for what the relative influences of the continuing temporal pooling activation and how fast it's dropping off

and what's causing it to drop off. I haven't really explored this much, but that's just where I got to. And hopefully it's given you some idea of the concepts involved in that algorithm.

Any questions? The predicted eye line off the chart right down there. Oh yeah, it's sort of cicading forward. And in that case, I just wrap it around. It's going to come into A or something?

So I'd like to do this better. But what we should have is multiple words. And then you're going to have microcicades within it and then high-level cicades going to the next word. So you can start actually. What I was originally hoping to look at

was forming a higher-level sequence memory, like actually predicting the sequence of words on top, like in the next layer. I realized that I've got more fundamental issues to do with how the algorithm is implemented before I can start to look at that. So what's the impact of the predicted eye line?

So your predictions are the letter that is going to be seen next. Oh, this is not a prediction. But what does that contribute? You mean this? The dim line. Yeah, that's not a prediction. That's actually what the next movement is going to be. Right.

Oh, so I'm sorry. It's not a prediction. But it's the next move. But is that fed in also? Yes, so there's two inputs. Two inputs. One is sensory input. One's motor input. So the motor input is only given in as distal input. But it's always anticipatory input then.

Interesting. And there's different choices that could be made there as well about whether it comes in proximally or distally, and whether you also

have lateral distal connections in layer 4. I'm not familiar with the current thinking on that, actually. Yeah? So I was just curious.

Test, test, test. I was curious what ICB put this in JS in JavaScript, right? Well, it's compiled to JS from Closure. From Closure. Did you have to port the entire algorithm? Or did you? Yeah. OK.

Interesting. Thanks. Thanks. More questions?

Sorry, I might just say on the CombotX GitHub page, there's a link to this which just has some interactive online demos, different types of examples. Yeah, so check that out.