OpenSTEF: Open Source energy predictions
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 542 | |
Author | ||
License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/61446 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
FOSDEM 2023332 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
Quantum stateLastprofilFood energyForm (programming)Operator (mathematics)Open setComputer animation
00:27
StatisticsTerm (mathematics)Structural loadProfil (magazine)Food energyWeightMultiplication signLine (geometry)Graph (mathematics)Open setNegative numberNatural numberBitStaff (military)Right angleZoom lensComputer animation
01:18
Staff (military)Computer animation
01:37
Food energyMaxima and minimaUser profileMultiplication signAreaWorkstation <Musikinstrument>Graph (mathematics)Perspective (visual)Food energyProfil (magazine)BitStructural loadPresentation of a groupSlide ruleNeighbourhood (graph theory)Right angleOperator (mathematics)Graph coloringGroup actionDimensional analysisSoftware developerLevel (video gaming)MereologyProduct (business)Physical lawElectric generatorLogical constantPower (physics)WebsiteReal numberSign (mathematics)MappingComputer animation
07:05
ÜberlastkontrollePhysical systemOperator (mathematics)Multitier architectureElectric currentPlastikkarteMereologyChemical equationProjective planeOperator (mathematics)Software maintenanceComputing platformFood energyPower (physics)Video gameComputer animation
09:15
Computer fontComplete metric spaceStaff (military)BitMultiplicationType theoryoutputValidity (statistics)Heegaard splittingDemo (music)CollaborationismTime seriesLine (geometry)QuantileEndliche ModelltheorieDependent and independent variablesData storage deviceFunction (mathematics)Virtual machineElectronic mailing listSet (mathematics)Power (physics)Wave packetCalculationMultiplication signArithmetic meanSystem callComputer animation
11:16
PredictionInternet forumQuantilePresentation of a groupImage resolutionBit
11:35
Correlation and dependenceEmbedded systemExecution unitProfil (magazine)Projective planeFrame problemObject (grammar)Sampling (statistics)Bounded variationPower (physics)Structural loadoutputRevision controlThermal radiationClosed setSource codeComputer animation
12:47
DataflowIcosahedronVulnerability (computing)Execution unitInformationInterface (computing)Data storage deviceDemo (music)InternetworkingFigurate numberWave packet2 (number)Endliche ModelltheorieSource code
14:09
Execution unitMaxima and minimaDemo (music)PredictionPrice indexoutputHypermediaFunction (mathematics)Data modelParameter (computer programming)Asynchronous Transfer ModePrisoner's dilemmaMetric systemCodeRevision controlControl flowWindows RegistryWave packetInformationInterface (computing)BitPredictabilityComputer animation
14:33
Normed vector spaceGoogolPredictabilityRight angleSlide ruleEndliche ModelltheorieGraph (mathematics)Operator (mathematics)Uniform resource locatorDataflowDifferent (Kate Ryan album)Video gameAmenable groupComputer animation
15:23
DataflowTrailEndliche ModelltheorieConfiguration spaceImplementationSet (mathematics)Different (Kate Ryan album)Uniform resource locatorWave packetDatabaseStack (abstract data type)Online help1 (number)Computer animation
16:49
Maxima and minimaTotal S.A.Local ringPredictionStructural loadWorkstation <Musikinstrument>Gamma functionHorizonChannel capacityGroup actionPlotterUniform resource locatorWave packetStructural loadSlide ruleMultiplicationSource codeAreaSummierbarkeitInformationEndliche ModelltheoriePoint (geometry)Sign (mathematics)Physical systemMeasurementPredictabilityBitForm (programming)Water vaporCellular automatonProduct (business)Symbol tableRule of inferenceVideo gameWhiteboardStaff (military)Computer animation
19:38
WebsiteInformationNetzwerkverwaltungOnline chatMultiplication signFood energyMultiplicationStructural loadInformationOperator (mathematics)DataflowÜberlastkontrolleSoftware maintenanceForestShape (magazine)MeasurementOnline help
22:04
Gamma functionChannel capacityStructural loadHorizonComputer animationProgram flowchart
Transcript: English(auto-generated)
00:06
Okay, awesome to see there are so many people here, really cool that there's big interest in the energy topic. My name is Frederik Stuhl and I'm from Aleander which is a grid operator and I'll
00:24
be talking about open staff today. So first of all, I put here in the graph, this is a load profile. So the energy load somewhere in the grid and well, you can see how it fluctuates over time,
00:43
sometimes it's positive, sometimes it's negative. This means whether there's natural production or natural consumption. And now the question is, or you could ask, if we are at the red line right now, what will be the load in the future?
01:04
And that's what we want to predict and if you're interested in that, then you can use open staff because open staff means short-term energy forecasting. Okay, let's zoom out a bit first and before I go into a bit more detail about what open
01:22
staff does, first I want to talk about, give a short introduction about why this is relevant. I don't have hours, so I have to keep it short, but there's a lot to talk about here. But I want to start out with this picture and it's actually quite cool because I think
01:43
the last presentation talked about flexible energy that consumers can use and this is one of the many things that are changing in the energy sector. So consumers have flexible products that also start producing, consumers also have solar
02:03
panels, your local farmer might have a wind turbine somewhere, you have big wind parks on the sea. So there's all kinds of developments going on right now that make it harder for grid operators to forecast what's going to happen tomorrow or even the day after tomorrow.
02:25
And I'll put this picture here because all the things that I mentioned you can see right there and probably in the future it's only going to get harder and harder. For now I want to focus on the renewable energy part because it's also quite impactful.
02:47
As you can see on this graph here, this is for the Netherlands, the percentage of renewable energy production. And you can see that in just a couple of years, like five years, it has more than doubled
03:01
the electricity percentage that has been produced by renewable sources. And renewable sources don't produce at a constant load of course, they change all the time depending on weather. And this means it's harder to forecast and to put that into perspective I have another
03:21
slide here. And this is a typical consumption profile. If you have like your local neighborhood then this is what the energy load will often look like. So you have the five peaks which means it's a peak for every day and in the weekend it's a bit lower. You can see the dips, these dips in the middle of the day because there's a couple
03:42
of solar panels on some roofs. It still looks easy to predict. If you go to other places where there's way more renewable energy, you can see these energy profiles change dramatically. So here you can see really a profile for a big solar park.
04:06
You can see these huge negative peaks which on some days are there and some days they are not. Probably that's a cloudy day so there's no or less energy generated. And this is an energy profile for a big wind farm which you can see is, well, seems hard
04:26
to predict because there's seems no real. So a negative energy means that the consumer or the customer, I mean, is giving back to
04:43
the grid. So then it's negative for us. If it's positive it means, sorry, yeah, exactly. So if it means a big negative peak it means the customer is producing a lot of energy. So it's just a convention, you could also switch the sign but you have to choose one
05:03
convention. Yeah, power. Is this a site which uses electricity and also has solar because otherwise how can it be using power? Yeah, so it's not just one site, it's more like a general profile for, yeah, like a substation
05:25
for the grid operator but connected to it is a lot of solar. Okay, so there's load and there's also production. Yeah, exactly. So on all of these there's load and production but yeah, I just wanted to share this feeling.
05:45
So this can be difficult and this also leads to problems and this is two maps of the Netherlands and the colored areas are the areas where Alliander is currently active and on the left
06:03
is energy consumption and on the right is energy production and this map shows if you're a new customer and you want to be connected to the grid, if it's red it's probably difficult because there's no more room. According to the Dutch law the energy grid is full over there and you can see that
06:26
this is for huge areas in the Netherlands and also large areas on the consumption side. And of course Alliander is doing everything they can to solve this by building new cables and new substations but this takes time, a lot of time and we don't have the time as
06:44
you can see in the graph before. So I don't have that one but I assume it's very similar because we're not the only one who are having these issues.
07:04
So how can we solve this? Well one important thing is that we need grid insight and therefore this also includes forecasts, so transmission forecasts and these are important for all three parts in
07:24
electrical grid, so all three parties. So for customers, for DSOs such as Alliander and for TSOs such as Tenet which control the high voltage grid.
07:43
Using these forecasts operators can try to maintain grid safety and grid balance and can give customers as much electricity as they want and as they need because the need is high.
08:03
With these forecasts we can also enable smart solutions and I put here two brochure pictures of those solutions. One of them is a pilot Flexpower which was in Amsterdam. In Amsterdam which was about charging electrical vehicles and charging them faster if it's
08:29
possible and not charging them as fast if it's not possible. We at Alliander supplied forecasts for this project and another platform is the GoPax
08:43
platform which is like a trading platform for electricity where customers can trade with operators to either consume or to produce energy flexibly. This is also being used right now at Alliander and we also provide a forecast for that.
09:04
So it's no longer working so let's use it. So now let's talk about OpenStaff again because that's why I'm here.
09:22
I'm going to give a short introduction to OpenStaff and then I'm just going to give a short demo about OpenStaff, how you can make a forecast and also want to talk a bit about using OpenStaff in an operational setting. So first of all the primary thing OpenStaff can help you with is that it's just a complete
09:44
machine learning pipeline. So I'm just going to give a short list of what it can do. It handles input validation such as like checking whether your data is complete.
10:00
It has feature engineering so it automatically calculates for you lag features or other features that are based on input features. So for example if you input it with wind speed it can calculate wind turbine power
10:20
output for you or the same for direct normal irradiance. Next it is some kind of intelligent train validation split of the time series. It has support for multiple type of regressors.
10:44
So right now we have for example XGBoost which is at Alejandro the most commonly used one. But we also had a collaboration with Sonio which added Prolo to OpenStaff. And we also have support for probabilistic forecasts. So that means like not just one line but quantiles.
11:04
And the last it has integrated with a model and artifact storage using MLflow. Okay so what does this all mean then? Let's go to an actual demo. So I'm going to put this up here.
11:25
That's a low resolution. Let's zoom out, it's a bit too much. Okay so I'm just going to walk you through an example or how you could make a forecast.
11:45
So first we need to make some kind of conflict object that's just what you have to feed OpenStaff. Let's close this. So let's run this line. Next I put some example input in this project.
12:02
So we can load it and we can visualize it. So as you can see here, well this is upon a stator frame and here we have the load and we have a lot of predictors.
12:22
Well some of these, well the names should make sense. So for example the amount of radiation predicted by the kdmi or the temperature of these predictors are already in this example data. So if we have this, I can also plot it for you.
12:45
So you can see, well this is another power profile. Okay, so now imagine you have this and you want to know, well what's next? Then first we need to train a model.
13:03
So OpenStaff has a train model pipeline which basically does all those things I just mentioned. So we can just call the pipeline and let's hope the live demo does not fail me. It will take about 15 seconds I think to train a model and store it.
13:23
So you can see some info about what it's doing and it stored it. So let's have a look. MLflow comes with an interface so we can directly see that we trained a model.
13:48
So right here this was the run. Now let's hope this works. I see that my internet is no longer working so apparently then this figure does not work.
14:03
Oh wait I'm not showing it. Well this is the MLflow interface and you can see that we just trained a model. You can also click on the model or on the train run.
14:24
This is just MLflow and you can see a bit more information about what happened during the training. Next of course we want to make a prediction. So again OpenStaff has a pipeline for that so we can just say okay I want a prediction.
14:46
So it's loading the model and using data to create a prediction and then we can visualize that as well. And we have a graph right there. So this is the forecast that it made for the next.
15:04
Well this was some example data in 2021 but about 48 hours of forecast. So that's OpenStaff in practice. Let's go back to the presentation.
15:21
Do this slide. Yes so well this flow has been a manual flow but of course in reality at least for if you're a grid operator you want to do this in an operational setting. So this means that you want to do daily forecasts for a lot of different locations
15:43
with all kinds of configurations. And OpenStaff also comes with a so-called reference implementation about how you could do this. So this is a picture of what you would have to do. So we have OpenStaff right here which is basically I just showed you the training
16:02
and forecasting pipeline. Then we have another package which is called OpenStaff DBC database connector which can connect to a database. And we use MySQL and InfluxDB to store all the data required to run it operationally.
16:26
And we also have a Grafana dashboard built upon this database stack. So we can also see what's going on. And again as I already have shown you can use MLflow to keep track of all the models and all the runs that are being done to see what's going on.
16:44
So I want to show this dashboard as well. So let's put it up here.
17:00
So this dashboard is just example data so it's not our real dashboard. But here you can for example see some load that was there on the system. You can also see that for example this is not just one area but it has a sum of two systems
17:25
which is quite common in electrical grid that you have a lot of measurement points that you have to add together with different signs. And you can see for example here's then a live forecast of this location as well.
17:46
You can also see plots of the feature importance that are obtained during training of the model. You can see on which data the model has been trained. Over here these plots are really small but here you can see them.
18:06
So it's a dashboard where you can see everything that Oberstaff does for every location that you could be interested in. When you talk about forecast are you within the forecast also taking other forecasts like the weather forecast into account or are you forecasting that yourself?
18:23
No so we use all kinds of data and the weather forecast is... So the question is whether we are using other forecasted data or whether we forecast do those forecasts as well. Like whether we forecast the weather ourselves.
18:41
And the answer is that it depends a bit. So in general we use the weather forecast for multiple sources and also for example price like the day ahead pricing. So we use those data but sometimes we also feed the prediction itself
19:04
or we feed one prediction into another prediction. So I mean you can play around with that. But yeah you have to feed Oberstaff with all the predictors that you wanted to know.
19:24
Okay let's move on. I've only one last slide and that's basically the key information.
19:41
Yeah because that was my presentation. So here I put all the info you might be interested in on this slide. And if there's any questions then feel free to ask.
20:07
Yes. So the information is really useful but what is the purpose for the net grid operator? What use does a grid operator make of this information? Is it for congestion management?
20:22
Is it for you know some kind of load shedding? What is the role of this exercise? Yeah so I think the question is why would the grid operator be interested in forecasts? I guess is what you're asking.
20:41
So well there's many reasons but I think you already mentioned congestion management is indeed an important reason but also well grid insight. So yeah the more congestion management is going to be used for the grid
21:03
the more important it is also to maintain grid safety. And grid safety is not just one operator. We are all connected to multiple grid operators so everyone has to communicate what they are going to do and what they expect the energy flow to be the next day.
21:23
So every operator can decide to do what's necessary to maintain grid safety. So that's what I mentioned before the transmission forecast. Every operator has to communicate to everyone who he is connected to
21:41
what they expect the load to be on the next day. I see that my time is up so I'm afraid I have to answer the questions in the chat. Or in the hallway. I think for time management we have to learn from these thoughts and see if we can manage to keep a couple of more minutes for questions.
22:01
Thank you very much for listening.