We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

OpenSTEF: Open Source energy predictions

00:00

Formal Metadata

Title
OpenSTEF: Open Source energy predictions
Title of Series
Number of Parts
542
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
The energy transition poses new challenges to all parties in the energy sector. For grid operators, the rise in renewable energy and electrification of energy consumption leads to the capacity of the grid to near its physical constraints. Forecasting the load on the grid in the next hours to days is essential for anticipating on local congestion and making the most of existing assets. The open source package OpenSTEF provides a complete software stack which forecasts the load on the electricity grid for the next hours to days. Given a timeseries of measured (net) load or generation, a fully automated machine learning pipeline is executed which delivers a probabilistic forecast of future load. This works for energy consumption, (renewable) generation or a combination of both. During this presentation, we will show how we have implemented this opensource tooling at the Dutch Distribution System Operator Alliander to fully automate forecasting the load on the grid.
14
15
43
87
Thumbnail
26:29
146
Thumbnail
18:05
199
207
Thumbnail
22:17
264
278
Thumbnail
30:52
293
Thumbnail
15:53
341
Thumbnail
31:01
354
359
410
Quantum stateLastprofilFood energyForm (programming)Operator (mathematics)Open setComputer animation
StatisticsTerm (mathematics)Structural loadProfil (magazine)Food energyWeightMultiplication signLine (geometry)Graph (mathematics)Open setNegative numberNatural numberBitStaff (military)Right angleZoom lensComputer animation
Staff (military)Computer animation
Food energyMaxima and minimaUser profileMultiplication signAreaWorkstation <Musikinstrument>Graph (mathematics)Perspective (visual)Food energyProfil (magazine)BitStructural loadPresentation of a groupSlide ruleNeighbourhood (graph theory)Right angleOperator (mathematics)Graph coloringGroup actionDimensional analysisSoftware developerLevel (video gaming)MereologyProduct (business)Physical lawElectric generatorLogical constantPower (physics)WebsiteReal numberSign (mathematics)MappingComputer animation
ÜberlastkontrollePhysical systemOperator (mathematics)Multitier architectureElectric currentPlastikkarteMereologyChemical equationProjective planeOperator (mathematics)Software maintenanceComputing platformFood energyPower (physics)Video gameComputer animation
Computer fontComplete metric spaceStaff (military)BitMultiplicationType theoryoutputValidity (statistics)Heegaard splittingDemo (music)CollaborationismTime seriesLine (geometry)QuantileEndliche ModelltheorieDependent and independent variablesData storage deviceFunction (mathematics)Virtual machineElectronic mailing listSet (mathematics)Power (physics)Wave packetCalculationMultiplication signArithmetic meanSystem callComputer animation
PredictionInternet forumQuantilePresentation of a groupImage resolutionBit
Correlation and dependenceEmbedded systemExecution unitProfil (magazine)Projective planeFrame problemObject (grammar)Sampling (statistics)Bounded variationPower (physics)Structural loadoutputRevision controlThermal radiationClosed setSource codeComputer animation
DataflowIcosahedronVulnerability (computing)Execution unitInformationInterface (computing)Data storage deviceDemo (music)InternetworkingFigurate numberWave packet2 (number)Endliche ModelltheorieSource code
Execution unitMaxima and minimaDemo (music)PredictionPrice indexoutputHypermediaFunction (mathematics)Data modelParameter (computer programming)Asynchronous Transfer ModePrisoner's dilemmaMetric systemCodeRevision controlControl flowWindows RegistryWave packetInformationInterface (computing)BitPredictabilityComputer animation
Normed vector spaceGoogolPredictabilityRight angleSlide ruleEndliche ModelltheorieGraph (mathematics)Operator (mathematics)Uniform resource locatorDataflowDifferent (Kate Ryan album)Video gameAmenable groupComputer animation
DataflowTrailEndliche ModelltheorieConfiguration spaceImplementationSet (mathematics)Different (Kate Ryan album)Uniform resource locatorWave packetDatabaseStack (abstract data type)Online help1 (number)Computer animation
Maxima and minimaTotal S.A.Local ringPredictionStructural loadWorkstation <Musikinstrument>Gamma functionHorizonChannel capacityGroup actionPlotterUniform resource locatorWave packetStructural loadSlide ruleMultiplicationSource codeAreaSummierbarkeitInformationEndliche ModelltheoriePoint (geometry)Sign (mathematics)Physical systemMeasurementPredictabilityBitForm (programming)Water vaporCellular automatonProduct (business)Symbol tableRule of inferenceVideo gameWhiteboardStaff (military)Computer animation
WebsiteInformationNetzwerkverwaltungOnline chatMultiplication signFood energyMultiplicationStructural loadInformationOperator (mathematics)DataflowÜberlastkontrolleSoftware maintenanceForestShape (magazine)MeasurementOnline help
Gamma functionChannel capacityStructural loadHorizonComputer animationProgram flowchart
Transcript: English(auto-generated)
Okay, awesome to see there are so many people here, really cool that there's big interest in the energy topic. My name is Frederik Stuhl and I'm from Aleander which is a grid operator and I'll
be talking about open staff today. So first of all, I put here in the graph, this is a load profile. So the energy load somewhere in the grid and well, you can see how it fluctuates over time,
sometimes it's positive, sometimes it's negative. This means whether there's natural production or natural consumption. And now the question is, or you could ask, if we are at the red line right now, what will be the load in the future?
And that's what we want to predict and if you're interested in that, then you can use open staff because open staff means short-term energy forecasting. Okay, let's zoom out a bit first and before I go into a bit more detail about what open
staff does, first I want to talk about, give a short introduction about why this is relevant. I don't have hours, so I have to keep it short, but there's a lot to talk about here. But I want to start out with this picture and it's actually quite cool because I think
the last presentation talked about flexible energy that consumers can use and this is one of the many things that are changing in the energy sector. So consumers have flexible products that also start producing, consumers also have solar
panels, your local farmer might have a wind turbine somewhere, you have big wind parks on the sea. So there's all kinds of developments going on right now that make it harder for grid operators to forecast what's going to happen tomorrow or even the day after tomorrow.
And I'll put this picture here because all the things that I mentioned you can see right there and probably in the future it's only going to get harder and harder. For now I want to focus on the renewable energy part because it's also quite impactful.
As you can see on this graph here, this is for the Netherlands, the percentage of renewable energy production. And you can see that in just a couple of years, like five years, it has more than doubled
the electricity percentage that has been produced by renewable sources. And renewable sources don't produce at a constant load of course, they change all the time depending on weather. And this means it's harder to forecast and to put that into perspective I have another
slide here. And this is a typical consumption profile. If you have like your local neighborhood then this is what the energy load will often look like. So you have the five peaks which means it's a peak for every day and in the weekend it's a bit lower. You can see the dips, these dips in the middle of the day because there's a couple
of solar panels on some roofs. It still looks easy to predict. If you go to other places where there's way more renewable energy, you can see these energy profiles change dramatically. So here you can see really a profile for a big solar park.
You can see these huge negative peaks which on some days are there and some days they are not. Probably that's a cloudy day so there's no or less energy generated. And this is an energy profile for a big wind farm which you can see is, well, seems hard
to predict because there's seems no real. So a negative energy means that the consumer or the customer, I mean, is giving back to
the grid. So then it's negative for us. If it's positive it means, sorry, yeah, exactly. So if it means a big negative peak it means the customer is producing a lot of energy. So it's just a convention, you could also switch the sign but you have to choose one
convention. Yeah, power. Is this a site which uses electricity and also has solar because otherwise how can it be using power? Yeah, so it's not just one site, it's more like a general profile for, yeah, like a substation
for the grid operator but connected to it is a lot of solar. Okay, so there's load and there's also production. Yeah, exactly. So on all of these there's load and production but yeah, I just wanted to share this feeling.
So this can be difficult and this also leads to problems and this is two maps of the Netherlands and the colored areas are the areas where Alliander is currently active and on the left
is energy consumption and on the right is energy production and this map shows if you're a new customer and you want to be connected to the grid, if it's red it's probably difficult because there's no more room. According to the Dutch law the energy grid is full over there and you can see that
this is for huge areas in the Netherlands and also large areas on the consumption side. And of course Alliander is doing everything they can to solve this by building new cables and new substations but this takes time, a lot of time and we don't have the time as
you can see in the graph before. So I don't have that one but I assume it's very similar because we're not the only one who are having these issues.
So how can we solve this? Well one important thing is that we need grid insight and therefore this also includes forecasts, so transmission forecasts and these are important for all three parts in
electrical grid, so all three parties. So for customers, for DSOs such as Alliander and for TSOs such as Tenet which control the high voltage grid.
Using these forecasts operators can try to maintain grid safety and grid balance and can give customers as much electricity as they want and as they need because the need is high.
With these forecasts we can also enable smart solutions and I put here two brochure pictures of those solutions. One of them is a pilot Flexpower which was in Amsterdam. In Amsterdam which was about charging electrical vehicles and charging them faster if it's
possible and not charging them as fast if it's not possible. We at Alliander supplied forecasts for this project and another platform is the GoPax
platform which is like a trading platform for electricity where customers can trade with operators to either consume or to produce energy flexibly. This is also being used right now at Alliander and we also provide a forecast for that.
So it's no longer working so let's use it. So now let's talk about OpenStaff again because that's why I'm here.
I'm going to give a short introduction to OpenStaff and then I'm just going to give a short demo about OpenStaff, how you can make a forecast and also want to talk a bit about using OpenStaff in an operational setting. So first of all the primary thing OpenStaff can help you with is that it's just a complete
machine learning pipeline. So I'm just going to give a short list of what it can do. It handles input validation such as like checking whether your data is complete.
It has feature engineering so it automatically calculates for you lag features or other features that are based on input features. So for example if you input it with wind speed it can calculate wind turbine power
output for you or the same for direct normal irradiance. Next it is some kind of intelligent train validation split of the time series. It has support for multiple type of regressors.
So right now we have for example XGBoost which is at Alejandro the most commonly used one. But we also had a collaboration with Sonio which added Prolo to OpenStaff. And we also have support for probabilistic forecasts. So that means like not just one line but quantiles.
And the last it has integrated with a model and artifact storage using MLflow. Okay so what does this all mean then? Let's go to an actual demo. So I'm going to put this up here.
That's a low resolution. Let's zoom out, it's a bit too much. Okay so I'm just going to walk you through an example or how you could make a forecast.
So first we need to make some kind of conflict object that's just what you have to feed OpenStaff. Let's close this. So let's run this line. Next I put some example input in this project.
So we can load it and we can visualize it. So as you can see here, well this is upon a stator frame and here we have the load and we have a lot of predictors.
Well some of these, well the names should make sense. So for example the amount of radiation predicted by the kdmi or the temperature of these predictors are already in this example data. So if we have this, I can also plot it for you.
So you can see, well this is another power profile. Okay, so now imagine you have this and you want to know, well what's next? Then first we need to train a model.
So OpenStaff has a train model pipeline which basically does all those things I just mentioned. So we can just call the pipeline and let's hope the live demo does not fail me. It will take about 15 seconds I think to train a model and store it.
So you can see some info about what it's doing and it stored it. So let's have a look. MLflow comes with an interface so we can directly see that we trained a model.
So right here this was the run. Now let's hope this works. I see that my internet is no longer working so apparently then this figure does not work.
Oh wait I'm not showing it. Well this is the MLflow interface and you can see that we just trained a model. You can also click on the model or on the train run.
This is just MLflow and you can see a bit more information about what happened during the training. Next of course we want to make a prediction. So again OpenStaff has a pipeline for that so we can just say okay I want a prediction.
So it's loading the model and using data to create a prediction and then we can visualize that as well. And we have a graph right there. So this is the forecast that it made for the next.
Well this was some example data in 2021 but about 48 hours of forecast. So that's OpenStaff in practice. Let's go back to the presentation.
Do this slide. Yes so well this flow has been a manual flow but of course in reality at least for if you're a grid operator you want to do this in an operational setting. So this means that you want to do daily forecasts for a lot of different locations
with all kinds of configurations. And OpenStaff also comes with a so-called reference implementation about how you could do this. So this is a picture of what you would have to do. So we have OpenStaff right here which is basically I just showed you the training
and forecasting pipeline. Then we have another package which is called OpenStaff DBC database connector which can connect to a database. And we use MySQL and InfluxDB to store all the data required to run it operationally.
And we also have a Grafana dashboard built upon this database stack. So we can also see what's going on. And again as I already have shown you can use MLflow to keep track of all the models and all the runs that are being done to see what's going on.
So I want to show this dashboard as well. So let's put it up here.
So this dashboard is just example data so it's not our real dashboard. But here you can for example see some load that was there on the system. You can also see that for example this is not just one area but it has a sum of two systems
which is quite common in electrical grid that you have a lot of measurement points that you have to add together with different signs. And you can see for example here's then a live forecast of this location as well.
You can also see plots of the feature importance that are obtained during training of the model. You can see on which data the model has been trained. Over here these plots are really small but here you can see them.
So it's a dashboard where you can see everything that Oberstaff does for every location that you could be interested in. When you talk about forecast are you within the forecast also taking other forecasts like the weather forecast into account or are you forecasting that yourself?
No so we use all kinds of data and the weather forecast is... So the question is whether we are using other forecasted data or whether we forecast do those forecasts as well. Like whether we forecast the weather ourselves.
And the answer is that it depends a bit. So in general we use the weather forecast for multiple sources and also for example price like the day ahead pricing. So we use those data but sometimes we also feed the prediction itself
or we feed one prediction into another prediction. So I mean you can play around with that. But yeah you have to feed Oberstaff with all the predictors that you wanted to know.
Okay let's move on. I've only one last slide and that's basically the key information.
Yeah because that was my presentation. So here I put all the info you might be interested in on this slide. And if there's any questions then feel free to ask.
Yes. So the information is really useful but what is the purpose for the net grid operator? What use does a grid operator make of this information? Is it for congestion management?
Is it for you know some kind of load shedding? What is the role of this exercise? Yeah so I think the question is why would the grid operator be interested in forecasts? I guess is what you're asking.
So well there's many reasons but I think you already mentioned congestion management is indeed an important reason but also well grid insight. So yeah the more congestion management is going to be used for the grid
the more important it is also to maintain grid safety. And grid safety is not just one operator. We are all connected to multiple grid operators so everyone has to communicate what they are going to do and what they expect the energy flow to be the next day.
So every operator can decide to do what's necessary to maintain grid safety. So that's what I mentioned before the transmission forecast. Every operator has to communicate to everyone who he is connected to
what they expect the load to be on the next day. I see that my time is up so I'm afraid I have to answer the questions in the chat. Or in the hallway. I think for time management we have to learn from these thoughts and see if we can manage to keep a couple of more minutes for questions.
Thank you very much for listening.