Market Clearing – aFRR

New markets for reserve services and near-real-time power delivery require new, sophisticated tools for calculating optimal market clearing. In theory clearing a market is simple – you just need to match supply and demand. In practice though, complex bid and offer structures as well as integration between reserve services often mean that things are not quite so straight forward. Getting it right though is very important – clearing and pricing new markets reliably, transparently and quickly is essential for efficient market operation.

Optimeering has a unique combination of modelling expertise and market know-how that positions us to deliver robust, customised market clearing tools for any type of power market. Recently, we have developed the clearing algorithm for the upcoming pan-Nordic aFRR market for the SVK, Statnett, and Fingrid TSO consortia.

The four Nordic TSOs are planning to implement a common Nordic capacity market for aFRR (automatic frequency restoration reserves) in 2018, and our part of this work is developing and implementing the market clearing engine used to select that combination of bids that is most efficient (“maximises social welfare surplus”). Given that the bidding rules are quite complex (bids can be linked upwards, downwards, and in time, marked as indivisible, and be asymmetrical, to mention some) and the requirement to ensure that the market operates in a socio-economically efficient manner, the problem is not easy.

The problem itself is what we call a combinatorial optimization problem or (mixed) integer program. The linking and non-divisibility of bids is a critical and fundamental characteristic of the bid selection problem that means traditional clearing methods based on hourly bid price alone are insufficient. Given the size of the problem (hourly bids in multiple bidding zones), checking every single possible combination of bids is also not a feasible approach. Instead, a solution algorithm is needed to select bids for the aFRR market that accounts for the complex bid structures via the use of advanced mathematical optimization techniques. We have come up with some clever formulation approaches (if we do say so ourselves …) that take advantage of structure in the problem to clear the market optimally and very quickly (here we mean seconds, not minutes or hours). Our approach also works for several alternative pricing mechanisms, including pay-as-bid and marginal-cost pricing.

If you are interested in learning more about the aFRR market, or how we can help in other markets, please contact us.

The PUMA Algorithms

We are getting close to a Beta release of our first member of the PUMA Algorithm’s family, and I wanted to update everyone on what to expect and on progress so far.

The PUMA Algorithms are advanced models for the analysis and prognosis of power markets, that take detailed account of future uncertainty in input levels such as hydro inflow, demand, fuel and emissions prices. The first PUMA Algorithm, PUMA SP, is being developed as part of the PUMA Research Project sponsored by Research Council of Norway, and paid for by several major actors in the Nordic power market.

PUMA SP is a fundamental medium term time-frame model that captures the impact of multiple uncertainty drivers such as inflows, availabilities, demand, fuel and CO2 prices. In modelling something as complex as a power market, you have to make trade offs. Speed verses detail. Detailed modelling of hydro verses detailed modelling of CHP. Uncertainty verses perfect foresight. With PUMA SP we have taken the view that the user is best positioned to make these trade-offs, as they can change from analysis to analysis.

PUMA SP is therefore designed with flexibility and ease-of-use in mind, and can be configured to run at whatever level of detail you need. Want to run deterministically? Just one parameter. Uncertain inflows and fuel prices? The same. Add in uncertain demand? No problem. One nice thing – once you have one uncertain parameter, adding new ones does not add much solve overhead. So, if you need “quick-and-dirty”, that’s what you can have, whilst being able to use the same model and data later on to refine and model a fully detailed market response.

All Puma Algorithms are fully integrated with the PUMA Analytic Framework, and are provided as python packages and with a browser-based front-end. PUMA SP is currently in locked alpha testing with our development license group, but we are planning for it to be available to new customers in Beta at the end of 2017. Drop us a line to find out more.

PUMA Analytic Framework

Most of the models we build use quite a lot of data, and especially time series or time-stamped data. However, many of them don’t use huge, enormous petabytes of data. A data lake, not a data ocean. Handling this can be a bit tricky – spreadsheets are much too limited, and relational databases are generally too slow to read and (especially) write even moderate volumes of time series data. At the other end of the scale, large no-sql data solutions are just too complex and unwieldy, like trying to crack a nut with a hammer.

We needed something that could read and write time series and time-stamped data quickly; could classify, group and tag series (“inflow series”, “cases” or “scenarios”); could extract series even with missing or incomplete data; and had an overhead low enough that it could be installed on a desktop and accessed via the python tools we use every day.

We couldn’t find it, so we decided to build our own – the Puma Analytic Framework, for the flexible and rapid storage, retrieval, visualisation and manipulation of time series and time-stamped data. Implemented as a python package, the Framework uses a combination of a relational database and fast flat file storage and retrieval to create a tool to easily and cheaply store pretty large data volumes on the desktop, and use them in your data analysis and models. Being a python package, you have immediate access to all your usual python tools, such as numpy, scipy, pandas and Jupyter Notebooks.

The Framework makes developing new models and analytic tools easier and faster, by providing data in a readily accessible, standardised format. We use the Framework to store and prepare data for our own PUMA Algorithms power market model, and in the same way it can be fully integrated with your own existing and 3rd party power market modelling tools. This is one of the big advantages of the PUMA Analytic Framework in our opinion – it enables you to have a single, consistent database for all your models.

The PUMA Analytic Framework is in locked alpha testing and will be made available in Beta shortly, after which you will be able to download the package and documentation from our website for free. Email us in the meantime if you are interested in finding out more.

Optimeering granted research funds for major machine learning project

We are very pleased to announce that Optimeering has been granted funding from the Research Council of Norway for a substantial new project to develop new machine learning algorithms for forecasting power grid imbalance volumes. The project kicks-off in Q1 2017, with a budget of approximately 7 MNOK. The funding is part of the Council`s ENERGIX-program, and our project is among the few commercial projects fortunate enough to have been successful in this funding round. Optimeering’s team will undertake the project in close collaboration with Statnett and researchers from the University of Liège and the Norwegian University of Science and Technology.

This is the third time Optimeering has been granted research funding from ENERGIX, but today’s grant is by far the largest and represents a significant and positive development for the company. The two former grants (from 2013 and 2014) have both resulted in the development and subsequent commercialization of decision support models for the power sector, namely MARC (an agent based el-certificate market model) and PUMA (a fundamental stochastic power market model). Similarly, our new project will result in commercial models and tools for imbalance volume prognosis, for use by electricity grid operators in Norway and internationally.

The project, “Machine learning prognosis for system imbalance volumes”, aims to develop new and efficient machine learning algorithms for forecasting short-term imbalances in the power grid. Statnett, like all grid operators, faces high costs for managing real-time imbalances between supply and demand of power throughout its transmission network. Further, the number and size of these imbalances, which drive grid frequency deviations and thus negatively impact power quality, have increased steadily over the past decade. The drivers behind the increase in costs – higher intermittent production from renewables and tighter integration with surrounding countries – are expected to become stronger going forward. Therefore, being able to forecast system imbalance volumes has a great value to grid operators such as Statnett as well as other TSOs around the world.

We are of course very grateful to have the Research Council onboard yet again, and for their recognition of the significant potential – both commercially and for society more generally – of this project. We will be publishing details of our progress on our blog on a regular basis, so keep checking back for more info and updates!

Frokostmøte om elsertifikatmarkedet — fredag 29.april

I forbindelse med lanseringen av vår agentbaserte markedsmodell av elsertifikatmarkedet inviterer vi alle interessenter til en felles presentasjon av modellresultatene, modellen, og tjenestene vil tilbyr, fredag 29.april kl 08:00-09:00, i Tollbugata 35 i Oslo. Her blir det anledning til å stille spørsmål samt å møte andre interessenter over en kopp kaffe.

For dem som er interessert vil vi da, blant annet, kunne forklare hvordan investeringsbeslutningen på Fosen kan øke risikoen for svært høye sertifikatpriser i 2020.


Modellen som ligger til grunn for markedsanalysene (MARC) er utviklet av Optimeering og Thema, og finansiert av Forskningsrådet, Innovasjon Norge, samt en rekke private og offentlige aktører, deriblant NVE, Energimyndigheten, ENOVA, BKK og Statkraft.  Overordnet informasjon om modellen og våre modellbaserte analysetjenester finnes i denne brosjyren, mens en detaljert beskrivelse av MARC-modellen finner du her.

Agentbasert modell av elsertifikatmarkedet (MARC) klar for lansering

MARC – Multi-Agent Renewable Certificate model

Etter over 2 års utvikling er vår agentbaserte modell av sertifikatmarkedet (MARC) klar for lansering. Utviklingsprosjektet har hatt en ramme på nesten 3 millioner kroner og vært finansiert av et dusin private og offentlige aktører, og har etter alle solemerker oppnådd målet, nemlig å utvikle verdens første agentbaserte markedsmodell for et kvotemarked.


Elsertifikatprisen styres i stor grad av dagens sertifikatbalanse og forventinger om framtidig sertifikatbalanse. Etter hvert som markedet nærmer seg balanseperioden vil prisene i enda mindre grad vil styres av utbyggingskostnadene og i større grad av forventinger om framtidig sertifikatbalanse. For å bedre forstå hvordan markedet vil kunne utvikle seg fram mot balanseperioden lanserer vi, sammen med Thema Consulting Group, endelig en agentbasert markedsmodell av elsertifikatmarkedet.

Fram til i dag har fundamentale likevektsmodeller hovedsakelig vært benyttet til å analysere markedet. Slik modeller er basert på kostnadsdekning på tilbudssiden og har egnet seg godt til å analysere investeringsfasen av markedet, men etter hvert som vi nærmer oss slutten av denne fasen er det sannsynlig at prisene i større grad avvike fra utbyggingskostnadene. Skal man da kunne si hvor markedet vil ta vei trengs en tilnærming som tar høyde for aktørenes strategier og forventninger.

Agentbasert modellering har opprinnelig røtter fra spillteori og kybernetikk, men har i økende grad blitt bruk til å studere markeder hvor en likevekt har vist seg vanskelig å oppnå. Modelltilnærmingen skiller seg fra en standard likevektstilnærming vet at den ikke bygger på en overordnet antakelse om kostnadsdekning, men heller tar utgangspunkt i å fange de individuelle aktørenes adferd og la utfallet bestemmes av deres interaksjoner. Utfordringen er å beskrive markedsaktørenes adferd og modellere dette dynamisk. Utviklingen har derfor vært en tidkrevende prosess og strukket seg over 3 år.

Vi skrev i 2014, da prosjektet startet, en blogpost om hvorfor vi mente en agentbasert modell er bedre egnet til å modellere markedet for elsertifikater enn fundamentale likevektsmodeller. Disse punktene er kanskje enda mer aktuelle i dag og det er en glede å endelig kunne lansere denne modellen vår.

Modellen er utviklet av Optimeering og Thema, og finansiert av Forskningsrådet, Innovasjon Norge, samt en rekke private og offentlige aktører, deriblant NVE, Energimyndigheten, ENOVA, BKK og Statkraft. En forenklet beskrivelse av MARC-modellen finner du her.

Hva vi tilbyr

Markedsanalyser basert på MARC-modellen vil i all hovedsak være tilgjengelig gjennom deltakelse i vårt Sertifikatmarkedsforum EMF (El-Certificate Market Forum), som er et felles samlingssted for aktører i bransjen. Alle deltakerne i forumet møtes jevnlig for å diskutere mulige endringer i elsertifikatmarkedet og se nærmere på deres mulige innvirkning på sertifikatmarkedet ved hjelp av MARC-modellen.

Mer informasjon om modellen og våre modellbaserte analysetjenester finnes i denne flyeren. For øvrig informasjon ta kontakt med Karan Kathuria.

Cops with guns – should an increase in armed incidents mean we should arm the police?

Currently in the two countries closest to my heart – New Zealand and Norway – there is an ongoing debate about arming the police. The arguments for this are a little different in each country, and are complex. However without too much effort they can be simplified (over-simplified?) to

“there are more armed incidents (of whatever type), therefore the police need to be armed to protect themselves and the public.”

My aim here is not to argue the merits (or otherwise) of arming the police – that is far too much for a mere Optimeering blog entry. Instead I want to examine one of the pillars of the argument: that an increased number of incidents means the world is a more dangerous place. That is, can we say “things are worse than before”, simply because the number of armed incidents over the past couple of months/years/whatever time interval has increased?

The “it’s getting worse” argument is not just used when discussing arming the police. It appears often in other crime statistics (e.g. burglary), accident statistics, or similar. In all cases, we need to ask a simple question – is what we are seeing a real change, or is it just random variation (with the corollary that things are not actually getting worse)?

As this is an Optimeering blog, it’s clear what I am going to say – let’s look at what the numbers (and models) can tell us. Let us assume for a moment that the world is not getting worse – that is, that the expected number of armed incidents in any given year is not changing, but is the same year-to-year. How would the number of incidents year-on-year then vary? Would we see, for example, periods of upwards trend (or downwards trend) that may make it look like things are getting worse (or better)?

To do this, I built a small model of armed incident occurrence. In the model, I have assumed that we have a fixed average number of armed incidents per year, and that each armed incident is independent (meaning, for example, that the number of armed incidents next year is unrelated to the number that occurred this year, or that an incident occurring last week does not change the chance of one occurring next week). The expected time to the next armed incident does not depend on when the last one was, but just on the rate (e.g. if the average were 6 per year, we would expect one incident to occur in the next 2 months). For those of you into this sort of thing, I’ve assumed what we have here is a (stationary) Poisson process.

This may seem wrong – you may say, for example, that more armed incidents encourage copy-cat acts, and thus one incident is not independent of another. In fact, many processes we see day-to-day actually do often behave like this – including the goals scored in football games, the number of calls received by a call-centre per hour, and the incidents of the flu in different towns and cities. And in any case, what I am doing here is saying: wait, lets assume a world that is not getting more violent. In this world, we have the same expected number of armed incidents per year, although the actual number in a given year and their timing will vary (randomly). Each incident is independent, and the number occurring next year is independent of the number that occurred this year. That is, things are not getting any worse, but staying the same. In such a world, what could the pattern of armed incidents look like?

Running the model through once for a 10 year period and assuming an average of 6 armed incidents per year gives the results below. The first graph shows when each incident occurs over the 10 year period, and the second graph shows how many incidents occur per year.

AIgraph 1

Now, remember that this is all random – we have the same chance of events occurring in each year, and each event occurs randomly and independently of the previous event. In the model things are not getting worse, and one event does not cause (or inspire) more events. Even given this, we see what look like patterns – both trends (more incidents over time) and clusters (e.g. lots of events in years 8 and 9 very close to each other – it’s a crime wave!). In this example the number of incidents really do look like they are increasing – and the calls for a response (tougher laws, more surveillance, armed police, or whatever) could be expected to grow alongside. However, thinking that things are getting worse is actually wrong – in this example world, things are just the same as they’ve always been, and any variation is just down to plain chance.

Running the model for the next 10 years shows this clearly:

AIgraph 2

The number of incidents in falls from 9 in year 10 to 4 in year 11 and stays pretty low for 5 years afterwards, before a (completely random) uptick in year 17, 18 and 19.

So, a completely random process, where events (the armed incidents) are unrelated to each other, and where the chance of an event occurring does not change, produces something that looks like it has patterns. Where we see patterns, we tend to see meaning, and tend to try to produce a story to explain why. However, often there is no underlying reason – its all just chance. I could easily imagine a politician in year 9 or 10 campaigning to arm police (now!) to stop this dangerous development, and then to claim victory a couple of years later because, well, the number of armed incidents has of course gone down. And of course it is just rubbish – the changes were just due to luck, not the actions of the politician.

And in that case, making policy decisions, and especially fundamental decisions such as whether or not we should arm our police, on seemingly convincing patterns in the data record – such as a few years of increases in armed incidents – is misguided at best. It can be very tempting to try to find a reason why something is occurring, or to belive a reason you are given, but always, always be sceptical. It is often very, very hard to determine whether what you are seeing in the data is real or not (and just due to random chance). And its not just policy where we see this – it occurs in many walks of life, from picking prices in the oil or stock market, to justifying high wealth because I’m better in some way (rather than just lucky).

The point is not per se that it is random, its just that it is extremely hard to tell if it is or not. In that case, we should be very careful about basing big decisions on something that may simply just be an illusion.

RWC Round 4


Plugging the round 3 results into the model gives us the following ratings:

Ratings 5oct15

Not so much change this round, apart from the Australia’s surge into 2nd position. Australia are the clear form team of the RWC so far – their rating has increased 6 points since round 1. Interestingly, their home advantage score has declined, meaning that they are becoming are more formidable unit away from home (note that some of their headline rating increase is a result of this – ratings points shifting from the home advantage to the general rating as they have improved their away results). Ireland have also dropped 3 points following their narrow win over Italy.

Finally, NZ have not changed much at all, even despite being heavily criticised for their performance against Georgia. In fact, the result against Georgia was not too bad – the model picked a 41 point margin, whereas the final margin was 33 points (almost a “Green” result, actually). The so called minnows are not as bad as people expected – this is borne out in the results as well as the ratings (i.e. the general result that “minnow” games are closer than previous RWCs should not really have come as a surprise).


One interesting result of the model is that it can give us some indication of the home & away performance of the teams compared to their overall rating, and also (perhaps) of each team’s form. Looking at the prediction errors on the model’s training set, we can see that the ratings for some teams appear to over- or under-rate either their home or away performance somewhat. This is partially a result of the way the model fits ratings to results – the squared error means that big errors are penalised proportionally more than small ones, and the model may forgo symmetry in order to avoid big errors, even if this means that it gets some games slightly more wrong (the idea being that one big error is worse than several small ones).

This is very indicative only – it is really not much more than a intuitive interpretation of the model residuals (errors) and has nothing very scientific or statistical behind it. None-the-less it is interesting I think. Examining the results from the training set (which included all RWC games thus far), we see that most teams are rated fine for both home and away. However, for some teams the ratings may be skewed slightly. England for example, may be under-rated away – that is, they perform away slightly better (win by a little more or lose by a little less) than they should given their rating over the 2014-2015 period. Australia does the opposite.

form 5oct15

Note that, as the RWC games are double weighted, some of what we see here in the Away column (except for England) may result from this, with good wins in recent games providing the skew. That is, the away column could be interpreted as an indicator of recent form, with “Over” indicating that the round 3 rating is better than the earlier 2014-2015 results suggest, and “Under” the opposite. In that case, Australia and to a lesser extent Scotland are the form teams and Ireland is somewhat out of form.

Don’t put much emphasis on this – as I said this has very little solid behind it – but it can provide at least some more useful information when interpreting the model results and predictions.


And so to the final round predictions:

RWC Pool Round 4 Predictions

This round we have a model first – a draw prediction in the first game between Canada and Romania. The model really can’t decide between these two teams, so a draw it is…

Otherwise, in key match-ups deciding the QFs South Africa to comfortably beat the USA, Scotland to deal with Samoa, Australia to beat Wales (just), and Ireland to beat France (I disagree with the model on this one, but we’ll see). And England to go out on a win against Uruguay.

Rugby World Cup – Round 2 Review and Round 3 Picks

Just a quick update this time, as I am travelling. To recap round 2:

RWC Pool Round 2 Results

Again, not too bad. The model really only got one result badly wrong – Japan vs. Scotland, where it had picked a Japan win (albeit narrow). This highlights one of the downsides of this type of analysis – we do not really have that much data to go on for some teams, and so single results can have a big impact. Remember we are only using results from the past 2 seasons – going further back in time gives us more data to work with, but runs the risk of said data being out of date and not reflective of the teams’ rating now. We also picked the England-Wales game incorrectly, although that was always going to be close. The key difference (or so the model thought) should have been the home advantage to England – as it turned out, it may have been a disadvantage on the day…

Round 3 ratings and picks

The model has updated the ratings based on the results from round 2 (remember, RWC games count double):

Ratings 29sep15

The big changes are largely with the minnows shifting up or down depending on whether they lost by more or less than the model predicted in round 2. Scotland have improved by 4 points, although the model does not think this will be enough to save them against South Africa (see below). Japan drops by 9 – again, this is due to the limited data (number of games) to base the ratings on, which adds a certain amount of volatility. Interestingly, Argentina is continuing to improve, and is now rated just shy of Australia, England and Wales.

So, for the upcoming Round 3 games, the model’s picks are:

RWC Pool Round 3 Predictions

So, according to the model, the key games to watch are Samoa vs. Japan, and England vs. Australia. Again, the points difference for England is coming from the home advantage rating, suggesting a very close match is in store. Wales vs. Fiji may also be closer than the model suggests – it of course does not take account of injuries affecting a team’s rating.

Let’s see next week how things have gone!

RWC – Round 1 Review

If you recall, last week we published the predictions for round 1 of the RWC from our (fairly basic) rugby model. Let’s look how things went…

RWC Pool Round 1 Results

I’ve decided to group the predictions into 3 classes: green for those where the model picks the right winner and the margin prediction was within 7 points (i.e. a converted try) of the actual margin; blue for those where the right winner is picked, but the predicted margin is wrong by more than 7 points; and red for those where the model predicts the wrong winner (irrespective of margin). There is no great science behind this, but it provides us with an OK indicator of model accuracy without getting bogged down in too many numbers. Given the model itself is fairly approximate, it’s about right that the measurement of its effectiveness is fairly approximate too.

Apart from one glaring match, the model did not do too badly. Half of the games were rated green, and we even got one game bang on (USA v. Samoa). The model did not make the same mistake as many journalist pundits did (and continue to do) of under-rating Argentina – the model got that one pretty much right (and it would have been even better had the ABs not bombed so many tries…).

Of the two rated blue, the Ireland Canada game was difficult because of the large gulf between the teams’ ratings and the final score – any model will struggle to get a blow-out to within 7 points. And the model was actually out by only 9 – pretty close to green actually.

The two games the model did not predict accurately were the two surprises of the round – Tonga v. Georgia and the biggie, Japan v. South Africa. The model was not alone in not correctly predicting the Japan game – and I include the South African management in that.

So, a pretty good start. Round 2 starts tonight – I will update the model to take account of the round 1 games, and make some more predictions!