Optimeering increasing efforts in bringing advanced AI to the power markets

For the fourth time in five years, a model development project in Optimeering has been granted funding from the Research Council’s ENERGIX program – thereby providing acknowledgement and key financial support in making advanced AI accessible to decision makers in the energy market. 

This year the funding has been granted in order to develop machine learning models to support the regulators in the operative surveillance of trading activities in physical power markets.  

Drop shipping and optimisation – how to save retail?

Drop shipping – where retailers don’t stock a product, but rather obtain and ship it directly from the manufacturer only once the customer places an order – can be a boon for on- and offline retailers and manufacturers alike. A recent article suggests that profitability of manufacturers that use drop shipping is over 18% higher than manufacturers who do not. This suggests there is money “on the table” associated with order fulfilment, and new approaches to shipping can help get that money into the pockets of retailers and manufacturers alike.

Unfortunately, it’s not as simple as saying “yep, let’s use drop shipping”. For one thing, customer service needs to be maintained – not many customers are going to accept long shipping delays for example just to increase supplier profitability. Drop shipping needs to deliver a result as good as the alternatives. This comes down essentially to logistics, modelling, machine learning and optimisation – for anticipating demand, preparing production and shipping, analysing and predicting customer service requirements, and executing the production and shipping process that delivers fast but at minimum cost. This is where Optimeering can help – drop us a line if that 18% sounds interesting to you…

It also opens up some very interesting new business models. Large manufacturers or wholesalers take on the roll of production, storage (if any) and order fulfilment, and (online and offline) retailers focus on their specific customer concept – whether that is built around price, service, products offered or something else entirely. This is already beginning – online supermarkets are and will have to move in this direction rather than each of them replicating the fulfilment job. and, of course, good algorithms (like those made by Optimeering!) are essential to make it work, and work well.

Speeding up Mosel code – the sum operator

For those of you who don’t know, Mosel is the programming and modelling language that ships with FICO’s Xpress math programming solver. And if you don’t already know what it is, the rest of this blog post won’t be of too much interest – I would advise a cup of coffee, a stretch and a good eye-relaxing look out the window instead. But, for those of you that do use mosel, keep reading.

At Optimeering, we don’t actually use mosel much any more – pretty much all our modelling work is now done in python, including all our math programming modelling. However, when we were using mosel, we noticed a few issues with speed – particularly, how you use the conditional operator has a big impact on execution speed. We first really experienced this in the early prototyping of the PUMA model – the model’s datasets can get pretty big, and issues like this were very critical to good model performance.

So, last year we published a working paper within the PUMA project, with code, that explains things in detail and makes some recommendations regarding how you should use the conditional operator. I thought it was about time I made it available here – download and disseminate away!

Data Science Cheat Sheets

We are big fans of the team at DataCamp and the python and data science resources they’ve put together. Their Cheat Sheets are really useful – especially for people like myself who forget the correct syntax all the time. No matter whether you are new to python and data modelling, or have heaps of experience and are a python data science animal (if so, drop us a line…), check them out.

Quota Market Modelling

Quota markets are increasingly used to ration power market and environmental goods, including capacity, emissions permits and renewable subsidies. The obvious and most international of these markets is the EU ETS; however there are many more including the Swedish-Nordic El-Cert market, and the increasing number of capacity auctions and certificate markets popping up in Europe and elsewhere.

One characteristic of these markets is that the good that is traded is “artificial” – and typically the demand for it has been legislated by government. This means that the demand for the good is defined by rules, not by want or need, and thus behaves quite differently to “normal” markets. Quota markets are often either long (there are too many quotas or certificates) to meet the legislated demand, or short (there are not enough). If long, the price is zero. If short, then the price is set by a legislated penalty for not having enough quotas. Pricing in between is due to uncertainty – you don’t know whether the market is long or short. This dynamic results in behaviour like big swings in prices, or – often – price hikes, then price collapses as the market looks long, followed by rule changes to support price levels.

This means that such markets are not able to be analysed that effectively by traditional economic models that are based on the idea of equilibrium. Quota markets are not in equilibrium, so should not be analysed as if they were.

Instead, at Optimeering we attack this problem via the use of intelligent agent-based models, that combine AI techniques with modelling of actual market actors to simulate market behaviour under realistic market conditions. Together with Thema Consulting Group, we developed the MARC model for the Swedish-Norwegian El-cert market, that has been used by a range of developers, regulators and operators to better understand and predict future market outturns. We have a blog about the MARC model here (in Norwegian). To learn more how we can use agent modelling to help you better understand the quota markets that drive your bottom line, contact us here.

Seriestart – hvor viktig er første kamp?

Seriestarten i Tippeligaen 2016 er over. Eksperter, supportere, spillere og trenere trekker seg tilbake for å analysere hva som gikk galt, hva som gikk bra og muligens også hva første runde kan si om prestasjonene fremover. Er virkelig Vålerenga så dårlig som DB vil ha det til, hva med Bodø Glimt og Sogndal – er forskjellen så stor som tabellen etter én runde vil ha det til? Vi er hverken eksperter, spillere eller trenere, men har fundert om første runde kan si noe (statistisk) om hvordan resultatet blir seende ut etter siste runde i november.

Spørsmålet vi har stilt oss er: «hvor viktig er egentlig første runde?» Eller enda mer konkret: «Er det en sammenheng mellom resultater i første runde og endelig plassering etter siste runde?» I analysen har vi samlet data fra 2009 (fra 16-lags-serien ble innført) til og med forrige års sesong.

I figuren har vi plottet alle sluttplasseringer (y-aksen) mot antall poeng i første runde (x-aksen). Eksempelvis representerer sirkelen øverst til høyre 3 poeng i første runde og første plass etter siste serierunde. Størrelsen på denne er bestemt av antall ganger dette har inntruffet siden 2009. I dette tilfellet seks av syv ganger.


Vi ser at 3 poeng i første runde normalt sett har gitt en høyere tabellplassering enn 0 og 1 poeng ved at 3-poengs-kolonnen har størst sirkler på den øverste halvdelen av tabellen. Median-plasseringen for vinnere i første runde er 7. plass. For uavgjortslag er medianplasseringen 11 og for tapslag er den 9. Matrisen under viser sammenhengene på en litt annen måte.


I tabellen vises tabellplassering etter siste serierunde vertikalt (fra 1-16) og sesonger horisontalt (fra 2015-2009). For hver kombinasjon av sluttplassering og sesong har vi registrert antall poeng i første runde {0,1,3} og fargekodet med grønn for 3 poeng, gul for 1 poeng og rød for 0 poeng.

Tabellen viser samme resultat som figuren over – at tre poeng i første runde normalt sett har gitt en høyere tabellplassering etter siste runde. 2011-sesongen skiller seg ut som «unntaket som bekrefter regelen». Da for eksempel Sarpsborg 08 slo Molde i første serierunde, men Molde ble seriemester og Sarpsborg 08 rykket ned som det dårligst plasserte laget.

En viss sammenheng vil det nok være, ettersom 3 poeng i første runde er 3 hele poeng på veg mot seriegull, men det anbefales å lese tabellen og figuren mer for underholdningens skyld enn for signifikante sammenhenger. For eksempel kan vi neppe si at for å få 4. og 8. plass, er det viktig med 3 poeng i første runde. Og til alle Vålerenga-fans – dere kan fremdeles bli seriemester 2016!

Figurene under plotter samtlige sluttplasseringer for henholdsvis 3, 1, og 0 poeng i første runde:




Ser vi på samme analyse, men mellom første runde og plassering på tabellen etter fem runder ser vi en mye klarere sammenheng.



Etter fem runder, har et lag som vant første runde, historisk aldri vært dårligere enn 11 plass etter fem runder. Og laget som toppet serien etter 5 runder har alltid fått 3 poeng i første runde.

Med denne historiske sammenhengen, samt resultater fra treningskamper i 2016, har vi laget en liten simuleringsmodell som vi har brukt til å tippe serietabellen etter 5 runder. Her har vi kun sett på vinnprosent og ikke sett på (for eksempel) hvem som var motstander i treningskampene, eller om treningskampen var tett opp mot seriestart eller om det var rett over nyttår.

Tabellen etter fem runder blir da slik:

1 Haugesund
2 Bodø/Glimt
3 Lillestrøm
4 Start
5 Odd
6 Aalesund
7 Rosenborg
8 Vålerenga
9 Tromsø
10 Molde
11 Viking
12 Strømsgodset
13 Sarpsborg 08
14 Sogndal
15 Brann
16 Stabæk

Det første vi har gjort er å regne ut for hver plassering (etter fem runder) hyppigheten av seier, uavgjort og tap i første runde. Simuleringsmodellen trekker så tilfeldige resultater basert på disse resultatene. Videre har vi rangert de tre gruppene (seier, uavgjort, tap) basert på resultater i treningskamper. Ettersom det kun har vært førsterundevinnere på topp etter 5 runder historisk, vil modellen vår også velge en førsterundevinner på topp. Grunnen til at Haugesund er på topp er fordi de har høyest vinnprosent i treningskampene. Brann havner eksempelvis på 15 plass fordi simuleringsmodellen har trukket ut et uavgjortslag fra første runde til å havne på 15 plass basert på historien. Videre har Brann gjort det dårligst av alle uavgjortslagene i treningskampene og havner derfor lavest av alle uavgjortslagene. Samme logikk gjelder for Rosenborg – modellen velger ut et taperlag til å havne på 7.plass (har skjedd to av syv ganger) og at Rosenborg er det laget med best trengingskamphistorikk av taperlagene.

Ny bloggpost følger i april etter femte runde er ferdigspilt

Our Magic Oil Price Forecast – An Update

Given the various ups and downs in the oil market recently, I thought it was about time I updated our magic forecast from last January. For those of you who haven’t read my original post, I few years ago I made a “magic” oil price forecast in order to highlight the some typical problems the crop up again and again in many forecasts and price predictions (and especially those we see in the media). I chose oil because it is a particularly egregious example… and I made a “magic” forecast from fitting a sine curve to historical oil prices for no other reason than it seemed like a good idea at the time (and the prices looked sort of sine-curvey if you squinted a bit). The idea was then to see how this pretty rubbish forecast compared to those from the professional oil analysts that populate our banks and finance houses…

Somewhat to my surprise, the Optimeering’s magic sine curve has, um, done rather well. And much better than the expert analyst predictions. So here is an update – with historical prices converted to 2015 USD, and the latest prices added and compared to the model. Bad news, oil people and Norway, the bottom has not been reached – well, at least it hasn’t if you believe in magic (oil price forecasts) anyway…

updated forecast Jan 2016

Our new Oslo office

As of 1st of December our Oslo office is now located in Tollbugata 35, where we are joined by Econ Management Consulting, Snow Dragon and Sigra Group. And yes, it is just around the corner from our old place in Akersgata 1. We are still in the moving in process (so half the furniture is still to arrive…) but we are getting there. We look forward to welcoming our existing and new clients to Tollbugata!

Tollbu photo 1

Tollbu photo 2

Tollbu photo 3



RWC Quarters Review and Semi Final Predictions

A great weekend of rugby, almost predicted by the model (well, sort of). The model picked the right winners in 3 out of four games, but got the winner of the fourth game – Argentina v. Ireland – wrong. However, its not as good as it seems (for the model, anyway). There was only the Wales v. South Africa game where the model come within the 7 point margin. The other three matches, well, the points difference was not in the same ballpark. A forecast comfortable win for the ABs turned out to be a blow-out in their favour. The Ireland-Argentina game was expected to be very close, but a skilful Argentina side put paid to that and blew away the Irish. And Scotland – Australia, well, the model has to thank a certain refereeing decision for helping to ensure it got at least the winner right…

QF results

Taking account of these results on the ratings, we get some reasonably significant changes:

Ratings 22oct15

New Zealand rise 2 points to 104, France continue their recent fall to 81, Australia drop 1 to 94, Scotland are up 1 to 78, South Africa rise by 3, Wales fall by 1, Ireland drop by 2 and Argentina rise up into second in the ratings at 95. There are some other movements also, as the ratings are re-adjusted for the remaining teams, as a result of the quarter final results (and the fixing of a score for one of the USA games – my bad).

So, what then for the semi-finals? Throwing the match-ups into the model we get:

South Africa v. New Zealand. Winner: New Zealand by 10.

No doubt in the model’s opinion – NZ to run out clear winners by two scores. Let’s hope the maths are right!

Argentina v. Australia. Winner: Argentina by 1.

This is the surprise prediction – Argentina ahead (just) of the Wallabies. In fact, it should be even closer – the model cannot decide if the rating for Australia is 94 or 95. I chose 94, as 94.5 just makes no sense, and I wanted to produce a winner rather than a draw. So Argentina it is. Of course, the model takes no account of things like poor performance or returning players. So, if you think the return of Pocock will boost the Aussies, and that the Scotland game was an aberration, then it may be fair to expect an Aussie win. But it looks like it will be very, very close.