RWC Quarters Review and Semi Final Predictions

A great weekend of rugby, almost predicted by the model (well, sort of). The model picked the right winners in 3 out of four games, but got the winner of the fourth game – Argentina v. Ireland – wrong. However, its not as good as it seems (for the model, anyway). There was only the Wales v. South Africa game where the model come within the 7 point margin. The other three matches, well, the points difference was not in the same ballpark. A forecast comfortable win for the ABs turned out to be a blow-out in their favour. The Ireland-Argentina game was expected to be very close, but a skilful Argentina side put paid to that and blew away the Irish. And Scotland – Australia, well, the model has to thank a certain refereeing decision for helping to ensure it got at least the winner right…

QF results

Taking account of these results on the ratings, we get some reasonably significant changes:

Ratings 22oct15

New Zealand rise 2 points to 104, France continue their recent fall to 81, Australia drop 1 to 94, Scotland are up 1 to 78, South Africa rise by 3, Wales fall by 1, Ireland drop by 2 and Argentina rise up into second in the ratings at 95. There are some other movements also, as the ratings are re-adjusted for the remaining teams, as a result of the quarter final results (and the fixing of a score for one of the USA games – my bad).

So, what then for the semi-finals? Throwing the match-ups into the model we get:

South Africa v. New Zealand. Winner: New Zealand by 10.

No doubt in the model’s opinion – NZ to run out clear winners by two scores. Let’s hope the maths are right!

Argentina v. Australia. Winner: Argentina by 1.

This is the surprise prediction – Argentina ahead (just) of the Wallabies. In fact, it should be even closer – the model cannot decide if the rating for Australia is 94 or 95. I chose 94, as 94.5 just makes no sense, and I wanted to produce a winner rather than a draw. So Argentina it is. Of course, the model takes no account of things like poor performance or returning players. So, if you think the return of Pocock will boost the Aussies, and that the Scotland game was an aberration, then it may be fair to expect an Aussie win. But it looks like it will be very, very close.

RWC Quarter Final Predictions

It’s QF time, and the model is itching to make some predictions. Being an algorithm, it is not subject to emotion, nor any references to 2007 (as, of course, results from 2007 fall outside its training data set). So, no French hoodoo then, and a predicted victory for the ABs (for those who don’t know what I am talking about, France defeated the All Blacks in the QFs in 2007, and the fixture is repeated this year).

QF predictions

Australia is also predicted to triumph, with a victory margin of 19 points. However, the South Africa vs. Wales and Ireland vs. Argentina games are almost too close to call – the model does pick South Africa and Ireland respectively, but only by 1 and 2 points respectively. Into the mix here come other variables – nearly a home advantage for the Irish? Injuries to Wales and Ireland reducing their effective ratings? The model does not take account of these (and even if it did, I doubt there would be the data to support such a model)

So, a fine weekend of rugby in store!

RWC – Round 4 and Pool Review

The Results

Round 4 (and thus all pool games) is done and dusted, so let’s see how the model went:

RWC Pool Round 4 Results

So, Round 4 was definitely the model’s best – the winners of all games predicted correctly, and 9 out of 12 within 7 points! Note here I have counted the Canada-Romania prediction as a success, as although a draw was forecast, the end result was within 3 points. Disagree if you will, but the model agrees wholeheartedly with my decision…

Some interesting results that I would like to highlight: The NZ vs. Tonga game ended up pretty much as predicted – or at least, bang-on according to the form book. Perhaps that is a counter to those who have been critical of NZ’s performance? Ireland did better than expected against France, although whether than is because Ireland are playing better than their rating or France are playing worse is the question (and one that we answer with the end-Round-4 ratings – see below). Similarly South Africa vs. USA – a big victory, but perhaps again due to an over-rating of the USA rather than an under-rating of SA. And Scotland struggled much more than the model predicted to overcome Samoa. This highlights one of the weaknesses of the model: performance is variable, for every team. Just because a team has a rating of X does not mean that they cannot over- or under-perform in any given game. And in this game Samoa played pretty well (and really should have won).

Betting with the model

As an exercise, I have simulated a betting strategy using the model predictions. Based on the model prediction, for each game a 10 GBP simulated bet was placed on the predicted winner with a margin of either “12 or under” (if the prediction was 12 or under), or “13 and over” (if the prediction was 13 or over). For those games between top teams and minnows where these odds were not offered (as the victory margin was expected to be much higher), the bet with a 20 point spread than included the predicted margin was made (so, for example, if the prediction was a 52 point margin to team A, then the “Team A by 41-60 points” spread bet was chosen).

And, a bit to my surprise I must say, the model did fantastically well. The net position of the simulated account ended up just under 300 GBP over the pool phase, and was remarkably not particularly volatile. Ah, the power of algorithms…

Pool Betting Results

Ratings: End of Pool Play

Based on the results of the last round of pool games, I have re-run the model to update the ratings:

Ratings 12oct15

Not much change here over the ratings at the conclusion of round 3, perhaps indicating we are reaching a bit of stability and have a pretty good idea (based on results) of current rating levels. Of the teams in the Quarter Finals, France and South Africa are the big movers of the week, down 2 and up 2 respectively. So I guess this answers the question I raised above, of whether Ireland performed better than expected or France worse (well, in the model’s opinion anyway). Interestingly, Australia’s win over Wales was not of sufficient size to move the ratings of either team.

Ratings: Big Movers

I have somewhat arbitrarily defined a “big mover” ratings-wise to be any team that has moved more than 6 ratings points from its rating at the start of the RWC at any point during pool play. There are 9 big movers under that definition, as we can see here:

Ratings biggest movers

Aside from Argentina and Australia, all of the big movers are the smaller nations. This should not surprise – as the smaller nations do not play the bigger nations all that often, we have less data with which to calculate their ratings, and the overall levels are more difficult to predict in advance of the competition. So you see the impact of Japan’s victory over South Africa, which after the pool rounds looks to have been an extraordinary performance (or an extraordinary game plan) rather than an indication of significant improvement by Japan. Georgia’s jump after round 1 looks largely due to an over-rating of Tonga – along with Samoa they have performed much worse than expected in the RWC.

And Australia and Argentina are the clear form teams of the tournament, if we define “form” as performing better than was expected at the start of the comp.

Ratings: The Quarter Finalists

Finally, I thought I would extract the ratings and how they have developed over pool play, for each of the 8 quarter finalists:

QF teams ratings

Interestingly, all 8 teams are either rated the same or better than their rating at the start of the tournament. Individual performance variation adds a little bit of noise, with the ratings easily adjusting by one or two points with a big win or unexpected loss. Apart from Australia and Argentina however, the overall movements are small, and we can essentially say that the other 6 teams are performing pretty much as expected given results over the past couple of years. Including NZ.

In terms of ratings then, we have NZ on top, followed by Australia, Ireland and South Africa are equal third, ahead of Wales, Argentina, France and Scotland. In a couple of days we will see what this means for the Quarter Finals predictions.

Until then!

RWC Round 4

Ratings

Plugging the round 3 results into the model gives us the following ratings:

Ratings 5oct15

Not so much change this round, apart from the Australia’s surge into 2nd position. Australia are the clear form team of the RWC so far – their rating has increased 6 points since round 1. Interestingly, their home advantage score has declined, meaning that they are becoming are more formidable unit away from home (note that some of their headline rating increase is a result of this – ratings points shifting from the home advantage to the general rating as they have improved their away results). Ireland have also dropped 3 points following their narrow win over Italy.

Finally, NZ have not changed much at all, even despite being heavily criticised for their performance against Georgia. In fact, the result against Georgia was not too bad – the model picked a 41 point margin, whereas the final margin was 33 points (almost a “Green” result, actually). The so called minnows are not as bad as people expected – this is borne out in the results as well as the ratings (i.e. the general result that “minnow” games are closer than previous RWCs should not really have come as a surprise).

Form

One interesting result of the model is that it can give us some indication of the home & away performance of the teams compared to their overall rating, and also (perhaps) of each team’s form. Looking at the prediction errors on the model’s training set, we can see that the ratings for some teams appear to over- or under-rate either their home or away performance somewhat. This is partially a result of the way the model fits ratings to results – the squared error means that big errors are penalised proportionally more than small ones, and the model may forgo symmetry in order to avoid big errors, even if this means that it gets some games slightly more wrong (the idea being that one big error is worse than several small ones).

This is very indicative only – it is really not much more than a intuitive interpretation of the model residuals (errors) and has nothing very scientific or statistical behind it. None-the-less it is interesting I think. Examining the results from the training set (which included all RWC games thus far), we see that most teams are rated fine for both home and away. However, for some teams the ratings may be skewed slightly. England for example, may be under-rated away – that is, they perform away slightly better (win by a little more or lose by a little less) than they should given their rating over the 2014-2015 period. Australia does the opposite.

form 5oct15

Note that, as the RWC games are double weighted, some of what we see here in the Away column (except for England) may result from this, with good wins in recent games providing the skew. That is, the away column could be interpreted as an indicator of recent form, with “Over” indicating that the round 3 rating is better than the earlier 2014-2015 results suggest, and “Under” the opposite. In that case, Australia and to a lesser extent Scotland are the form teams and Ireland is somewhat out of form.

Don’t put much emphasis on this – as I said this has very little solid behind it – but it can provide at least some more useful information when interpreting the model results and predictions.

Predictions

And so to the final round predictions:

RWC Pool Round 4 Predictions

This round we have a model first – a draw prediction in the first game between Canada and Romania. The model really can’t decide between these two teams, so a draw it is…

Otherwise, in key match-ups deciding the QFs South Africa to comfortably beat the USA, Scotland to deal with Samoa, Australia to beat Wales (just), and Ireland to beat France (I disagree with the model on this one, but we’ll see). And England to go out on a win against Uruguay.

RWC – Round 3 Review

A pretty good round for the model, with only one game being picked incorrectly (England vs. Australia), 5 to within 7 points and 3 correct winners but wrong margin. Interestingly, we are seeing a couple of teams – England, Samoa are the two main examples – who appear to be under-performing when compared to recent results (and thus their rankings), and a couple of teams such as Japan and Australia who seem to be over-performing. This highlights one of the difficulties with this type of modelling – how do you account for form (let alone what is form – and is it simply “meaning” given to short term random variation) and squad development. Any model based on history will only pick-up rapid rating changes after the fact – and this is what we may be seeing here.

In any case, the results from round 3 are as follows:

RWC Pool Round 3 Results