Committee results - actual versus expected

[Phil]

While we are waiting for the Selection committee, I thought I would share some analysis I've done of previous results.

The details are on a Google Doc spreadsheet. If I did it right, it should be accessible to anyone here.

I'll copy the conclusions here:

Intro
Higher seeded team are expected to beat lower seeded teams. Obviously, it doesn't always work out that way, but when a team is seeded #1, a committee has judged that they are better than, say, a 16 seed, and more likely to win.
We can use this information to assess how well the committee does it's job, and we can assess how well a group of team (for example, from a conference) does against expectations.
Committee assessment
If a single 12 seed beats a 5 seed, we wouldn't necessarily say the committee erred, we might call it an upset. However, if the 12 seeds beat the 5 seeds consistently, we would conclude that the seeding is flawed.
One of the usual questions about the committee is whether they selected the right number of team from each of the conferences. A related question is whether the number of teams from the six power conferences in aggregate is too many or too few. We can look at the actual versus expected results to gain some insight into the question.
Conference Performance
A crude metric is to look at the win-loss record for the conference. However, imagine two conferences, each with a 1-1 record. On the surface,they had identical results. However, if the first conference had a 3 and a 6 seed, while the second conference had a 13 and a 14 seed, we would view the results differently. The first conference was "supposed" to win their first round, and should have a second round win before losing. In contrast, the other conference was expected to lose both first round games. If we look at expected versus actual, we should conclude that the first conference had disappointing results, while the second conference exceeded expectations.
Looking at expected number of wins versus actual wins helps us see which conferences did well, given their seeding and which ones didn't do so well.
The calculations
The calculations are straightforward. One can calculate the expected number of wins for each seed. Any team seeded 9-16 is not expected to win a game. an 8 seed is expected to win one game, then lose to the 1 seed, etc. The only issue not perfectly obvious is what to do with the 1 seeds. They are expected to win until they reach the Final Four. For simplicity, I won't rank the overall tops seeds, I will treat them equally. That means each 1 seed has a 50% chance of winning the semi-final and a 25% chance of winning the NC
The expectation are shown to the right:

 

[Phil]

Results
At present, I have a tab for 2010 and 2011
Go to the tabs the bottom to see the results for each year.
The box at the top shows the summary, while the detials necessary to do the calculations are shown below, and to the right.

My Conclusions
2010
In 2010, the big six conferences had 51 of the 63 wins, versus and expectation of 57. That's not bad, it does lend support to the claim that the big six are the power conferences, but it does hint that maybe there were a few more Big 6 teams versus other conferences than was warranted.

Conference by conference, some of the results were impressive. The ACC was spot-on, with six expected wins and six actual wins. The PAC10 and SEC couldn't be any closer, with 5.75 and 10.75 expected wins respectively, versus 6 and 11 actual. The BE slightly under performed, but by less than a single game. The B12 under performed, and may have been slightly over seeded, while the B10 performance was quite below expectation (recognizing sample size, one has to be cautious about drawing too strong a conclusion about an expectation of 6 wins and an actual of 3.)

2011
The 2011 results are very interesting. Overall, the committee just about nailed it. The expectation were that the Big Six conferences would win 58 games and the won 55, only off by three. The expectations were almost identical to the prior year, but the actual results came in closer. The weak case the non-big six might have for not getting enough respect in 2010 is now weaker. The committee is getting it just about right.

Another notable event in 2011 was the decision to invite nine BE teams, versus only four SEC teams. The results totally corroborated the committee decision, if anything, the BE significantly over performed against expectations, and maybe deserved more teams or better seeds.

Every other one of the Big Six conferences slightly under performed, while the Big East had nine teams, an expectation of 13.75 wins, and ended up winning 19 of the 63 games.

 

[DobbsRover2]

Well. I think the results do show that the Big 6 conferences do represent themselves reasonably well and about as expected. But it is kind of a self-fulfilling prophecy, with the brackets set up to support the assumptions of the committee. The question every year is, are those assumptions correct?

Take my local team Marist for example. The Red Foxes have dominated the MAAC tournament for 7 straight years. The first year in the streak that they danced was in 2006, and as the new fox on the block, they were given a #14 seed and thus had to play a #3 seed who beat them by 15, while a more established Hartford team got a #11 seed and won its first round game. The Hawks had 3 more wins pre-tourney, and maybe they were really better by 3 seeds, but you can question whether Marist could have also won the opener as a #11 seed because ......

By 2007, Marist was a more known quantity and the committee gave them a big promotion to a #13 seed. One place better and the Red Foxes went on a tear by winning their opener by 4 against a #4 seed and then beating a #5 seed by 14, before losing by 19 to #1 UTenn in the 3rd round. Perhaps they were hugely better in 2008, because they were given a stupendous #7 seed and were thus expected to win their first round game. They crushed in the opener by 19, and then lost in the next round to the #2 seed by 19.

In 2009 and 2010, Marist was back in double-digit seeds at #12 for both years, and they lost to #5 seeds by 7 and then 20. But in 2011, they were bumped up to a #10 seed and won the opener by 10 before losing to #2 seed Duke by 5.

By expectations, Marist should have gone 1-6 instead of 4-6 in the tourney since 2006. And they shouldn't be losing to a #2 seed by only 5 as a #10 seed. I really have no idea what to have expected for the Red Foxes during the six tourney runs, but I know they exceeded expectations and my suspicion is that they could have done even better if the committee in many years didn't think of them as just "that MAAC champion" and rather as a very dangerous team to the mid-range big 6 teams.

The analysis and spreadsheet mentions mainly the big 6 conference contribution to wins, but I would look more at the other group. In 2010 despite getting few high seeds the non-big 6 teams won 12 games instead of 6, with a big shout out to Xavier, who almost upped that total even more and fullfilled the 3 wins expected of a #3 seed. But 12 rather than 6 is double the expected, and that's really huge. The committee apparently blew it on the non-big 6 teams that year.

In the end we'll never know how well the non-big 6 teams would do if they got more respect, and perhaps it doesn't matter, because it's not going to change significantly. Marist has won 4 tourney games in the last 6 years with only one single-digit seed, while Texas has won just 1. Marist got a #13 seed this year, Texas got a #9. Except for the fact that one is non-big 6 and other is big 6, this I do not grok.

 

[wire chief]

Dobbs, you make a good case, reminiscent also of the Green Bay flap.

 

[CamrnCrz1974]

Phil, this is really some fantastic work on your part. Well done!

 

[Phil]

DobbsRover, thanks for your response. I concentrated on the Big six, mainly because I started with a question about the relative strengths of the BE and SEC over time, and it was easy enough to extend to the other conferences.

The same concept can be applied in different ways, although it becomes harder to draw conclusions when you look at an individual team, rather than a larger group. I'm a fan of Giorgis and Marist, and agree that Marist has performed very well and probably should have gotten better seeds faster. If you didn't know the circumstances, winning four and losing six doesn't sound like something to write home about, but when you take into consideration the seeds, and see that they were expected to win one, yet were victorious four times, that become more interesting, and is exactly the type of thing that can be done.

I only had two seasons, but wanted some feedback before doing more. With more years, you can start to look at individual teams, not just conferences, so you've encouraged me to extend this, not just for this year, but back into the past.

 

[UcMiami]

Phil - very interesting and Dobbs - very good response. As they say numbers are funny things.
Given the nature of the women's game where the differentiation between the top end and the middle is so huge, I think it might be interesting to look at a smaller sampling - dropping out the obvious mismatches. In the first round dropping out the 1/16, 2/15, 3/14, 4/13 games - these are generally so lopsided that the committee should nail these every year. In the second round dropping out the 1/? and 2/? games. Third and fourth rounds drop out the games where a seed above 6 made it through by upset. And in the FF drop out anything above a 3 seed game.
I realize your initial analysis was a comparison of the BCS conference seeding/strength. I think Dobbs points to a valid issue that maybe doing an agragate of the mid-majors would add a little perspective (I think leaving out the bottom conferences is important as they generally have no chance.) To look at the committee work in identifying correct seedings you would need to narrow the field to the area where they are most likely to screw up i.e. the differences between groups of teams within a few seeding lines of each other.

 

[Phil]

I like the idea of grouping the mid-majors. Is there a definitive list, or is this subjective?

I also agree that there's a difference between a 1/16 game and an 8/9. I think we all agree that a 9 beating an 8 barely qualifies as an upset, and doesn't really indicate that the seeding was wrong, while a 1/16 is different.

A 1/16 upset would be just an upset, but four or even three such events could only happen with poor seeding. In contrast, if three or four nines, beat an eight, it is mildly interesting, but not really a black mark for the committee. I'll mull over whether there's an easy way to adjust for this.

 

[jsloanjr]

Typically, the chalk wins more than loses in the 1st 2 rounds. The regionals are were you see the good teams separate or they we overrated by the committee. IMO, there are a few 3 seeds that should be 2 seeds.......Looking forward to a fun ride to Denver........

 

[DobbsRover2]

Just wanted to note that on the men's side last night, the "Marist-like" VCU just pulled another huge and totally expected upset as a #12 seed -- no surprise after almost making it to the NC game last year. As Rodney would have said, "Sheeeesh, they don't get no respect."

This data mining is all very complex stuff fraught with sometimes invalid assumptions, fuzzy boundaries, and black swans. Completely agree that looking at the performances of teams that are #13 - #16 seeds is not that valuable since they are set up to lose, when some might pull out wins against say a #8 seed. Except in rare circumstances though these teams are first round bow-outs, and by expectations rightly so. But then a red fox appears in black swan clothing and wins 2 games as a #13 seed, and that team should be included in the comparisons. In effect, there are teams such as Marist and Green Bay that you need to always include in analyses because they play well above the committee's assigned seed. So the mid-major issue is tricky. Also, some conferences like the MAAC would not usually be considered one of the more major mid-majors, but when its represented by a dominating Marist, it really is.

But going back to the original intent of the spreadsheet and analysis, I'm all for reports that show how the BEast and SEC have done in tournament play during this century.

 

[Phil]

I updated my analysis to break out the mid-majors specifically. It does put a different cast on the results. While the power conferences still dominate, the fact that the mid-majors won ten games while seeded to win five does strongly suggest that the mid-majors are still not getting the respect they deserve.

I haven't yet updated 2011 (I did catch a miss in the ACC, I missed the 8 seed of VA, now updated.)

I also added some graphs.

2010 results