From: "Jeff Bihl"
Subject: On Ranking Violation Percentage
Date: Fri, 5 Jan 2001 19:18:46 -0500
The problem with picking one criteria to judge all retrodictive systems
by is that not all retrodictive systems are written to meet the same
criteria. In my opinion the purpose of a retrodictive type system is to
be sure that an individuals criteria are applied exactly the same to all
teams. The only way that ranking violation percentage is a fair criteria
by which to judge two systems is if two individuals used minimizing that
percentage as their criteria for their system.
The main reason why I do not believe in minimizing ranking violation
percentage as a criteria in my own systems is that is that when you rank
teams in this manner a game will either matter a lot (move the team that
won ahead of the team it defeated) or it won't matter at all and be
totally ignored. The biggest problem with this in the long run is that
it does not punish really bad losses enough nor does it reward really
big wins enough. A strange fact is that a really big win (beating a team
ranked much higher than you) has a greater chance being ignored than
small one (beating a team just above you). I will demonstrate this
later.
The other problem I have is than you stand absolutely nothing to gain by
playing a team ranked below you and absolutely nothing to lose by
playing a team ranked ahead of you. In my opinion if there is something
to lose there should be something to gain and vise versa. I try to keep
risk and reward balanced.
Here is a demonstration to illustrate my point. Lets make five teams
named A, B, C, D, and E. Those five teams have played round robin and
split with each other all going 2-2. Lets set up another group of five
teams which will be teams F, G, H, I, and J who have played round robin
and split with each other all going 2-2. Now let's say that all the
teams in the first group (A-E) went 5-0 in their games in a full round
robin against the other group (F-J). As Mark has noted, a team's record
can be used effectively as a retrodictive rating in pro sports because
the schedules are usually balanced. In this particular ten team group
winning percentage works perfectly as a ranking because all the
schedules are perfectly balanced. They all played each other one time.
So a system whose goal was to minimize the number of upsets in the
standings would have the same results as the conference standings:
1. (tie) A, B, C, D, E 7-2
6. (tie) F, G, H, I, J 2-7
Each group is still essentially tied in a system whose criteria is to
minimize ranking violation percentage because you can rank the teams in
each group of 5 in any order and you will get the same number of
violations. There is no way to separate them by this method alone. I
will say they are tied by this method.
Now lets add an 11th team, team K, to the mix. Team K beats teams A, B,
and C in its first 3 games. Now team K is number 1 in the system whose
goal is to minimize ranking violation percentage. Teams A through E are
still essentially tied for #2. There is still no way to separate the 5
(A-E) by this method alone. A, B, and C would not be punished for the
loss at all. Teams F-J are still tied for last.
Let's consider to what happens to the rankings of a system whose goal is
to minimize ranking violation percentage in a few scenarios for team K's
remaining games.
Example 1: Team K loses to team D in its fourth game.
Team D is #1, Team K is #2, Teams A, B, C, and E will be ranked
according to who beat who in the 2-2 round robin split, and F through J
are still tied for last. As you see Team D was helped by a decent win
and team K was hurt by a decent loss.
Example 2: Team K loses to team F in its fourth game.
Team K is not hurt by this loss and team F is not helped. K is still #1,
A though E #2, and F through J tied for last. There is still no way to
separate F-J by this manner. So we see that team K is hurt by a decent
loss but is not hurt at all by a bad loss. Team D was helped by a good
win in example 1 but team F could not be helped at all by a great win.
Example 3: Team K loses to teams F and G in its fourth and fifth games.
Just like the fourth game against F, the fifth game against G doesn't
change the rankings at all.
Example 4: Team K loses to teams F, G, and H in its fourth, fifth and
sixth games.
This is the strangest situation of all. Team K has lost three games to
above average teams and lost three games to teams that are exactly as
far below average as the teams that they beat are above average. So they
have an even record against an exactly average schedule. One would think
that they would have an average ranking. That would mean being ranked
6th among the 11 teams (behind A through E and ahead of F through J). In
the system whose goal is to minimize the number of past upsets average
is the one ranking that they CAN'T have. There is equal reason to put
them either first or last, which makes 3 of their games in violation
either way. If they are in the middle then all 6 games are in violation.
If you take an average of best possible rankings for minimizing the
ranking violation percentage they do end up average but the average of
the possible results violates the criteria the system started with in
the greatest manner possible.
Example 5: After a loss to I in the seventh game after the games in
Example 4 K would drop all the way to last. The wins in the first three
games mean nothing.
This is just my opinion and why I choose not to do it this way. My
system has a parameter that is essentially standard deviation of a
Normal Distribution. The higher I raise the number the more like the
system above it acts. I like to keep it at a value where it keeps the
risk/reward for playing hard or easy games in the best balance. That's
just the way I think is best. I'm sure others would not. I don't put
much weight in judging a system by looking at the results. I think you
look at the algorithm and see if you agree with the way teams are
rewarded and punished. I believe that if you agree with that then you
should accept the results.