From: federation2005 at netzero dot com Sent: Thursday, November 08, 2007 1:50 AM There is a slightly different stat I've been using when compiling compedia like the one you've set up, which is immune to average pull-down or average pull-up effects. Whereas if you use average, where all it takes is one really bad judge to pull everything out of order, with this stat nobody can mess up the totals. This also means that it's not necessary to vet out good from bad judges, either. The way it works is simple. One team is considered better than the other if they're considered so by the majority of judges. This comparison is used to compile the stat as follows: If there were 100 teams, there would be 99 such comparisons per team. For each comparison, score 0 for the better team, 1 for the worse team, 0.5 for both if tied. Sum up the comparisons, add 1. The result will be between 1 and 100 and will mimick a "consensus rank". In the near future, I'll demonstrate it with the entire Massey compilation. Date: Tue, 13 Nov 2007 18:04:55 GMT Subject: Consensus Rating and "Wisdom of the Masses" This is an edited copy of a letter I sent recently to Massey concerning the general issue listed in the subject header. It's something I think you might also be interested in. On the November 2 compilation of the rating comparison page, based on taking the "average" (see note below) rank of all the ranking systems involved, the Massey page had the following as the top 6: #1 Ohio St, #2 LSU, #3 Arizona St, #4 Kansas, #5 Oregon, #6 Boston College However, this is the rank order, based on *actual* consensus of the ranking systems listed on that page: #1 Ohio St, #2 LSU, #3 Boston College, #4 Arizona St, #5 Kansas, #6 Oregon The tabulation of the consensus on a one-on-one basis, in fact, shows how this pans out: Ohio St 72 rating systems vs. LSU 46 rating systems Ohio St 76, Boston College 41 Ohio St 82, Arizona St 35 Ohio St 82, Kansas 35 Ohio St 93, Oregon 25 LSU 62, Boston College 56 LSU 66, Arizona St 52 LSU 80, Kansas 38 LSU 96, Oregon 22 ** Boston College 60, Arizona St 57 ** Boston College 73, Kansas 44 ** Boston College 71, Oregon 47 Arizona St 68, Kansas 47 Arizona St 73, Oregon 45 Kansas 61, Oregon 57 The overall "won-loss" records based on a "tourney" of one-on-one comparisons has the following as its top 10: Team Record W L T W+T/2 Ohio St 9-0 119 0 0 1.0 LSU 7-1 118 1 0 2.0 Boston College 8-0 117 2 0 3.0 Arizona St 8-0 116 3 0 4.0 Kansas 8-0 115 4 0 5.0 Oregon 7-1 114 5 0 6.0 West Virginia 7-1 113 6 0 7.0 Oklahoma 7-1 112 7 0 8.0 Missouri 7-1 110 8 1 9.5 South Florida 6-2 110 8 1 9.5 Generally, average is an ill-defined measure, because the ranks are ordinals, not numbers, therefore arithmetic operations (+, -, *, /, averaging) on them are undefined. The lack of robustness, particularly with respect to pull-ups and pull-downs from bad judges (as exemplified, here, by the severity of the pull-down of Boston College) is a manifestation of this problem. Median, too, is a better measure than average, since it makes use only of comparisons (which are legal operations on ordinals), and more closely approximates the ideal of direct one-on-one comparisons. The overall "tourney" result, in fact, will serve to simultaneously subsume both the "average" and median stats you presently use. A similar set of notes apply to the comparisons of ranking systems, themselves. Here, the role of "judger" and "judgee" is swapped, and now it's the teams that witness the comparison of the rating systems, rather than the other way around. The sorting for the rating systems, based on "correlation to consensus", was: #1 SD, #2 FIT, #3 CLA, #4 ARG, #5 BOB, "Correlation" suffers the same problem as "average", one of whose manifestations (in fact), here, is the inability to put the "top 25's" on the same scale as everyone else. In contrast, there are two measures which are well-defined, more robust and allow direct comparisons of "top 25" with everyone else. In the process, both the stats "consensus" and "top 25 consensus" are subsumed. One tabulates the number of times that a ranking system was part of the majority in all the one-on-one comparisons. This results in the following top 5: Pct. Poll Agree Disagree .978 CLA 6970 158 .975 FIT 6950 178 .975 SD 6947 181 .965 BCS 6881 247 .964 COF 6868 260 and top 10, when including the partial rating systems: Pct. Poll Agree Disagree .978 CLA 6970 158 .975 FIT 6950 178 .975 SD 6947 181 .975 AP 2602 68 .968 USA 2585 85 .967 MCS 2582 89 .966 HAR 2579 91 .965 BCS 6881 247 .964 COF 6868 260 .962 SOR 6859 269 .962 BIH 6859 269 The other tabulates the number of agreements with the overall consensus "tourney". For this week, the "tourney" has 10 2-way ties and 2 3-way ties, for a total of 7124 non-tying comparisons. The top 5 are: Pct. Poll Agree Disagree .979 CLA 6971 153 .976 SD 6950 174 .975 FIT 6944 180 .965 BCS 6876 248 .965 COF 6872 252 and top 10, overall: Pct. Poll Agree Disagree .979 CLA 6971 153 .976 SD 6950 174 .975 FIT 6944 180 .974 AP 2601 70 .967 USA 2583 87 .966 MCS 2580 92 .965 BCS 6876 248 .965 HAR 2577 93 .965 COF 6872 252 .963 SOR 6858 266 The two stats are closely parallel and tend to produce the same overall comparisons. Based on these 2 measures, in fact, one can say that the CLA, SD and FIT outperformed the AP this past week. As expected, the AP and USA are near the top, performance-wise. This is a consequence of the general phenomenon called the "wisdom of the masses". A good example of that, by the way, is seen in intelligence tests. If I take a set of responses to such a test and tabulate a consensus based on majority vote, the consensus will generally obtain high score on the test, possibly even outperforming each of the test-takers the consensus was derived from. The consensus comparison for Boston College, at #3, was close (LSU 62, Boston College 56), while at the same time the AP and USA put them at #2. So, it might be interesting to see how close their actual AP and USA poll numbers were. They should have been neck and neck in both the AP and USA for that week. Date: Tue, 13 Nov 2007 18:23:20 GMT On the cite of the top 10 for November 2 Team Record W L T "Rank" Ohio St 9-0 119 0 0 1.0 LSU 7-1 118 1 0 2.0 Boston College 8-0 117 2 0 3.0 Arizona St 8-0 116 3 0 4.0 Kansas 8-0 115 4 0 5.0 Oregon 7-1 114 5 0 6.0 West Virginia 7-1 113 6 0 7.0 Oklahoma 7-1 112 7 0 8.0 Missouri 7-1 110 8 1 9.5 South Florida 6-2 110 8 1 9.5 The last state, "Rank" is compiled as 1 + L + T/2. In the list, one can also see the advantage of computer ranking systems. South Florida is a dark horse with the human polls, while with computer-based systems everybody is a dark horse and equally biased against, so that everyone gets equal consideration. A third stat, meant to replace the "inversion %" counts the number of strong inversions. This occurs where a team from a group is rated lower than an opponent it played (and defeated) outside that group, wherever the group in question is undefeated from the outside. This selects out only the subset of inversions where the two teams are not transitively equal. Ideally, this number should be 0; and 0 is possible. Date: Wed, 14 Nov 2007 09:58:54 GMT A second addendum -- the complete listing for the 2007 November 2 edition of the Massey comparison page: Rank Team Record W L T 1.0 Ohio State (9-0) 119 0 0 2.0 LSU (7-1) 118 1 0 3.0 Boston College (8-0) 117 2 0 4.0 Arizona St (8-0) 116 3 0 5.0 Kansas (8-0) 115 4 0 6.0 Oregon (7-1) 114 5 0 7.0 West Virginia (7-1) 113 6 0 8.0 Oklahoma (7-1) 112 7 0 9.5 Missouri (7-1) 110 8 1 9.5 South Florida (6-2) 110 8 1 11.0 Georgia (6-2) 109 10 0 12.0 Auburn (6-3) 108 11 0 13.0 Connecticut (7-1) 107 12 0 14.0 Virginia Tech (6-2) 106 13 0 15.0 Florida (5-3) 105 14 0 16.0 Alabama (6-2) 104 15 0 17.0 Michigan (7-2) 103 16 0 18.0 Clemson (6-2) 102 17 0 20.0 Wake Forest (6-2) 100 19 0 21.0 Southern Cal (6-2) 99 20 0 22.0 Kansas St (5-3) 98 21 0 22.0 Boise St (7-1) 98 21 0 23.5 California (5-3) 96 22 1 23.5 Texas (7-2) 96 22 1 24.0 South Carolina (6-3) 96 23 0 27.0 Kentucky (6-3) 92 25 2 27.0 Virginia (7-2) 93 26 0 27.5 Tennessee (5-3) 92 26 1 28.0 Purdue (7-2) 91 26 2 29.5 Hawai`i (8-0) 90 28 1 30.0 Penn State (6-3) 90 29 0 32.0 Georgia Tech (5-3) 88 31 0 33.0 Cincinnati (6-2) 87 32 0 35.0 Oklahoma St (5-3) 84 33 2 35.0 Illinois (6-3) 84 33 2 36.0 Wisconsin (7-2) 84 35 0 36.0 Florida St (5-3) 83 34 2 38.0 Oregon St (5-3) 82 37 0 39.0 Brigham Young (5-2) 81 38 0 40.0 Colorado (5-4) 80 39 0 41.0 Rutgers (5-3) 79 40 0 42.0 Texas A&M (6-3) 78 41 0 43.0 Vanderbilt (5-3) 77 42 0 44.5 Arkansas (5-3) 75 43 1 45.0 UCLA (5-3) 74 43 2 45.5 New Mexico (6-2) 74 44 1 47.0 Mississippi St (5-4) 73 46 0 48.0 Troy (6-2) 72 47 0 49.0 Texas Tech (6-3) 71 48 0 51.0 Miami FL (5-3) 69 50 0 51.0 Utah (6-3) 69 50 0 51.5 Air Force (6-3) 68 50 1 52.5 Maryland (4-4) 67 51 1 54.0 Louisville (5-4) 66 53 0 55.5 Michigan St (5-4) 64 54 1 55.5 Central Florida (5-3) 64 54 1 57.0 Houston (5-3) 63 56 0 58.0 Wyoming (5-3) 62 57 0 59.0 Fresno St (5-3) 61 58 0 60.0 East Carolina (5-4) 60 59 0 61.0 Nebraska (4-5) 59 60 0 62.0 North Carolina St (3-5) 58 61 0 63.0 Central Michigan (5-4) 57 62 0 64.0 Stanford (3-5) 56 63 0 65.0 Northwestern (5-4) 55 64 0 66.0 Washington (2-6) 54 65 0 67.0 Indiana (5-4) 53 66 0 68.0 Ball St (5-4) 52 67 0 69.0 TCU (4-4) 51 68 0 70.0 Navy (4-4) 50 69 0 71.0 Tulsa (5-3) 49 70 0 72.0 Washington St (3-5) 48 71 0 73.0 Iowa (4-5) 47 72 0 74.0 Pittsburgh (3-5) 46 73 0 75.0 Arizona (3-6) 45 74 0 76.0 North Carolina (2-6) 44 75 0 77.0 UTEP (4-4) 43 76 0 78.0 Mississippi (2-7) 42 77 0 79.0 Bowling Green (4-4) 41 78 0 80.0 Southern Miss (4-4) 40 79 0 81.0 Middle Tennessee St (4-5) 39 80 0 82.0 Miami OH (4-5) 38 81 0 83.0 Florida Atlantic (4-4) 37 82 0 84.0 Nevada (4-4) 36 83 0 85.0 Buffalo (4-5) 35 84 0 86.0 Notre Dame (1-7) 34 85 0 88.0 Western Kentucky (5-3) 32 87 0 88.0 Louisiana Tech (3-5) 32 87 0 90.0 Duke (1-7) 30 89 0 90.0 Baylor (3-6) 30 89 0 90.0 Akron (3-5) 30 89 0 92.0 New Mexico St (4-5) 28 91 0 93.0 San Diego St (2-5) 27 92 0 93.0 Army (3-5) 27 92 0 95.0 Ohio U. (4-5) 25 94 0 96.0 Toledo (4-5) 24 95 0 97.0 San Jose St (3-5) 23 96 0 98.0 Syracuse (2-6) 22 97 0 99.0 Louisiana-Monroe (3-5) 21 98 0 100.0 Arkansas St (3-5) 20 99 0 101.0 Kent St (3-6) 19 100 0 102.0 Memphis (4-4) 18 101 0 103.0 Temple (3-5) 17 102 0 104.0 Western Michigan (3-6) 16 103 0 105.0 UNLV (2-7) 15 104 0 107.0 Colorado St (1-7) 13 106 0 107.0 Minnesota (1-8) 13 106 0 107.0 Eastern Michigan (3-6) 13 106 0 109.0 Iowa St (1-8) 11 108 0 110.0 Alabama-Birmingham (2-6) 10 109 0 111.0 Marshall (1-7) 9 110 0 112.0 Tulane (2-6) 8 111 0 113.0 Louisiana-Lafayette (1-7) 7 112 0 114.0 Rice (1-7) 6 113 0 115.0 SMU (1-7) 5 114 0 116.0 Utah St (0-8) 4 115 0 117.0 Idaho (1-8) 3 116 0 118.0 North Texas (1-7) 2 117 0 119.0 Florida Int'l (0-8) 1 118 0 120.0 Northern Illinois (1-8) 0 119 0 Inversions: BEM #109 #101 San Jose St 23 Utah St 20 KLK #116 #110 San Jose St 23 Utah St 20 MKV #111 #105 San Jose St 23 Utah St 20 FEI #118 #104 San Jose St 23 Utah St 20 ACR #114 #113 San Jose St 23 Utah St 20 KLK #20 #17 Boston College 14 Virginia Tech 10 MKV #28 #13 Boston College 14 Virginia Tech 10 STH #26 #25 Boston College 14 Virginia Tech 10 MOR #18 #17 Kansas 30 Kansas St 24 BDF #11 #9 Kansas 30 Kansas St 24 STH #26 #24 Boston College 24 Georgia Tech 10 ACR #29 #20 Arizona St 33 Colorado 14 ACR #29 #27 Arizona St 44 Oregon St 32 KAM #15 #11 Arizona St 31 California 20 The percentages and counts of the two stats mentioned for poll-comparisons is listed here, along with the inversion count for Division I-A games: Pct1 Pct2 Inv. Poll Count1 Count2 .979 .978 0/50 CLA 153/7124 158/7128 .976 .975 0/50 SD 174/7124 181/7128 .975 .975 0/50 FIT 180/7124 178/7128 .974 .975 0/37 AP 70/2671 68/2670 .967 .968 0/37 USA 87/2670 85/2670 .966 .967 0/37 MCS 92/2672 89/2671 .965 .966 0/37 HAR 93/2670 91/2670 .965 .964 0/50 COF 252/7124 260/7128 .965 .965 0/50 BCS 248/7124 247/7128 .963 .962 0/50 SOR 266/7124 269/7128 .962 .962 0/50 MAA 270/7124 270/7128 .962 .962 0/50 BIH 273/7124 269/7128 .961 .962 0/50 WOB 275/7124 273/7128 .961 .961 0/50 ARG 276/7124 280/7128 .961 .961 0/50 ASH 279/7124 281/7128 .960 .960 0/43 DEV 217/5362 213/5358 .959 .959 0/50 BOB 289/7124 293/7128 .958 .958 0/50 MIN 297/7124 302/7128 .958 .958 0/50 MAS 297/7124 297/7128 .958 .957 0/50 AND 302/7124 306/7128 .957 .958 0/50 RUD 305/7124 302/7128 .957 .957 0/50 WOL 305/7124 305/7128 .957 .957 0/50 PSR 307/7124 307/7128 .957 .957 0/50 DOL 307/7124 304/7128 .956 .957 0/50 HOW 310/7124 310/7128 .956 .955 0/50 IMS 311/7124 318/7128 .956 .957 0/50 MLR 313/7124 310/7128 .955 .954 0/50 JNK 320/7124 329/7128 .955 .956 0/50 KRA 321/7124 315/7128 .955 .955 0/50 MEA 322/7124 319/7128 .955 .955 0/50 WIS 323/7124 322/7128 .954 .954 0/50 D1A 325/7124 327/7128 .954 .953 0/50 LAZ 327/7124 332/7128 .954 .954 0/50 COL 327/7124 331/7128 .954 .954 0/50 WEL 328/7124 326/7128 .953 .952 0/50 SAG 338/7124 345/7128 .952 .952 0/50 SEL 344/7124 341/7128 .951 .951 0/50 REW 350/7124 350/7128 .951 .950 0/50 TRX 351/7124 356/7128 .950 .950 0/50 RFL 355/7124 358/7128 .949 .949 0/50 ABC 364/7124 362/7128 .948 .949 0/50 RSE 368/7124 362/7128 .948 .948 0/50 BSR 370/7124 370/7128 .947 .947 0/50 ISR 375/7124 377/7128 .947 .947 0/50 BAS 376/7124 377/7128 .947 .946 0/50 RW 378/7124 383/7128 .946 .947 0/50 MB 384/7124 380/7128 .946 .945 0/50 MAU 387/7124 392/7128 .945 .945 0/50 ER 390/7124 393/7128 .945 .945 0/50 SAU 392/7124 395/7128 .944 .944 0/50 WIL 399/7100 395/7104 .944 .944 0/50 WLK 401/7124 401/7128 .943 .943 0/50 RTH 408/7124 407/7128 .942 .942 0/50 TSR 415/7124 413/7128 .941 .941 0/50 FMG 418/7124 419/7128 .941 .942 0/50 MCK 419/7124 416/7128 .941 .941 0/50 MJS 420/7124 418/7128 .940 .940 0/50 CPR 427/7124 428/7128 .940 .941 0/50 SE 427/7124 423/7128 .939 .940 0/50 LYD 432/7124 430/7128 .938 .938 0/50 ACU 439/7124 439/7128 .937 .937 0/50 DOK 449/7124 447/7128 .937 .937 0/50 WTS 450/7095 450/7100 .936 .937 0/50 GRN 453/7124 450/7128 .936 .936 0/50 BCD 454/7124 456/7128 .936 .936 0/50 SLT 456/7124 455/7128 .936 .936 0/50 BMC 457/7124 457/7128 .935 .936 0/50 GUN 460/7124 457/7128 .935 .934 0/50 CPA 463/7124 468/7128 .934 .935 0/50 CSL 468/7124 462/7128 .934 .934 0/50 HAN 470/7124 470/7128 .933 .933 0/50 ECK 474/7124 475/7128 .933 .933 0/50 SOL 476/7124 477/7128 .932 .931 0/50 SLN 486/7124 490/7128 .930 .929 0/50 BPI 502/7124 503/7128 .929 .929 0/50 DES 507/7124 508/7128 .927 .926 0/50 KEE 521/7124 528/7128 .926 .926 0/50 MAR 526/7124 527/7128 .925 .924 0/50 EL 537/7124 539/7128 .923 .923 0/50 MAY 547/7124 547/7128 .923 .923 0/50 HEN 548/7124 549/7128 .922 .921 0/50 NOL 558/7124 561/7128 .920 .920 0/50 MRK 567/7124 567/7128 .919 .919 0/50 GBE 579/7123 575/7127 .916 .916 0/50 SPR 599/7124 601/7128 .915 .914 0/50 MGN 609/7123 610/7127 .913 .913 0/50 LAW 617/7124 621/7128 .913 .913 0/50 BIL 617/7123 619/7127 .912 .912 0/50 DP 625/7124 626/7128 .911 .911 0/50 PFZ 631/7124 631/7128 .911 .910 1/50 BEM 632/7124 639/7128 .911 .911 0/50 REI 635/7124 635/7128 .909 .909 0/50 RPI 651/7124 650/7128 .908 .908 0/50 GM 655/7124 656/7128 .907 .907 0/50 TSW 659/7124 662/7128 .907 .907 0/50 SQ 660/7124 663/7128 .904 .903 0/50 CGV 684/7124 689/7128 .903 .903 0/50 PIG 689/7124 691/7128 .902 .903 2/50 STH 695/7124 693/7128 .901 .901 0/50 UCS 706/7124 708/7128 .900 .900 2/50 MKV 709/7124 713/7128 .900 .899 1/50 MOR 710/7124 717/7128 .898 .898 2/50 KLK 726/7124 730/7128 .898 .898 0/50 SP 726/7124 726/7128 .896 .896 0/50 DUN 742/7124 743/7128 .896 .896 0/50 HAW 742/7124 740/7128 .895 .895 1/50 KAM 746/7124 751/7128 .894 .894 0/50 ERD 758/7124 757/7128 .894 .893 0/50 DWI 758/7124 761/7128 .891 .890 0/50 OAF 778/7124 784/7128 .891 .891 0/50 BRN 778/7124 776/7128 .887 .887 0/50 PR 796/7062 799/7067 .887 .887 1/50 FEI 802/7124 806/7128 .879 .878 1/50 BDF 861/7124 867/7128 .871 .871 0/50 CMV 915/7105 914/7109 .858 .858 0/50 GMP 989/6985 992/6989 .851 .851 0/50 NUT 1062/7124 1064/7128 .824 .824 3/50 ACR 1257/7124 1258/7128