Date: Mon, 20 Aug 2001 19:50:17 -0700 (PDT) From: Steve Wrathell Subject: CF Computer Rankings Study C.P.A. RANKINGS (TM) - COLLEGE FOOTBALL & NFL (c) 2001 Steven Wrathell, CPA, PC For this system's description & links to other CPA Rankings pages (NCAA & NFL) & more, go to: http://www.cae.wisc.edu/~dwilson/rsfc/rate/wrathell.html College Football Computer Rankings Study (& some NFL) (temporary version) First, I have to apologize that this has taken so long. I had a computer crash (it turns out it was my mother board). Fortunately, I had a new computer for my tax business, and I did have some printouts of my study, but it wasn't complete when my 486 computer crashed. The studies I did were for 1998 and 1999. I also have the 2000 data, but it will take some work to prepare it. I had a long and elaborate discussion and explanation of my study and the issues involved in rating football teams, retrodictive versus predictive, etc. If I can recover the data (I'm presently working on getting the 486 rebuilt), or if I can get everything re-typed, I'll replace this page with something much more complete. Other ways to measure systems have been developed since I did my 1998 study: Todd Beck measures predictive and retrodictive performances (for predictive systems) in his Prediction Tracker. Kenneth Massey now lists a Ranking Violation rate (a good retrodictive measurement) on his Comparison pages (for both 1A and 1AA). My description page has links to both of these. If ranking studies interest you, go to my description page, click on "Parent Directory," for David Wilson's site that links to a study by Mark Hopkins and Jeff Bihl's reply. It seems that another study, by Eugene Potemkin, is no longer on the web. The fact that the BCS now uses 8 computer systems in its formula raises the issue as to which systems are best and how one best measures this. Suggested methods to determine the BCS systems have included geographical diversity, length of time doing ratings, whether victory margins are utilized (and, if so, whether "blow-outs" are capped), and whether, somehow, the system would have pleased Miami Hurricane fans (at the expense of FSU) in 2000-2001). DISCLAIMER INFO Those who have retrodictive systems will have perspectives that are opposite those of predictors, of course. Non-predictive systems have objectives that differ from predictive systems and may ignore information used by predictive systems, such as victory margins, HFA's, etc., and this does put them at a disadvantage in this study. However, in this mini-presentation of my study, I am listing only the systems that did the best, without stating which systems did poorly. Ratings were taken from the Massey Comparison and any system not listed on it at a key date is excluded. For 1998, I had studied only a sample of the systems and a large number were excluded, but I included many of the predictive systems. My perspective will differ from many people, yet I think it warrants consideration. By the way, as I am measuring my own system in my studies, these are not "independent studies." HOW THIS WORKS The method I used was to compare the rankings of systems in mid-season (in this mini-report, I'm using only data from 10/31/1998 and 10/30/1999) and comparing them against the final post-bowl rankings. For this mini-study, I'm using only the final Massey Comparison Consensus. For example, if, a system, at 10/31/1998 ranked Oklahoma as #5, while they finished #1, the variance is 4. The total variances for all 112 (1998) or 114 (1999) teams are added up for each system. System's top 25's are also rated against the 112 or 114 final consensus, too, so the polls and NYT could be measured. The lower the score, the better a system did. The 3 letter "initials" used are per Massey's Comparison. The Massey Consensus itself is called "mc#." The rationale for this being meaningful is that the differences between the late October rankings and the post-bowl ratings are caused mostly as the result of the games in between. Those who had the teams that proved to be better rated higher have the lower variances. The point I ended each list was determined by where there seemed to be a fair- sized difference in the numbers. Being near the "bottom" of any of the lists below isn't bad, since only the "top" systems are listed. The BCS systems are in uppercase. Remember, this is supposed to be for fun. Don't let your blood pressure get too high because of this. Peace. Love. Understanding. Take it easy. Don't worry; be happy. 10/30/1999 vs 1/4/2000 MC For all 114 teams (top 19, of 61, systems) 1 arg 858 2 tsr 906 3 mc# 920 4 cpa 932 5 mic 950 6 SAG 968 7 hnm 982 8 how 1000 9 sau 1004 10 mau 1014 11 rmr 1016 12 gup 1026 13T BCS 1052 from J Palm list 13T sln 1052 15 bas 1058 16 whi 1060 17 max 1062 18T pig 1070 18T MAT 1070 10/30/1999 vs 1/4/2000 MC For top 25 teams (top 16 systems plus polls, of 61) 1 mic 118 2 gup 126 3 how 138 4 arg 151 5T hnm 153 5T sau 153 7 rmr 154 8 SAG 155 9 mar 157 10 str 158 11T cpa 160 11T ss 160 13 dev 161 14 mc# 162 15 buc 164 16 BCS 169 from J Palm list 25T USA 176 48 AP 193 10/31/1998 vs 1/4/1999 MC For all 112 teams (top 14 systems) 1 mc# 626 2 arg 658 3 cpa 710 4 ono 716 5 how 766 6 sau 780 7 MAT 784 8 yea 794 9 spa 814 10 sel 820 11 SAG 822 12 bmc 836 13 sch 854 14 MAS 869 10/31/1998 vs 1/4/1999 MC For top 25 teams (top 14 systems plus polls) 1 ono 86 2 sel 97 3 sch 103 4 yea 105 5 pig 107 6 arg 114 7T bmc 115 7T AP 115 9 cpa 117 10 spa 118 11 mc# 120 12T how 121 12T sau 121 14 USA 122 15 MAT 123 16 SAG 126 Summary of results from Todd Beck's Prediction Tracker. For full results, see his page. My description page links to it. Straight-up wins, 2000, NCAA-1A (out of 641 games) 1 arg 484 2T vegas line 477 2T DUN 477 2T cpa 477 2T yl 477 6 pom 476 7 MAS 475 8 SAG 473 9 MAT 472 Summary of results from Todd Beck's Prediction Tracker. For full results, see his page. My description page links to it. Retrodictive wins, 2000, NCAA-1A (out of 641 games) 1T cpa 538 1T arg 538 3 MAT 534 4 frz 533 5 lb-elo 532 Summary of results from Todd Beck's Prediction Tracker. For full results, see his page. My description page links to it. Straight-up wins, 2000, NFL (out of 259 games) 1 cpa 172 2 arg 170 3 MAS 168 4T kambour 165 4T MAT 165 4T pfz 165 7T vegas line 164 7T SAG 164 Summary of results from Todd Beck's Prediction Tracker. For full results, see his page. My description page links to it. Retrodictive wins, 2000, NFL (out of 259 games) 1 pfz 192 2 cpa 189 3 MAS 186 4T MAT 185 4T fly 185 6 ls-w/ tm hfa 184 Summary based on Kenneth Massey's College Football Comparison For full results, see his page. My description page links to it. Ranking Violation %, 2000, NCAA-1A Estimated retrodictive wins ((1-RV%) x 641), subject to possible rounding errors Pre = predictive & Ret = retrodictive, per David Wilson's listing (Go to my description page and click on "Parent Directory.") R V % Est W Type 1 sor 15.6% 541 Ret 2 col 16.4% 536 Ret 3 mac 16.7% 534 Ret 4T cof 16.8% 533 Ret 4T MAT 16.8% 533 Pre 4T frz 16.8% 533 Pre 7T wob 17.0% 532 Ret 7T csl 17.0% 532 Ret 7T fly 17.0% 532 Pre 10T wol 17.2% 531 Ret 10T wel 17.2% 531 Ret 12T cpa 17.3% 530 Pre 12T sel 17.3% 530 Ret 12T sun 17.3% 530 Ret 15 wil 17.4% 529 Ret In the event of ties, the charts based on Todd Beck's site are listed in the order he listed them (thus, using his "tie-breakers"). In the chart based on Kenneth Massey's site, the order of systems listed, when tied, is based on their correlation to the consensus. The information in the charts derived from Todd Beck's and Kenneth Massey's sites was calculated by Mr. Beck, and Mr. Massey, respectively. The manner of presentation was determined by CPA Rankings and may have been different if prepared by Mr. Beck or Mr. Massey. For their presentations, see their sites. Division 1AA ranking violation calculations are not listed above, but are determined by Mr. Massey. It should be noted that the retrodictive calculations by Todd Beck and the ranking violation % by Kenneth Massey are essentially making the same measurement. The difference is that Beck includes the effect of the systems' HFA's, which requires the use of ratings, rather than the ordinal rankings. Massey's use of the ordinal rankings allows for a measurement of all systems, including those for which could not be measured using Beck's methods. Beck's method also allows for the retrodictive calculations of mean error and bias. Massey's R V % method favors retrodictive systems. The measurements by Beck are essentially limited to predictive systems. The methods used by CPA Rankings, in the first few charts, favor predictive systems.