Abstract:
Junior cross-country skiers’ performances prior to participation in the 2006 Junior Olympics were compared to their results in the 2006 Junior Olympics using USSA points as a measure of performance. Junior class and division (team) were also included as independent variables. Prior performance as determined by USSA points is a poor indicator of performance in the Junior Olympics.
Introduction:
Cross-country skiing times from different races, even those of the same length, are not comparable because the terrain is different for each race. Furthermore, snow conditions may vary, even from hour to hour, on the same course. Merely comparing times of skiers over similar distances is not an accurate comparative assessment of skiers’ abilities. The United States Ski and Snowboard Association (USSA) points list was developed to allow comparison between skiers who may have entered several different races. USSA points are awarded to registered cross-country skiers for participation in sanctioned ski races. A lower value in USSA points indicates that a skier is a better, more competitive skier. USSA points and similar International Ski Federation (FIS) points are used to help select the U.S. national teams, to seed racers in both mass and interval start races, and to monitor the progress of athletes in physiological studies (Bodensteiner & Metzger 2006; Staib, Im, Caldwell, & Rundell 2000).
The USSA formula that allocates points to skiers is based on race performance. It includes a number of variables that capture the relative ability of skiers in the race. Who enters the race and how they place are used in determining the penalty. Each race’s penalty is based upon the current USSA points of top finishers in the race. The type of start or race and a minimum penalty also are used in the calculation of USSA and FIS points assigned to a skier’s race (Bodensteiner & Metzger 2006, International Ski Federation, 2006). Despite the common and, at times, mandatory use of the system, the USSA point system has been criticized by racers and coaches over the years for failure to accurately capture a skier’s ability (Anonymous, 2006; Smith, 2002; Trecker 2005).
Methods:
Given the importance and criticism of USSA points, this study develops a systematic comparison of prior USSA points results of skiers to their USSA points earned in a common competition. One would hypothesize that a skier’s points prior to a competition would predict a skier’s points earned within the competition. Points earned by Junior skiers (ages 14 to 19) in the 2005-2006 season are compared to USSA points in the 2006 Junior Olympics. The use of linear regression allows one to determine if a linear relationship exists between prior performance and performance in the Junior Olympics and whether other, easily obtained variables can improve the ability to predict performance at the Junior Olympics. (Hill, Griffiths, & Judge, 1997; Johnston, 1984)
Before the Junior Olympics, skiers earned USSA points in different races throughout the northern part of the United States. Skiers within any of the ten USSA districts competed against each other, but there was limited competition among skiers from different districts. The top 400 skiers then competed in the Junior Olympics in March, 2006 in Houghton, Michigan. The end of season Junior Olympics allows skiers to be directly compared on the same course and with the same snow conditions, so USSA points assigned in these races can be used in this study free of the bias of course and snow conditions.
A general linear model (equation 1) with USSA points earned in the Junior Olympics as the dependent variable and USSA points prior to the Junior Olympics, junior class (J2, J1, or OJ) division (team) were used as independent variables. The parameters c and ak (where k = 1, 2, and 3) were estimated. Estimated parameters in bold are matrices of parameters associated with a matrix of dummy variables. Equation 1 is the most comprehensive linear model used.
yi = c + a1*Pi + a2* JCLASSi + a3*DIVi + ei equation 1
Where
yi = USSA points in the 2006 Junior Olympics for the ith skier,
c = an estimated constant,
Pi = USSA points prior to the Junior Olympics for the ith skier,
a1 = the estimated parameter associated with Pi,
JCLASSi = a matrix of junior classes with dummy variables for OJ, J1, and J2 where the value is 1 in the ith skier’s junior class and zero for other classes,
a2 = a matrix of estimated parameters associated with JCLASSi,
DIVi = a matrix of regional divisions with dummy variables for Alaska, Great Lakes, Midwest, Intermountain, Rocky Mountain, Mid-Atlantic, New England, Far West, High Plains, and Pacific Northwest where the value is 1 in the ith skier’s division and zero for other divisions,
a3 = a matrix of estimated parameters associated with DIVi, and
ei = the residual value for the ith skier.
The model was run using USSA points from all three individual races at the Junior Olympics (yi): freestyle, classic, and sprint. USSA points prior to the Junior Olympics included (Pi) for distance, sprints, and overall points were used in separate regressions. Thus, there are several versions of equation 1 that use different techniques (classic and freestyle) and USSA disciplines (sprint, distance, and overall).
While equation 1 represents the most extensive model tested, other models using a subset of the independent variables were also tested to determine the stability of the model. When sets of independent dummy variables would have resulted in a full rank matrix, one of the variables was not included in the regression. Technical definitions associated with cross-country skiing terms can be found in the USSA’s Nordic Competition Guide (Bodensteiner & Metzger, 2006). Analyses were run using the GLM procedure in SAS 9.1 for Windows.
Data:
Pre-Junior Olympics distance, sprint, and overall USSA points; names; USSA numbers (to confirm this data with results from the Junior Olympics); junior class (J2, J1, or OJ); and year of birth information were obtained from the national list of USSA points, which had been updated just prior to the Junior Olympics. Data were downloaded on March 27, 2006. Junior Olympic classic, freestyle, and sprint USSA points; skier’s division (team); name; and USSA number were obtained from itiming.com via the web in the week following the 2006 Junior Olympics. In all cases, as complete a data set as possible was used in the regression. However, some skiers entered the Junior Olympics without prior USSA points or with only a partial set of information. The most common missing data were USSA sprint points prior to the Junior Olympics. Whenever a valid number was available for a skier, that skier was entered in the data set for a particular regression analysis. In a few cases, skiers did not start or finish a race or were disqualified during the race. The largest data set included information for 271 skiers.
Results:
USSA Points prior to the Junior Olympics – the simplest models.
The first part of the statistical analysis was to determine if USSA points alone could predict USSA points in the Junior Olympics. The model used to test this question was:
yi = c + a1*Pi + ei equation 2
Since skiers have sprint, distance, and overall points prior to the Junior Olympics and compete in sprint, freestyle distance, and classic distance events, there are six logical combinations of dependent and independent variables. Table 1 shows the results of each regression.
Table 1: Results from the regression of USSA points earned at the Junior Olympics (yi) on USSA points earned prior to the Junior Olympics (Pi). Equation 2
| yi JO Points (Source) | Pi Prior (Source) |
estimated c |
estimated a |
r2 |
| Freestyle | Overall | 87.1 | 0.57 | 0.59 |
| Freestyle | Distance | 82.8 | 0.59 | 0.59 |
| Classic | Overall | 116.9 | 0.79 | 0.36 |
| Classic | Distance | 106.7 | 0.85 | 0.37 |
| Sprint | Overall | 74.4 | 0.80 | 0.54 |
| Sprint | Sprint | 84.8 | 0.60 | 0.49 |
Note: All estimated parameters were significant at the 0.0001 level.
At best, the USSA points earned prior to the Junior Olympics predict only 59% of the variability in the final USSA points earned at the Junior Olympics. Equation 2 is least effective when used to predict the classic results, explaining only 36% of the variability when the independent variable is Overall USSA points prior to the Junior Olympics. Figure 1 shows the relationship between the Overall USSA points prior to the Junior Olympics and USSA points earned in the Junior Olympics classic race. The top five skiers based upon prior USSA points also ended up with results close to what one would expect. However, after this elite group of skiers, the prior USSA points exhibit poor predictive ability for the remaining skiers. Some skiers with relatively high USSA points skied well and moved up dramatically at the Junior Olympics. The reverse was also true; some skiers skied less competitively than one would have predicted from their prior USSA points. While this is to be expected to some extent (athletes have good and bad days), the large number of skiers who deviated from the expected indicates something other than a few atypical performances by a small number of skiers has occurred. While the correlation between prior USSA points and the freestyle and sprint race results were better than the classic, the same general pattern is evident the results of these two races are plotted. The top skiers were identified by prior USSA points while predictive power diminishes for average and relatively weaker skiers at the Junior Olympics. In fact, even finish order is poorly predicted by prior USSA points.
Figure 1. Relationship between Overall USSA points prior to the Junior Olympics and USSA points earned in the classic race at the 2006 Junior Olympics.
Figure 1 also shows that this data set is heteroscedastic. The heteroscedasticity of the data is discussed in the Appendix.
USSA Points prior to the Junior Olympics – adding independent variables
Given that USSA points earned prior to the Junior Olympics are relatively poor predictors for results at the Junior Olympics, whether or not it is it possible to use other readily available information to improve the estimate of where a skier would finish is of importance. Equation 1, a more robust model, was estimated for the same six data sets used for equation 2. Equation 1 includes the JO class of the ski and the division (team) of the skier. The r2 associated with each equation is shown in Table 2.
Table 2. Comparison of Equation 2, only prior JO points, with Equation 1, prior JO points, Junior class, and division (team).
| yi JO Points (Source) | Pi Prior (Source) |
equation 2 r2 |
equation 1 r2 |
| Freestyle | Overall | 0.59 | 0.69 |
| Freestyle | Distance | 0.59 | 0.68 |
| Classic | Overall | 0.36 | 0.51 |
| Classic | Distance | 0.37 | 0.52 |
| Sprint | Overall | 0.54 | 0.65 |
| Sprint | Sprint | 0.49 | 0.64 |
Using Junior class and division and team of the skier improved the r2 for all six combinations of Junior Olympics USSA points and points earned prior to the Junior Olympics. Unfortunately, the best r2 is 0.69, indicating that there is still a substantial amount of unexplained variability in the data set. Equation 1 is an improvement, but still does not leave one with the ability to use the model with confidence if the purpose is to use past performance to predict expected performance.
Because there is little difference between the use of overall points and other prior USSA points as independent variables in equation 1, only results for equation 1 with overall points are reported. Table 3 shows the variables, estimated parameters, and P values for each independent variable for the classic, freestyle, and sprint races at the 2006 Junior Olympics.
Table 3. Estimated parameters and probability level for the parameters, in parentheses, for equation 1. Estimations are for all three individual events at the Junior Olympics using skiers’ overall USSA points, division (team), and junior class as independent variables.
Independent Estimated Parameter and P Value (Pr > |t|)
Variable Classic Freestyle Sprint
Constant 135.90 83.47 44.38
(<0.001) (<0.001) (0.003)
OVERALL 0.89 0.55 0.77
(<0.001) (<0.001) (<0.001)
NE -46.43 -17.78 -22.73
(0.005) (0.015) (0.063)
MA -7.61 4.50 5.13
(0.731) (0.647) (0.743)
GL -28.74 -21.40 53.06
(0.102) (0.044) (0.012)
MW 1.15 -6.50 0.87
(0.961) (0.405) (0.946)
HP 50.07 56.54 69.19
(0.047) (<0.001) (<0.001)
IM -5.15 20.21 58.61
(0.754) (0.006) (<0.001)
RM -4.40 -3.12 33.66
(0.794) (0.677) (0.004)
FW -32.77 -17.09 51.88
(0.090) (0.047) (<0.001)
PN -2.75 0.63 23.66
(0.887) (0.942) (0.079)
J1 -16.16 9.91 26.69
(0.163) (0.053) (0.002)
J2 -93.08 8.23 13.23
(<0.001) (0.211) (0.231)
Notes: Alaska and OJ are omitted to avoid estimation of a full-rank matrix.
NE = New England, MA = Mid-Atlantic, GL = Great Lakes, MW = Midwest,
HP = High Plains, IM = Intermountain, RM = Rocky Mountain, FW = Far West,
PN = Pacific Northwest.
Each of the equations is estimated with Alaska omitted as a team and the OJ class omitted. This prevents full rank estimation of the equation. The Classic estimation shows that New England and Far West skiers ski relatively faster than Alaskan skiers given their predicted times. High Plains skiers are slower than predicted relative to the Alaskan skiers. The estimated parameters for other divisions are not significantly different from zero. In the freestyle race, the estimated parameter for the dummy variable representing skiers from the New England, Great Lakes, and Far West indicated that, given their prior USSA points, members of these teams were relatively faster than the Alaskan skiers as indicated by USSA points earned in the Junior Olympics race. The phrase “relatively faster” is important. In general, Alaskan skiers finished ahead of Great Lakes skiers, although the estimated parameter associated with the Great Lakes is negative. The dummy variables for teams improve the estimation by adjusting for a skier’s team given the other variables used in the estimation, especially the overall USSA points prior to the Junior Olympics. Using Alaska and the Great Lakes as an example, the average Alaskan skier entered the Junior Olympics with a better USSA points ranking and than the average Great Lakes skier. The Alaskan skiers also outperformed the Great Lakes skiers on average at the Junior Olympics. However, in the freestyle competition at the Junior Olympics, the Great Lakes skiers’ improvements from predicted to actual performance was substantially better than that of the Alaskan skiers. Dummy variables capture this distinction.
In the freestyle race, the estimated parameters for the High Plains and Intermountain teams were positive. In the sprint race, the teams from New England again had a significant, negative estimated parameter while the Great Lakes, High Plains, Intermountain, Rocky Mountain, Far West, and Pacific Northwest all had significant, positive estimated parameters. Both the Far West and Great Lakes had significant, negative estimated parameters in the freestyle race but significant, positive estimated parameters in the sprint race. (New England skiers can take heart that they outperformed their expected results and won the Alaskan Cup despite whatever disadvantage may accrue to weaker seeding.)
The estimated parameter for junior class was also significant for one of the classes in each of the equations, indicating that including class in the estimate improves the equation. Junior class can help predict USSA points earned.
Stability of the Models
It would be tempting to state that the use of additional variables improves the equation and would help somebody trying to use prior USSA points in estimating performance or performance gains. However, several factors argue against this.
1. This data set represents only the top junior skiers, ages 14 to 19, over one season.
2. The three versions of equation (1) estimated with classic, freestyle, and sprint results from the Junior Olympics are not similar. Both the constant and parameter associated with the overall points vary considerably with the different estimations, indicating that the model is not stable.
3. The parameters associated with dummy variables representing divisions (teams) and junior classes are not consistent and, in some cases, change dramatically from estimation to estimation. For example, Great Lakes skiers have a positive and significant parameter associated with the dummy variable in the freestyle equation, but they have a negative and significant parameter associated with the dummy variable in the sprint equation.
4. The r2 values associated with all equations estimated are not strong enough to justify the use of the model to predict the future results of skiers.
Given these concerns, it is likely that estimating these equations using data from other years or older skiers would generate substantially different equations. It is unlikely that the model would be stable (that is, the estimated parameters would be similar), if different versions of the model were estimated or different data sets were used.
Conclusions:
This paper provides a clear test of the ability of USSA points to compare the relative ability of skiers. The initial points of skiers earned in their best races prior to the Junior Olympics were used to estimate a linear regression model with points earned in three separate races at the Junior Olympics less than a month after the prior points list was released by the United States Ski and Snowboard Association. The prior points were a poor predictor and the general model showed poor stability from estimation to estimation. While these results were derived from a data set composed of junior skiers, they support the broader anecdotal concerns about USSA points. This study provides a reliable quantitative basis for those concerns with a substantial and consistent data set. Most observers of cross-country ski racing would not be surprised by these results. However, the instability in the data set is striking and is less easily observed through casual observation of ski results. Not only are the predictions relatively poor, those poor predictions vary with the subset of the data and the specific model used to make the prediction. USSA points should be used with caution and with other information for critical decisions in cross-country ski racing. Their value in monitoring skier performance in physiological trials is questionable.
References:
Anonymous. (2006). U.S. Olympic Cross Country Team Announced. Retrieved October 6, 2006 from http://www.fasterskier.com/news2962.html .
Bodensteiner, L., & Metzger, S. (2006). 2006 USSA Nordic Competition Guide. Park City, UT.
Hill, C., Griffiths, W., & Judge, G. (1997). Undergraduate Econometrics. J. Wiley & Sons, New York.
International Ski Federation. (2006). Cross Country Rules and Guidelines of the FIS Points 2006/07. Retrieved October 11, 2006 from http://www.fis-ski.com/data/document/pktrgl0607-neu.pdf
Johnston, J. (1984). Econometric Methods (3rd ed.) McGraw-Hill, New York.
Smith, C. (2002). U.S. Olympic Team Selection. Retreived July 17, 2006 from http://www.xcskiracer.com/rants.shtml
Staib, J.L., Im, J.,Caldwell, Z., & Rundell, K.W. (2000). Cross-country ski racing performance predicted by aerobic and anaerobic double poling power. Journal of Strength and Conditioning 14(3), 282-288.
Trecker, M. (2005). Following the Olympic Trials, Who’s Hot, Who’s Not, and the Strange Anomalies of USSA Scoring. Retrieved July 17, 2006 from http://www.fasterskier.com/opinion2749.html
Appendix – Heteroscedasticity in the Data Set:
This portion of the study on heteroscedasticity is placed in the appendix because most people interested in skiing will not be interested in statistical methods and assumptions. They want to know if current USSA points predict future skiing results. However, from an analytical viewpoint, improper use of statistics can lead to incorrect results and correct procedures lead to improved analysis. One assumption of linear regression is that the variance of the random error term is 2 for all x. If this is not the case, then the estimate remains linear and unbiased but it is no longer the best linear unbiased estimator and standard errors are often incorrect (Johnston, 1984). Confidence intervals and results of statistical tests can be misleading. This appendix covers four topics: heteroscedasticity in equation 2, correcting for heteroscedasticity using data transformations, heteroscedasticity in the complete data set, and a brief conclusion.
Heteroscedasticity in equation 2
Equation 2 is the intuitive equation to test whether prior performance as measured by USSA points can predict future performance.
yi = c + a1*Pi + ei equation 2.
Figure 1 shows a much wider variance in the dependent variables as USSA points increase. White’s test for heteroscedasticity indicates a probability of greater than 99.99% that heteroscedasticity does exist (test statistic= 15.37 with two degrees of freedom).
Correcting for heteroscedasticity using data transformations
Data may be adjusted using transformations to eliminate heteroscedasticity (Hill et al, 1997, Johnston 1984). In the data set used in this study, the variance in the residuals is larger for the larger values of the independent variable. Two logical transformations are to take the logarithm of the independent variable and the square root of the independent variable. Separate regressions were estimated using equation (2) where
(a) Pi = the square root of the competitors USSA points earned prior to the Junior Olympics and
(b) Pi = the natural logarithm of the competitors USSA points earned prior to the Junior Olympics.
In both cases, the r2 value improved less than 0.02, and the White’s test indicated that heteroscedasticity remained a problem.
Heteroscedasticity in the complete data set
The complete data set, including division and junior class of the competitor, not only improves the estimation, it is less likely heteroscedasticity exists. White’s test for heteroscedasticity indicates a probability of approximately 80% that heteroscedasticity does exist (test statistic= 49.46 with 42 degrees of freedom). Most researchers would not reject the null hypothesis at this level. This indicates that the additional independent variables have the greatest impact on improving prediction for skiers with the higher (less competitive) prior USSA points.
Conclusion:
The original goal of this study was not only to determine what statistical model would work best for the data, but to determine if USSA points were a good predictor of future performance of athletes. From a practical standpoint, a complex model used in the prediction would indicate that USSA points alone are a poor predictor and a complex model would be difficult to justify and administer. The heteroscedasticity and the development of more complicated, but still unstable, models reinforce the results of the main paper. Prior USSA points are poor predictors of Junior races.
