One month ago, I wrote an article (“Building a Better Magic Player: We Have the Technology”) examining a number of the ‘tournament preparation’ articles that have been written for Magic players over the years, and I included a survey designed to begin exploring whether making choices in line with these types of preparation (i.e., playtesting in certain ways, eating breakfast prior to a tournament) can predict success at Magic: The Gathering tournaments. After collecting data from 1,565 Magic players, I’ve organized the answers into graphs and have presented some preliminary thoughts and conclusions.
Because StarCityGames.com readers run the gamut from totally disinterested in process (“show me the graphs”) to fairly well-versed in statistics, I’ve bolded main prevalence findings (i.e., X number of players exhibited behavior A) to go along with the graphs, but I’ve provided other information as well for those who are a bit more interested in the data.
So, as our favorite overall-wearing plumber is wont to say, “Here we go!”
Out of the 1,565 Magic players who took the survey, 1,539 answered this question. Interestingly, 575 players, or 37% of the sample, either disagree or strongly disagree that they playtest a variety of decks before selecting a deck. This may actually be a function of the “Hive Mind,” especially since the advent of the StarCityGames.com Open Series, since the high frequency of competitive events has increased the speed with which the ‘best deck(s)’ is/are identified.
Out of the 1,565 Magic players who took the survey, 1,560 answered this question. The results are much more stark than they were for the previous question, and it’s clear that the majority of players tend to play decks with which they are comfortable, with a staunch 33 players, or a mere 2% of the sample, strongly disagreeing that they’ll play a deck with which they’re comfortable. While I won’t get into whether this actually predicts “success” yet, we might be interested in knowing that the individuals who answered this question with “Strongly Agree” and who answered the question about Total DCI Rating had an average Total DCI rating of 1755 (s: 117), while those who answered this question with “Strongly Disagree” and who answered the question about Total DCI rating had an average Total DCI rating of 1818 (s: 122).
Out of the 1,565 Magic players who took the survey, 1537 answered this question, and I find these results to be very surprising! There have been a number of articles over the last several years emphasizing the importance of testing both pre- and post-sideboarding games, and most of them are based on the premise that most Magic players don’t test both kinds of games. What we find, though, is that 1,166 players, or 76% of the sample, either agree or strongly agree that they playtest games both pre- and post-sideboarding! Perhaps those articles have had an effect…
Now we approach a part of the report that will surprise at least 407 StarCityGames.com readers who took the survey: there was a second page of questions. While missing these individuals’ responses to the later questions will lower the power of our analyses, we still have an ample number of participants (1,158).
All of the remaining 1,158 Magic players answered this question, and the results are fairly typical of this college- and slightly-older-than-college-age population. The Centers for Disease Control and Prevention (CDC) recommend that anyone over the age of 17 get between 7 and 9 hours of sleep each night for optimal functioning and health, and it seems like 528 players, or 46% of the sample, meet this recommendation before tournaments. This result is similar to one that I recently reported based on a survey of around 1,200 college students for a research project (49.2% of undergraduate college students reported sleeping between 7 and 9 hours per night).
What is especially interesting to me, though, is how many players I meet at every large tournament I attend who say “Hey bro, I’m on like two hours of sleep, and even during that period of time, I hired people to dress like me and party so I could keep on keepin’ on!” Although I have no idea whether the people who responded to this survey represent the people with whom I interact at most tournaments, the fact that only 105 players, or 9% of the sample reported sleeping fewer than 5 hours the night before a tournament, suggests that we may have a collective urge to… overplay… the extent to which we party.
We didn’t lose anyone for this question! The majority of us — 935 players, or 81% of the sample — are eating breakfast before we sling spells. It would be interesting to know, though, “what constitutes breakfast?”
And now back to playtesting.
It looks like the majority of us are involved in fairly small playtesting groups, with 822 players, or 71% of the sample, preparing for tournaments with between 1 and 6 other Magic players. The extremes on both ends (preparing with more than 10 Magic players or preparing by oneself) are similarly small, at 61 players and 79 players, respectively.
It looks like the vast majority of us are between ages 18 and 34, with 1,041 players, or 90% of the sample, falling between those ages.
Now, on to DCI rating. While I asked each player for their specific DCI Total rating, I’ve decided to use the rating breakdown for Grand Prix byes to separate the variable for presentation and analysis (i.e., less than 1850, 1850-1949, 1950-2049, greater than 2049). Before looking at this data, I had to remove individuals who reported unrealistic ratings. On the high end, at the time of the survey’s completion, the highest DCI Total rating in the world was 2212, and so I eliminated all responses with a rating over 2212. In addition, I eliminated all responses indicating a rating lower than 1400. This left us with 1,039 players, the majority of whom had a rating of 1849 or lower (769 players, or 74% of the sample).
And now, we move on to the final variable: success at Magic: The Gathering. Unfortunately, there isn’t a single variable that can really, truly capture what it means to be successful at this game. At the most basic level, each of us defines success differently. For the purposes of this survey, though, success was tied to achieving beyond a specific threshold—making the Top 8 of an SCG Open, Pro Tour Qualifier, State or Regional Championship, or National Championship, or the Top 32 of a Grand Prix, Pro Tour, or SCG Invitational. However, we can’t just set up a count because such a variable treats the following two individuals the same:
Player 1: An excellent Magic player, but very busy. Played in one tournament in the past year, an SCG Open, in which he took 4th place.
Player 2: A mediocre but dedicated Magic player. Played in thirty tournaments in the past year, and took 7th place in a Pro Tour Qualifier once.
My inclination is that these two exemplars display different levels of “success.” What we want to examine, really, is success above a mean threshold. In order to do this, we also asked the question: In the past year, how many times have you played in an SCG Open or Invitational, Pro Tour Qualifier, State or Regional Championship, National Championship, Grand Prix, or Pro Tour?
In order to look at success, I generated a variable that is defined as (Number Top 8 or 32 appearances / Number of tournament appearances). This variable assigned all remaining Magic players (after eliminating erroneous responses, such as players with ‘numtop8/32 > numtournament’) a value between 0 and 1. Because we want to look at a better than average performance, I took the mean value of this variable across the remaining 786 Magic players, which is 0.13 (the median value, for those who are interested, is 0.0). I categorized all players with a value of 0.13 or higher as having had ‘success.’ For the record, this method prevented 26 individuals with at least one Top 8 or Top 32 from being defined as ‘successful.’
I realize that this method doesn’t fully capture what it means to be successful (i.e., an individual can play in one tournament, get lucky, and be categorized as having had success). However, only 2.8% of the sample played in a single tournament and also made the Top 8 or Top 32 of that tournament, and so such a pattern probably did not skew the data unreasonably.
But what about Building a Champion, the idea behind the survey in the first place? While I initially wanted to include all of the variables that we investigated, including Total DCI rating, the fact that succeeding in a tournament causally influences rating suggests that it might not be appropriate, theoretically, to include that variable in the overall model. For the sake of interest, I ran a simple logistic regression model using DCI Total rating and tournament success prior to the main analysis.
Recall that DCI Total rating has four categories:
< 1850
1850-1949
1950-2049
2050+
I calculated unadjusted odds ratios (UOR) for DCI Total rating, using the first category (< 1850) as the comparison category. I found that, compared to players with a rating lower than 1850, players with a rating between 1850 and 1949 are 3.48 times as likely to have achieved success. Similarly, we find that compared to players with a rating lower than 1850, players with a rating between 1950 and 2049 are 13.62 times as likely to have achieved tournament success. Unfortunately, because only 16 players in the sample both had a rating over 2049 and reported accurate data, the UOR for that group has a high standard error, meaning that it cannot be interpreted with as much confidence (for the record, those players purportedly are 58.38 times as likely to have achieved tournament success).
All of these findings are statistically significant (p < .001), suggesting that there is a very small probability (less than 0.1%) that they was observed due to chance alone. [card name="Recall"]Recall[/card], however, that doing well in a tournament causes one's rating to increase, and that all of the data used in this study is cross-sectional and retrospective (i.e., all tournaments played will already have influenced rating). As such, these values reasonably are extreme, and these results are not surprising.
Now, let’s move to the theory itself.
Based on what we’ve read and what we’ve experienced as Magic players, we might suppose that we can predict tournament success based on the model presented above (for more information on the theory behind these ideas, I suggest the articles cited in my lead-in article “Building a Better Magic Player: We Have the Technology”). Because the outcome (“tournament success”) is binary (either “yes” or “no”), we can run a simple logistic regression model to examine adjusted odds ratios (AOR, we’ll get into how to interpret these in a moment).
It is important to understand that significant findings reported here don’t infer causality (i.e., we don’t prove that A causes B, only that those who are members of category A are more likely to be members of category B by a certain percentage).
We find that two variables, in particular, predict tournament success, controlling for all of the variables involved in the model.
- Compared to those who ‘strongly agree’ that they play a deck with which they are comfortable, even if it’s not considered to be the “best deck,” those who ‘disagree’ are 1.70 times as likely (or 70% more likely) to have achieved tournament success. This finding is statistically significant (p = .039). Interestingly, those who ‘strongly disagree’ are not statistically more likely to achieve success, suggesting that some level of moderation (generally playing the best deck, but not taking it to extremes) may be the best approach.
- Compared to those who playtest with more than 10 Magic players for a tournament, those who playtest with 1-6 other players are 1.93 times less likely (or 93% less likely) to achieve tournament success. This finding is statistically significant (p = .030). Similarly, compared to those who playtest with more than 10 Magic players for a tournament, those who playtest with 6-10 other players are 2.44 times less likely to achieve tournament success. This finding is statistically significant (p = .009). Interestingly, playtesting by oneself (i.e., with no other players) does not appear to harm our chances of achieving success relative to playtesting with more than 10 players.
Players who belong to these groups (I summarize them in non-precise terms in the image) are more likely than those who do not to achieve tournament success. As a disclaimer, there are a number of limitations to these analyses. Most importantly, the sample was non-random, and so the extent to which it can be generalized to the population of Magic players is limited. Similarly, we do not know the extent to which the survey that I used was valid (measured what we intended it to measure) or reliable (would produce the same results if administered to the same individuals in similar conditions). As such, the results should be approached with caution.
However, I do think that this gives us a good starting point in attempting to become more successful players! We might consider branching beyond our comfort levels and playing the best deck more frequently, and we might consider expanding our playgroups more significantly because it seems as though we benefit from increased input when it comes time to rack up finishes.