At the request of Andy, I'm reposting here:
The restriction of Thorn of Amethyst and Monastery Mentor took effect September 1 2017. That means we are over 6 months out from that restriction and I think it is worth looking back at how effective those moves were. As many of you know, Ryan Eberhart (@diophan) and I spend quite a bit of time collecting metagame data from major paper events and the MTGO challenges. We do this for a couple of reasons. Personally, I use this data when it comes to creating new decks. The version of Snapcaster Control that I've played in the last three challenges was heavily influenced by what I saw from our challenge data. The prevalence of Shops and Planeswalkers in both Oath and Xerox (i.e. cantrip heavy blue decks) motivated me to shift the removal suite to Lightning Bolts and Fiery Confluences instead of Swords to Plowshares and Balance. This was further justified by an absence of Eldrazi and Merfolk decks in the format. Honestly, that was my primary reason and hope when we started collecting data: that what we gathered would be used to promote innovation in a small format like Vintage.
Alternatively, Ryan and I wanted to provide an accurate picture of the Vintage metagame for use in discussions involving the Restricted list. Much of what we read previously tended to be hyperbolic, opinionated, and poorly reasoned. We hoped people would use our data in forming conclusions like scientists or researchers. In both cases what we wished to happen didn't actually happen. Most responses to our posts consisted of hyperbolic, opinionated, and poorly reasoned arguments, just now with cherry-picked data. There was very little commentary on trends and how to combat them, no brewing of decks. We went from posting results weekly after each Challenge, to monthly aggregations of the previous month's events, to not posting or gathering data from February. In effect, Ryan and I burnt out, on both playing Vintage and collecting data about the format. We asked for help and nothing really materialized. The reason I'm bringing this up is that I don't know if we will continue this in the future. So if you do find this beneficial, please let us know and considering helping out if you play Vintage on MTGO. The Challenges continue to be excellent EV, with the top 32 (basically any 3-3 and several 2-4s) making their entry fee back. Power is affordable - a set of VMA Power 9 costs less than 100 dollars. Complete decks range from 120 tix for Dredge, 300 tix for DPS, 500 tix for Ravager Shops, and 700 tix for UWR Mentor or UW Landstill. Which serves as a pretty good segue into the next section...
Paper vs Online Metagames
We hear a lot of comments concerning real or perceived differences between these two metagames, often in the context of B&R discussions. While I appreciate that players may play Vintage in widely divergent paper metagames, that doesn't invalidate data collected in other metagames. At the end of the day, the DCI is going to base their decisions on the data they have available. This likely is limited to the large sanctioned events of European, North American, and Japanese Eternal Weekends, along with the results from MTGO Leagues and Vintage Challenges, so that's where we've focused our efforts. And, frankly, the MTGO metagame has several advantages compared to paper Vintage. The cost of decks is lower, even considering proxies, which allows players more freedom in deck selection. Events are more frequent and typically larger than their paper counterparts. We are looking at four 40+ person events whereas a local tournament may have one monthly event with 17-32 players. This gives us a much larger sample size from which to draw conclusions. And finally, players on MTGO tend to do very well in paper tournaments such as the 2018 North American Champs. Winner Andrew Markiton (MTGO: Montolio), finalist Rich Shay (The Atog Lord), Patrick Fehling (Clone9), Brian Kelly (brianpk80), and Eric Vergo (caggii) all are regulars on MTGO.
Before the Restriction
The Gush and Gitaxian Probe restriction took effect April 24 2017, so we used the May through July challenges to establish a baseline. Individual events can be found by searching TMD, but the compiled data is available here.
Following the Restriction
We changed our spreadsheets when we moved to monthly reporting. It allowed us to do a month-by-month breakdown of events. Note: February's metagame breakdown is drawn from the Top 32 results of that month's challenges. As mentioned previously, we didn't do our usual data collection for that month. Also, January is missing an event in which Ryan and I were unable to participate.
As can be seen by the monthly breakdown, October and November are dramatically different from the other months. The most likely explanation is that the proximity of these events to the North American Vintage Champs altered player attendance and behavior. North American Vintage Champs was October 19-21 and Ravager Shops was absolutely dominant. It met in the finals, won the tournament, placed 5 decks in the top 8, 11 decks in the top 32, and had a 58.9% win rate against the field. Yet on MTGO, Shops portion of the metagame actually fell. Among many players, there was concern that a Shops restriction was imminent, so they played other decks, leaving Shops as the "best deck" primarily played by those unfamiliar to the format. Many established players flocked to the deck that supposedly "beat" Shops, Inferno Titan Oath, as Oath's percentage of the metagame tripled from 6.4% in September to 19.0% in November. And still others went next level the various Mentor/Xerox decks that tend to beat Oath. Those decks put up an impressive 63% win rate in October and November. Now I'm not in the habit of ignoring data, but data should make sense. If it doesn't, you have to wonder what factors might be influencing or introducing bias into your study.
If you exclude the October and November like we did above, there is a remarkably consistent picture of Shops' dominance.The combined results show a 59.0% win rate, virtually identical to the 58.9% win rate at Champs, slightly decreased from the 59.2% win rate in the pre-Thorn metagame. The metagame share is slightly decreased but trending upward. These trends seem to hold so far in March, as you can see below. Shops has a 31.4% metagame share and a 62.1% win rate. In my opinion, the results from October and November appear as outliers rather than a true indicator of Shops place in the metagame. However, one of the reasons to write these in-depth reports is to solicit differing opinions, similar to peer-review. I invite whoever is so inclined to chime in below with their thoughts. If you feel this is some sort of adaptation by the rest of the metagame, I am curious to hear what you think that was and why the metagame revert back to its previous state.
What beats Shops?
For those that do not follow other formats, Standard underwent several bannings in January. Ian Duke's explanation of those bans is well worth a read as it provides useful insight into WotC's reasoning and approach to B&R decisions. Ian spends quite a bit of time discussion the matchups of Standards top 2 decks, Temur Energy and Ramunap Red, and how these decks have a favorable matchup profile against the field, suggesting that the metagame is unable to adjust. Let's take a look at Shops' matchup profile since September:
With November and September removed:
Ironically, Shops only "bad" matchup (and I admit to being a bit lazy with the statistics here - if you want the raw data and the sample sizes, it's here), is the "Other" category where we throw decks that don't fit into other categories. Apparently, the Monored Hate deck with Null Rod and Ensnaring Bridge went 5-0 against Shops in October and November... Outside of what are essentially rogue decks, Shops either has a good matchup of >55% or is essentially even (between 45% and 55%). This includes Oath, which is Shops' worse matchup but only at 47.5%. Decks that were traditionally thought to be good matchups, like Dredge and Landstill (the most popular variant in the"Blue Control" category) actually end up struggling against Shops.
The goal of this post isn't to propose specific actions: it's to establish the need for such action. The previous restriction of Thorn of Amethyst has not discernibly altered the win rate or metagame share of Shops in the MTGO Vintage Challenges or in the NA Vintage Championship. If such action was indicated then, it holds that additional action is indicated now. Of course I have my own opinion on what I think should be done. However, I want to allow some time for players to read and process this. Comments, thoughts, and opinions are welcome and encouraged.
Edit: Added Archetype vs Archetype Win Rates with the Champs months removed.