@brianpk80 said in Vintage Challenge - 5/29/17:

The DCI has essentially invited complaining and whining on an unprecedented scale now. Any restriction that isn't strictly necessary is one too many, IMO.

The DCI is finally listening to concerns of Vintage players after nearly a decade of neglect.

Except your own poll disproves this. The DCI is not listening to most players, and acceded, instead, to the wishes of a vocal minority.

But let's take your larger point at face value, that after years of what you call "neglect," the DCI should not only be more aggressive in management of the format, but should also embark on a more ambitious hunt for possible restrictions.

I would submit that such a project is inherently divisive and ultimately corrosive, even if there are upsides. Only a minority of players supported the restriction of Gush, and a tiny minority the restriction of Probe.

If the DCI undertakes further restrictions to placate a small minority, it will ultimately engender anger, resentment, and cynicism among the larger part, and ultimately damage the format, as many past restrictions have. Rich claims that the restriction of Brainstorm killed New England Vintage for a time. While I think that is an overstatement, it is undeniable that restrictions harm. Witness the anger after Lodestone Golem's restriction.

There is far too much complacency in calls for restriction. Too much ease and lack of circumspection in which they are demanded or proposed. The restriction of Gush and Probe will invite more unnecessary and harmful restrictions.

The restriction of Gush and Probe should be a wake up call, or a firebell in the night (you quoted one President, so I'm following suit ;p).

None of the predictions the DCI put forward have occurred or are likely to, and it should be nothing less than a total and abject embarrassment for the DCI. To have been so humiliatingly misled into a pair of restrictions that have had no empirically measurable effect whatsoever, let alone the effects they predicted, should garner nothing less than withering scorn. Pretty much every aspect of this B&R list change, from the predictions to the explanation to the hamfisted timing to clumsy setup in March to the overall mode and method of soliciting feedback has been mismanagement of the format of the highest order, and merits universal opprobrium.

last edited by Smmenen

@Smmenen said in Vintage Challenge - 5/29/17:

If you compare these results with any of the recent months this year pre-restriction, it's pretty much the exact same metagame - or at least, within the same parameters - half Shops, two Mentor, and a Delver with another random blue deck in the Top 8.

This is the point that I disagree with insofar as I think that the current results shown on mtgo mirror closely the "immediately ater a big change the decks that were still good from the previous incarnation dominate" that other Magic formats experience.

That we then have two paper events with a larger distribution of different decks (including multiple blue decks cutting back on or eschewing mentor completely) in the top 8.

I still think we are in the midst of the evolution of the metagame, and that it is too early to decide if the DCI "succeeded or failed." (If such can even be judged, given a lack of insight into their reasoning)

I suppose what I'm saying is there are fewer mentors in two recent paper top 8's than there were in typical top 8's pre-restriction (and if I am wrong in this generalization I welcome that data to indicate otherwise). I think we are still seeing things develop, and that it will take more than a month for things to settle in.

Anyone know the p value of Brian's poll? Can we stop referencing that as some impervious correlation to the player base's actual collective opinion?

I'll be honest, I think some of you guys are giving the DCI way too much credit regarding how much consideration they give the Vintage format. And while I feel like decoding their quarterly messages can be fun and entertaining, I think there is almost zero value to be gleaned from the exercise. I wouldn't be surprised if their Friday lunch order received more contemplation than the Vintage format does.

I do believe the DCI has metrics for when a format becomes "unfun." Keep in mind, this is a business. If the games's not fun, people aren't playing as much.

I do believe MTGO provides them with a ton of information that could determine if people are having fun (player frequency patterns, rage quits, game length, chat activity). There is no indication that any of this information is being used, however.

We have circumstantial evidence that prominent players have voiced and been asked their opinion on the format for years. Some of those players have close ties to the DCI. It's not hard to assume they lean on the opinions of prominent players or ex-testers from time to time. I think they've even admitted they don't playtest the format at all, so this is understandable (even though I don't agree with it).

I think it's always been a shoot from the hip exercise for Vintage. Sometimes they get it right, sometimes not.

@enderfall THANK YOU. The fact that this poll of a small but vocal portion of the community is being brought up at all as having any sort of value behind its face value of "this is what some players gathered in one place think" is honestly quite silly. It's exactly the kind of data that the DCI should be ignoring when making such decisions and any argument relying on that data is likewise something that probably doesn't deserve serious consideration.

@enderfall The pvalue is most definitely significant in terms of wanting change. This wasn't a small sample, and I did a sample bias validation here on the drain. The results were largely the same as on the Facebook poll. Confirming that the Facebook sampling bias was unlikely.

The sample of Yes+No was 194. Of that sample:
Change: 141
Mentor: 118
Gush: 71
No Change: 53
Probe: 44
Bunch of other options...

You get the following 95% CIs if we set the denominator to that Yes+No group:
Change: 65.84 to 78.82
Mentor: 53.58 to 67.74
Gush: 29.81 to 43.80
No Change: 21.18 to 34.16
Gitaxian Probe: 16.99 to 29.23

The smallest vocal minority seems to be those that are insisting change wasn't wanted. Based on the estimation, the number who wanted change at a bare minimum was 2/3 of the Vintage population. At least half wanted Mentor to get hit.

Wizards decided that Mentor was a symptom though, and not the disease so they hit Gush+Probe. I'd agree with this assessment as there are numerous other cards that also benefit from 0 CC draw spells.

@Smmenen off topic but I just want to point out that the placebo effect is a legitimate treatment, especially in the sense of somatic disorders where mental illness may manifest as physical symptoms. You give the patient reassurance and placebo to improve his/her mental health and the physical symptoms may improve.

On second thought, Rich's protest should be a qualified vow of silence, where he ONLY says the word "deterministic". Imagine Hodor, but infinitely better. (Imagine him ordering fast food.)

@Hydra said in Vintage Challenge - 5/29/17:

@enderfall THANK YOU. The fact that this poll of a small but vocal portion of the community is being brought up at all as having any sort of value behind its face value of "this is what some players gathered in one place think" is honestly quite silly. It's exactly the kind of data that the DCI should be ignoring when making such decisions and any argument relying on that data is likewise something that probably doesn't deserve serious consideration.

Yes, like all climate change denialists, flat earthers, etc. Why should we use data or science? You really think that random solicited email feedback they've solicited is more representative or statistically reliable than these larger polls? Because that's the alternative. The DCI explicitly invited feedback. Brian's polls are much more representative than random email feedback.

Brian's survey follows a sampling methodology that is used in many other ways, such as political or marketing polling. Like all samples of large groups, it can't represent the entire population, but the sample was large enough to generate statistically significant insights, as Vaughnbros pointed out. Moreover, it wasn't just a Brian's poll. If you line up the other polls, such as the two taken on this board, they corroborate each other, despite having different samples.

There is simply no way to crunch the numbers to suggest that "most players" wanted Gush restricted. It's like Donald Trump's inauguration attendance boast. And anyone claiming that most Vintage players did want Gush restricted has the burden of presenting data that contradicts that poll or is stronger than the poll Brian created. Otherwise, they are literally making up facts.

There is a broader issue here though.

We are now nearly 25 years into the experience of Magic, and more than two decades of constructed Magic, and the DCI is essentially operating no differently than it did in the game's early years.

I think the DCI needs to substantially improve the rigor of it's approach. It has the tools. Survey methodologies are now not only more refined, but less costly thanks to internet tools such as survey monkey. It shouldn't be hard for the DCI to create a panel of Vintage players, and track survey responses over time or to create surveys of the player base.

Hell, Wizards already does this for marketing and sales purposes and product development. There is no good reason that they couldn't also do this for DCI management. If the DCI really wants to track player feelings regarding matters, that is a much more valid approach then their current one, of soliciting random email/social media feedback, which is hardly random. Email and social media feedback will always be biased towards those who want a change, and undercount those who prefer the status quo. It's a terrible way to manage formats, and hardly representative of the player base.

It's also possible that there are much stronger quantitative and computer modeling tools available to the DCI now, and that it should employ these tools to manage formats. This isn't the 1990s, yet the DCI appears stuck there.

@Naixin said in Vintage Challenge - 5/29/17:

@Smmenen off topic but I just want to point out that the placebo effect is a legitimate treatment, especially in the sense of somatic disorders where mental illness may manifest as physical symptoms. You give the patient reassurance and placebo to improve his/her mental health and the physical symptoms may improve.

Yes, I know that. I also pointed out that the use of placebo is legitimate as part of a study. But if someone were to defend the restriction of Gush as an effective placebo treatment, they've already lost the argument. We don't prescribe placebo's with bad side effects.

last edited by Smmenen

@vaughnbros

I think what @enderfall and @hydra were getting at is that the set of "vintage players who post on TMD" and "vintage players who post in (one of) the (private) vintage facebook group(s)" are already a small subset of the greater vintage community. Not only are they small groups, they're probably not a very representative sample of the whole, and there's probably a huge amount of overlap between the voters in both polls.

As an optional poll, both biased heavily toward people who like talking about B&R, who like using social media, and (something we tend to forget a lot when making sweeping judgements about the community) they ALL speak English.

For what it's worth, while I am an active member of TMD and the Facebook group in question, play a lot of vintage and had very strong opinions on the decision, I did not vote in either poll, because I try not to contribute to those sorts of threads (obviously I fail at that goal sometimes, like now).

Not only do I think that a TMD/Facebook poll is a non-representative sample of the vintage community ... I'm not even convinced that TMD/Facebook have a consensus on what the vintage community even IS. I certainly haven't seen much discussion on the subject.

When it comes to B&R, WotC cares about the format being fun. But fun for who? A lot of TMD users don't actually play a lot of vintage. A lot of active vintage players play in small metagames that have different metagame issues. Do MTGO-exclusive vintage players count less? Do they count more? There are a lot more vintage players that play 1-3 times a year than there are that play once a week (clearly true from tournament attendance numbers), and policy changes impact those two groups differently.

Maybe most dramatically, I'm pretty convinced that the amount of vintage-fans DWARFS the amount of vintage-players. That is, people who follow vintage coverage, read vintage articles, watch the VSL and Eternal Weekend, but do not play vintage and have no interest in playing vintage. These people still care about the B&R list, and they absolutely speak up about those opinions. There are absolutely decks and strategies and metagames that are more or less entertaining to watch than they are to play. There are more of them than there are of us, and I would imagine they spend as much (or much more) on the game, on average, than vintage players do.

I'm really doubting that polls (and the discussion threads) we have on TMD and Facebook (and Twitter and Reddit and anecdotally talking with friends) are representative of the Vintage community at large ... and if I'm being perfectly honest, I'm not sure what "The Vintage Community" even means. I'm unsure if WotC has their own definition, but I know for TheManaDrain, it's a giant question I've been struggling to answer.

@vaughnbros said in Vintage Challenge - 5/29/17:

@enderfall The pvalue is most definitely significant in terms of wanting change. This wasn't a small sample, and I did a sample bias validation here on the drain. The results were largely the same as on the Facebook poll. Confirming that the Facebook sampling bias was unlikely.

The sample of Yes+No was 194. Of that sample:
Change: 141
Mentor: 118
Gush: 71
No Change: 53
Probe: 44
Bunch of other options...

You get the following 95% CIs if we set the denominator to that Yes+No group:
Change: 65.84 to 78.82
Mentor: 53.58 to 67.74
Gush: 29.81 to 43.80
No Change: 21.18 to 34.16
Gitaxian Probe: 16.99 to 29.23

The smallest vocal minority seems to be those that are insisting change wasn't wanted. Based on the estimation, the number who wanted change at a bare minimum was 2/3 of the Vintage population. At least half wanted Mentor to get hit.

At a minimum. And the top end of that range is 67% of the Vintage player base wanted Mentor restricted. Even the top end of the 'wanted Gush restricted' range is 43.80%, which is still far short of a majority.

Rich claimed that most players wanted Gush restricted. There is no reasonable parsing of that poll to suggest as much.

Wizards decided that Mentor was a symptom though, and not the disease so they hit Gush+Probe. I'd agree with this assessment as there are numerous other cards that also benefit from 0 CC draw spells.

And, the metagame results so far, show this to be wrong. The disease continues untreated, as there has been no meaningful change in the Vintage metagame, based upon the premier event results thus far.

We've suffered restrictions for no apparent benefit, with unnecessary collateral damage, including harming decks like DPS, Doomsday, etc.

last edited by Smmenen

@Smmenen @Brass-Man already hit the nail on the head, but it bears repeating: the subset participating in those polls that were conducted are far too limited to make the data actually worthwhile. The number of people who participate in one way or another with the format dwarves the number of people participating on TMD or as part of that specific Facebook group, and in order for extrapolation of the data to be valid you have to have some way of making sure the group you are using is actually representative of the larger group. In this instance the specific requirements are too exclusive for the extrapolation to be meaningful. I work for Nielsen collecting TV ratings for a living, I'm familiar with the methodology behind all of that and feel confident in saying that there's not enough diversity in the sample in question, no matter the actual number of people within these groups that participated in those polls. Even if we represented a sizeable population of those who participate in Vintage magic (we don't), there's too many exclusions involved in the narrow scope to make any general statements from it.

Edit: and that isn't even addressing the cross-contamination aspect of the sample having multiple individuals replying to the different polls, adding in meaningless repeat data points that there's no real viable way to scrub without actually knowing everyone's Facebook and TMD username to eliminate the extra votes.

last edited by Hydra

To be clear, I don't think that the poll is completely worthless. It does give us insight into the opinions of a subset of TMD/Facebook users.

I just think it's a stretch to say that WotC made a decision that conflicts with the data, when we don't even know the audience they're trying to impact with this change. WotC and the TMD/Facebook community have necessarily (sometimes even adversarially) different goals.

@Brass-Man I don't think it's completely worthless either, as I said from the start I think the value it has is simply it's face value: people in these limited places feel X about it. It is, however, worthless for the purpose Steve and others seem to want to use it for, because it's not broad enough to extrapolate from in a meaningful way.

@Hydra said in Vintage Challenge - 5/29/17:

@Smmenen @Brass-Man already hit the nail on the head, but it bears repeating: the subset participating in those polls that were conducted are far too limited to make the data actually worthwhile. The number of people who participate in one way or another with the format dwarves the number of people participating on TMD or as part of that specific Facebook group,

That's not true.

First of all, the Facebook group has over 1000 people. Estimates of the number of active MTGO Vintage players is probably under 100, and no more than 200 at most. There isn't a single tournament in the year that even approaches the size of the FB group. It very likely contains large portions of any significant Vintage player base in North America and Europe.

Second, you haven't made any claims or assessments about the diversity of the Facebook group or respondents that would support your claim. I disagree that it's not representative. It's got global membership and features most of the players who actually play Vintage regularly. How do you know it's not the diverse in the ways you feel are important?

Even if it's true that it doesn't have the people who play once or twice a year, I think it's fair to say that it's probably biased in favor of people who want to see change, rather than don't, reinforcing my argument, not undermining it.

Third, and most importantly, it's the best we have. You said the DCI should ignore polls. Yet, it's actively soliciting email feedback. That's a far worse, and far less representative technique. Far better to use polls/survey, no matter their limitations, than to make decisions based upon statistically insignificant data.

.

@Hydra said in Vintage Challenge - 5/29/17:

@Brass-Man I don't think it's completely worthless either, as I said from the start I think the value it has is simply it's face value: people in these limited places feel X about it. It is, however, worthless for the purpose Steve and others seem to want to use it for, because it's not broad enough to extrapolate from in a meaningful way.

Which is what exactly? Because I think you are misreading the purpose for which I introduced the poll into evidence in this thread...

last edited by Smmenen

@Brass-Man

I don't really understand this mindset that no data is better than some data. We have poll results showing that the restriction was wanted by a statistically significant number of people. By saying all polls are going to be flawed. What you are really saying is there is no way to get any information on what the player base wants. That just seems false.

Is it possible there are other samples to use? Sure. How about metagame data that showed a large number of people bringing Mentor Gush to tournaments. That's a strong indication that they thought it was the best deck. How would you even get at the actual opinions of this more "accurate" sample? Force people to take a survey before/after at each and every event?

@Smmenen said in Vintage Challenge - 5/29/17:

At a minimum. And the top end of that range is 67% of the Vintage player base wanted Mentor restricted. Even the top end of the 'wanted Gush restricted' range is 43.80%, which is still far short of a majority.

Rich claimed that most players wanted Gush restricted. There is no reasonable parsing of that poll to suggest as much.

I came to post the results of the poll with Confidence Intervals as some degree of error was asked for them. I'm not here to pick a side. I'll help you to interpret if you wish though too.

You are correct that 43.8% falls short of a majority, and that clearly there were more people that wanted Mentor restricted than Gush (evident by the intervals not overlapping). However, if you phrase the question instead of as what % of Vintage player's wanted this specific change, and instead framed it as: How many wanted this change vs no change at all? You'd realize that Gush is above No change. The difference is NOT statistically significant (evident by the intervals overlapping), but there is still a difference suggesting that more would have favored a Gush restriction than no action at all.

As to the rest of your post. Would you consider 1 month a sufficient amount of time to make a conclusion?

It seems to me the meta is still feeling its way out. Evident by 0 Eldrazi, 1 Oath down from 5 Eldrazi, 4 oath last week. Also evident by Thorns having a 44% win% last week and a 70% win% this week along with PO dropping from a 56% win% to a 34% one. People clearly came less prepared for Shops and more prepared for PO. These types of fluctuations are a pretty clear indication that the metagame is not yet solidified. Some people seem to just be playing whatever was good pre-restriction. But guess what Thorn+Mentor was not even 50% of the metagame this week or last! (falling just short of that mark) This is well down from the 60+% that it was at in the pre-restriction March P9 tournament.

@Hydra

How do we get at the whole population? If keeping track of player's actual interests is important how do we do it?

last edited by vaughnbros

@vaughnbros said in Vintage Challenge - 5/29/17:

@Brass-Man

I don't really understand this mindset that no data is better than some data.

Exactly. Or worse: completely anecdotal data.

Hydra's ridiculous claim that the DCI should completely ignore such polls, which are actually pretty robust in terms of their sample size, would leave no alternative but essentially relying on email feedback, which has no statistical value whatsoever, and is biased in predictable ways.

We have poll results showing that the restriction was wanted by a statistically significant number of people. By saying all polls are going to be flawed. What you are really saying is there is no way to get any information on what the player base wants. That just seems false.

Exactly. Moreover, if people are going to make empirical claims, it requires empirical evidence. We can quibble over the value of that evidence or caveat or critique it, but it's the best evidence we have.

If Rich - or anyone else - is going to claim that "most players wanted Gush restricted," they should be required to present evidence to support such claims, especially when we have decent evidence that suggests the exact opposite.

last edited by Smmenen

@Smmenen The issue with the data provided is coming from a very specific, niche group: players who have active online presences that either A. Participate in the Vintage Facebook group or B. Read TheManaDrain. Diversity within that group is meaningless because you have already put constraints on the data that are too exclusive. Magic has an online component that people can chose to participate in, but it's a paper game, and the vast majority of the player base isn't going to have enough of an online presence to weigh in on these things, which is why even when they solicit online feedback they assuredly are looking at data from other sources that we don't have access to because it's not made public, and any opinion-based data they collect is going to be taken with a grain of salt because of the biased nature of the source. In my line of work, this would be like only counting the TV ratings based on people who subscribe to cable. It doesn't matter how diverse the population within that group might be, by the nature of the data used you are excluding too large a portion of the population for the extrapolation to truly mean anything.

With that in mind, I consider using those polls in a way other than "people who participated in these polls feel X" to be a misuse of the information and misrepresenting data. If you feel this is ridiculous on my part then I don't know what to tell you, other than if you were hired by a company to collect general data on their product and presented only these polls they would throw it out for being too narrow.

You've already stated all those points before and I've already responded to each of them in order.

First, the vintage Facebook group has over 1000 people - that's far larger than any player base, and you keep ignoring the fact. The vintage Facebook group is larger than any known vintage community and has representatives from virtually all of them.

I think you misapprehend the nature of the vintage community. The vintage communitt was born off of websites like this one, starting on Bdominia. It was how we actually built a tournament scene out of a dead format. Sites like this constitute the Vintage community. There was no vintage community (then call type one) before websites like this one.

Second, you said:

"With that in mind, I consider using those polls in a way other than "people who participated in these polls feel X" to be a misuse of the information and misrepresenting data. "

Well I don't know what to tell you but the idea that a polling or survey sample cant be representative a larger group flies in the face of common every day practice. that's why all the samples have margins of error, like ACS samples.

social scientists, businesses and government institutions use surveys that way all the time. Groups track public opinion on any number of matters test products that way and do election polling.

Finally if we were to ignore polls, we would be left with anecdotal data which is far far worse. I consider it incredibly hypocritical for someone who cares about data validity to suggest that the DCI should ignore polling data while it actively solicitates email feedback. that's what's ridiculous.

Anecdotal email feedback can't possibly be more representative or less biased then large sample polling data from a large diverse population.

last edited by Smmenen
  • 73
    Posts
  • 23936
    Views