Tuesday, September 14, 2010
Pending changes: Further thoughts
Thanks to Jospeh Reagle (here's his blog) for pointing me to Wikipedia:Pending changes/Metrics/Preliminary Analysis. I hadn't seen it before my last post, and I appreciate the work that Howief into quantifying this.
However, my first impression of reading Howief's good-faith attempt here reinforces my suspicion, voiced here and elsewhere, that Pending Changes is something no one who supports it actually has a rational reason for adopting it. Most of his analysis is devoted to simply getting a handle on what happened, which is borne out by his statement at the beginning: "This analysis is meant to serve as a starting point to help focus the data for community discussion." No one began this test with expressed assumptions of what would happen, and now that it is over no one can say whether the test succeeded or failed -- and why.
Chris.urs-o's criticism of the analysis is very insightful, better than what I wanted to say. The one point Chris did not raise, which I'll make here, is that no data was provided for just how long did it take for a change caught by the Pending changes software to be processed. This is an important piece of information for considering just how far will pending changes scale. No matter how well this change might work, only so many editors will review and act on the edits Pending changes affects; if not enough editors participate, pending changes will break at some point.
I could ask for more analysis, but I suspect that the fix is in: love it, hate it, or just don't care, Pending changes will be foisted upon the English Wikipedia. According to this story in the latest Signpost, Jimmy Wales was asked by the Wikimedia Foundation to interpret the discussion on Pending changes. Despite the general belief that voting is not a viable resolution for disputes, Wales states that it is "clear that there’s absolutely no consensus for simply turning the system off and walking away."
As I have written, Wales is not impartial on this matter: he has advocated for Pending changes in the past. Noting the straw poll has been closed at a 65% to 35% vote in favor of Pending changes, he claims "that there’s absolutely no consensus for simply turning the system off and walking away." This is in disagreement with a far more thoughtful interpretation by Cenarium. Cenarium concluded that, despite C's own views on Pending changes, there was no consensus to keep it in place; however, this did not mean Pending changes could not be adopted in the future, rather that it was time to build on consensus and "calmly analyze the trial, the merits of PC, and discuss of possible new implementation proposals which would have to acquire consensus for adoption". But in the end, it appears when Wales is in favor of something, although over one in three people are opposed to it, he believes their opinions are irrelevant. I always thought that "consensus" was something that all participants could agree to; I guess I was wrong.
I don't mean to sound cynical in that paragraph; as I type this I actually feel closer to sadness and disappointment. I don't consider Pending changes to be something so critical to Wikipedia's survival to justify overriding the desires of such a large group of people who respectfully disagree. If it's a good thing, it will eventually be adopted; if it isn't, then we shouldn't be in a hurry to adopt it. Approaching this disagreement in this way, those who work to develop and refine policy will gain the confidence of the rest of the Wikipedia community, who would rather work on its contents than engage in the bitter, and often inconclusive, disputes over the policy. I know I would rather be able to have faith and trust in them, but despite my frequent displays of cynicism I am probably incredibly naive.
Geoff
Technocrati tags: online information, wikipedia
However, my first impression of reading Howief's good-faith attempt here reinforces my suspicion, voiced here and elsewhere, that Pending Changes is something no one who supports it actually has a rational reason for adopting it. Most of his analysis is devoted to simply getting a handle on what happened, which is borne out by his statement at the beginning: "This analysis is meant to serve as a starting point to help focus the data for community discussion." No one began this test with expressed assumptions of what would happen, and now that it is over no one can say whether the test succeeded or failed -- and why.
Chris.urs-o's criticism of the analysis is very insightful, better than what I wanted to say. The one point Chris did not raise, which I'll make here, is that no data was provided for just how long did it take for a change caught by the Pending changes software to be processed. This is an important piece of information for considering just how far will pending changes scale. No matter how well this change might work, only so many editors will review and act on the edits Pending changes affects; if not enough editors participate, pending changes will break at some point.
I could ask for more analysis, but I suspect that the fix is in: love it, hate it, or just don't care, Pending changes will be foisted upon the English Wikipedia. According to this story in the latest Signpost, Jimmy Wales was asked by the Wikimedia Foundation to interpret the discussion on Pending changes. Despite the general belief that voting is not a viable resolution for disputes, Wales states that it is "clear that there’s absolutely no consensus for simply turning the system off and walking away."
As I have written, Wales is not impartial on this matter: he has advocated for Pending changes in the past. Noting the straw poll has been closed at a 65% to 35% vote in favor of Pending changes, he claims "that there’s absolutely no consensus for simply turning the system off and walking away." This is in disagreement with a far more thoughtful interpretation by Cenarium. Cenarium concluded that, despite C's own views on Pending changes, there was no consensus to keep it in place; however, this did not mean Pending changes could not be adopted in the future, rather that it was time to build on consensus and "calmly analyze the trial, the merits of PC, and discuss of possible new implementation proposals which would have to acquire consensus for adoption". But in the end, it appears when Wales is in favor of something, although over one in three people are opposed to it, he believes their opinions are irrelevant. I always thought that "consensus" was something that all participants could agree to; I guess I was wrong.
I don't mean to sound cynical in that paragraph; as I type this I actually feel closer to sadness and disappointment. I don't consider Pending changes to be something so critical to Wikipedia's survival to justify overriding the desires of such a large group of people who respectfully disagree. If it's a good thing, it will eventually be adopted; if it isn't, then we shouldn't be in a hurry to adopt it. Approaching this disagreement in this way, those who work to develop and refine policy will gain the confidence of the rest of the Wikipedia community, who would rather work on its contents than engage in the bitter, and often inconclusive, disputes over the policy. I know I would rather be able to have faith and trust in them, but despite my frequent displays of cynicism I am probably incredibly naive.
Geoff
Technocrati tags: online information, wikipedia
Labels: wikipedia
Wednesday, September 08, 2010
Pending changes, & how it defines Wikipedia
Well, the English language Wikipedia I mean.
For those of you who aren't that familiar with the latest conflicts at Wikipedia -- which has now come to include a large share of active Wikipedians -- probably the most important is the deadlock over whether or not to continue a test of the "pending changes" feature.
The proposed feature is simple enough. For certain groups of editors -- those editting anonymously or with a new account -- their contributions are sent to a queue where they must first be approved before being made visible to one and all -- such as people who are simply reading Wikipedia, who make up over 90% of its users. Otherwise, the edit is deleted.
This is something that Jimmy Wales has long advocated for (see here for one example), and as a result many Wikipedians consider it a good thing. And maybe it is; I'm undecided only because after a two month test, no one has presented any factual evidence that it solved any problems. I haven't seen any in the lengthy, at times bitter, and for the moment deadlocked discussion over its adoption. But what I have seen are examples of long-existing tensions that exist over the ideals of Wikipedia, but which have not been adequately discussed. (For all of his supposed experience with Wikipedia, I haven't seen Jimmy Wales even acknowledge their existence.)
One of these tensions is presented quite clearly on Wikipedia's front page: "the encyclopedia anyone can edit". This is both the strength and the weakness of Wikipedia -- that anyone can, and does, edit its articles. The fact that something useful has resulted -- let alone a reference work which is as useful as it is -- has been pointed out so many times to become banal. When critics pointed out that allowing "anyone" to edit, this included vandals and malicious individuals, the response was to point out that bad edits were almost always promptly reverted. And one could say it was part of the price for creating an otherwise invaluable resource; after all, when dealing with experts and powerful people, one must put up with their eccentricities to benefit from them.
However, the steady trend since its inception has been to limit the meaning of "anyone" with more and more exceptions in order to improve its quality and usefulness. Some of these exceptions are quite reasonable: vandals, kooks and cranks, people who simply can't play nice with others. Others fall into a more difficult category, such as those who want to advocate for a specific political agenda or for their own economic gain. One can't do the simple thing and exclude them all because in many cases, they are the exact experts Wikipedia needs in order to be a useful reference. So Wikipedians are forced to develop essays such as Tendentious Editting or Civil POV pushing to define the problem and help each other confront it.
And then there are some policies which came about in a quick, knee-jerk reflex whose rationale have never been adequately explained, such as denying anonymous and new editors from creating articles. This came about because of the Seigenthaler incident, where an anonymous editor made a libelous allegation about John Seigenthaler which remained undetected for four months. This denial has remained mostly out of inertia: denying anonymous editors all editting rights is a perennial proposal which, although always rejected, has strong support. And if one creates an account then waits a few days, one will be able to create articles -- so no one sees this as anything more than an inconvenience for new editors. On the other hand, it is not an effective security fix: determined troublemakers need to simply create an account and wait for it to age before abusing it to wreak havoc. This change is only a speed bump, which may discourage more new editors than it justifies in dissuading vandals; no one really knows.
Now keep in mind that I'm not criticizing putting limits on just who edits; all of us want Wikipedia to be as accurate and useful as possible. My point is that there is a tension here, between allowing anyone to edit and creating reliable content, which fails to be considered in these decisions. And when I tried to participate in the discussion on whether to continue this test, I was unpleasantly surprised to find that no one had provided the needed information to show whether this test actually improved content or not. Instead a lot of anecdotes, first impressions and opinions were being tossed around. I don't know whether to be discouraged or just sad at this.
Before I continue, I'd like to also point out that I'm not saying that these tensions are not a weakness; they are simply a fact of Wikipedia's nature. Further, by identifying the existence of these tensions I am not consciously forcing a Hegelian dialectic upon Wikipedian dynamics; rather, I am picturing a statics diagram with opposing forces in an equilibrium. In other words, this is a situation where modifying one force in the tension without sufficient care or information can destroy the existing equilibrium and lead to everything falling apart.
A second tension illustrated in this debate is an old one: the clued versuses the clueless. This tension has existed either explicitly or implicitly since practically the beginning of Wikipedia. Larry Sanger's notorious emphasis on credentials was simply one extreme version of this tendency. Most Wikipedians, whether they are professional experts, serious amateurs, or just individuals with a desire for learning, tend to be elitists; we spent our spare time reading or tinkering in order to understand something better, while everyone else was watching television, playing games, or otherwise frivolously occupied. And we often did this both at our own monetary expense, and at the derision of our peers. So, yes, Wikipedians can be elitist in how they respond to new editors; it's an assumption experienced editors often find themselves fighting against.
In this conflict, over whether to turn this experiment into a full-scale adoption, I see this tension playing out on several levels. One level is the change itself: that new and unregistered editors are somehow less clued than the established ones. While a reasonable assumption, it inadvertently insults those new editors who are knowledgeable, forcing them to run a gauntlet to make even trivial changes of spelling or punctuation. But another, and more serious, level can be seen in how the two camps are discussing this proposal: many of the participants seem from their tone to be one step away from responding to their opposite number with something along the lines of "well, if you were smart and actually knew what was going on, the solution would be obvious to you and you'd agree with me."
And, in my humble opinion, I believe that this elitism is more prevalent on the side which wants to adopt this change and make "Pending changes" permanent.
The problem for anyone not invested heavily in this discussion -- like me -- is that I don't see any proof that this will fix any of the problems it is supposed to fix. No one has provided the evidence showing not only that it will solve the problem of incorrect information being inserted into articles, but to what degree. (This feature can be subverted, some times rather simply. I'll leave that as an exercise for the reader.) And then there is the trade-off of discouraging new editors from further contributions: one of the enticements that is often mentioned by new users is the thrill of seeing one's edit accepted for all to see.
There is the advantage that some articles no longer need to be indefinitely semi-protected or fully protected -- but no one has provided evidence whether applying "Pending changes" to certain contentious articles will work better in protecting content than protecting those articles. However, instead of providing a comparison between these two approaches, those for adopting "Pending changes" simply assert that it is the best approach, and that it should be adopted immediately.
I am all for any change which improves Wikipedia's content, but Wikipedia has grown so large and so complex that I doubt anyone understands how it works comprehensively any more. I used to criticize Jimmy Wales because in many of his responses to its problems it was clear he no longer understood either the community or its dynamics; only those in constant interaction with the community, with editting articles and debating policy would have any chance of knowing. He had been out of touch with the day-to-day activities of Wikipedia for years now. More recently, despite the fact I participate in some way daily on Wikipedia, I've realized that I have grown out of touch. And it is clear that many other core members are too, for people will assume the concensus around a given article or discussion page is identical to its state when they last viewed it, when it may have changed radically in a few months or even weeks.
And here is yet another tension present in Wikipedia: between its constant changing and the need to manage that evolution. Its increasing rate of change, most of the time but not always for the better, has left practically all of its members behind and out of touch. There are no more "clued" Wikipedians -- assuming they existed in the first place. Some may see this as a good thing because it is this uncontrolled evolution which is responsible for Wikipedia's success; but if this true, then why does Wales, a philosophical libertarian, feel compelled to intervene when there is a problem? Wikipedia needs some management, and for management to be successful one needs information.
I like to think that the dynamics of Wikipedia can be controlled, through the same approach which existed in the beginning, through open discussion and building consensuses. It through this discussion and consensus-building that information is shared. The problem is that often, to arrive at a timely decision shortcuts needed to be made. Yes, one last tension that exists in Wikipedia! (I guess now that I've seen a few structural tensions The old shortcuts, however, no longer work; we need to either find and create new ones, or accept the need to laboriously build consensuses at every new change. Because the one thing that will kill Wikipedia is have a single person who makes all of the decisions; Wikipedia has flourished because of its "out of control" nature, not despite it.
Geoff
For those of you who aren't that familiar with the latest conflicts at Wikipedia -- which has now come to include a large share of active Wikipedians -- probably the most important is the deadlock over whether or not to continue a test of the "pending changes" feature.
The proposed feature is simple enough. For certain groups of editors -- those editting anonymously or with a new account -- their contributions are sent to a queue where they must first be approved before being made visible to one and all -- such as people who are simply reading Wikipedia, who make up over 90% of its users. Otherwise, the edit is deleted.
This is something that Jimmy Wales has long advocated for (see here for one example), and as a result many Wikipedians consider it a good thing. And maybe it is; I'm undecided only because after a two month test, no one has presented any factual evidence that it solved any problems. I haven't seen any in the lengthy, at times bitter, and for the moment deadlocked discussion over its adoption. But what I have seen are examples of long-existing tensions that exist over the ideals of Wikipedia, but which have not been adequately discussed. (For all of his supposed experience with Wikipedia, I haven't seen Jimmy Wales even acknowledge their existence.)
One of these tensions is presented quite clearly on Wikipedia's front page: "the encyclopedia anyone can edit". This is both the strength and the weakness of Wikipedia -- that anyone can, and does, edit its articles. The fact that something useful has resulted -- let alone a reference work which is as useful as it is -- has been pointed out so many times to become banal. When critics pointed out that allowing "anyone" to edit, this included vandals and malicious individuals, the response was to point out that bad edits were almost always promptly reverted. And one could say it was part of the price for creating an otherwise invaluable resource; after all, when dealing with experts and powerful people, one must put up with their eccentricities to benefit from them.
However, the steady trend since its inception has been to limit the meaning of "anyone" with more and more exceptions in order to improve its quality and usefulness. Some of these exceptions are quite reasonable: vandals, kooks and cranks, people who simply can't play nice with others. Others fall into a more difficult category, such as those who want to advocate for a specific political agenda or for their own economic gain. One can't do the simple thing and exclude them all because in many cases, they are the exact experts Wikipedia needs in order to be a useful reference. So Wikipedians are forced to develop essays such as Tendentious Editting or Civil POV pushing to define the problem and help each other confront it.
And then there are some policies which came about in a quick, knee-jerk reflex whose rationale have never been adequately explained, such as denying anonymous and new editors from creating articles. This came about because of the Seigenthaler incident, where an anonymous editor made a libelous allegation about John Seigenthaler which remained undetected for four months. This denial has remained mostly out of inertia: denying anonymous editors all editting rights is a perennial proposal which, although always rejected, has strong support. And if one creates an account then waits a few days, one will be able to create articles -- so no one sees this as anything more than an inconvenience for new editors. On the other hand, it is not an effective security fix: determined troublemakers need to simply create an account and wait for it to age before abusing it to wreak havoc. This change is only a speed bump, which may discourage more new editors than it justifies in dissuading vandals; no one really knows.
Now keep in mind that I'm not criticizing putting limits on just who edits; all of us want Wikipedia to be as accurate and useful as possible. My point is that there is a tension here, between allowing anyone to edit and creating reliable content, which fails to be considered in these decisions. And when I tried to participate in the discussion on whether to continue this test, I was unpleasantly surprised to find that no one had provided the needed information to show whether this test actually improved content or not. Instead a lot of anecdotes, first impressions and opinions were being tossed around. I don't know whether to be discouraged or just sad at this.
Before I continue, I'd like to also point out that I'm not saying that these tensions are not a weakness; they are simply a fact of Wikipedia's nature. Further, by identifying the existence of these tensions I am not consciously forcing a Hegelian dialectic upon Wikipedian dynamics; rather, I am picturing a statics diagram with opposing forces in an equilibrium. In other words, this is a situation where modifying one force in the tension without sufficient care or information can destroy the existing equilibrium and lead to everything falling apart.
A second tension illustrated in this debate is an old one: the clued versuses the clueless. This tension has existed either explicitly or implicitly since practically the beginning of Wikipedia. Larry Sanger's notorious emphasis on credentials was simply one extreme version of this tendency. Most Wikipedians, whether they are professional experts, serious amateurs, or just individuals with a desire for learning, tend to be elitists; we spent our spare time reading or tinkering in order to understand something better, while everyone else was watching television, playing games, or otherwise frivolously occupied. And we often did this both at our own monetary expense, and at the derision of our peers. So, yes, Wikipedians can be elitist in how they respond to new editors; it's an assumption experienced editors often find themselves fighting against.
In this conflict, over whether to turn this experiment into a full-scale adoption, I see this tension playing out on several levels. One level is the change itself: that new and unregistered editors are somehow less clued than the established ones. While a reasonable assumption, it inadvertently insults those new editors who are knowledgeable, forcing them to run a gauntlet to make even trivial changes of spelling or punctuation. But another, and more serious, level can be seen in how the two camps are discussing this proposal: many of the participants seem from their tone to be one step away from responding to their opposite number with something along the lines of "well, if you were smart and actually knew what was going on, the solution would be obvious to you and you'd agree with me."
And, in my humble opinion, I believe that this elitism is more prevalent on the side which wants to adopt this change and make "Pending changes" permanent.
The problem for anyone not invested heavily in this discussion -- like me -- is that I don't see any proof that this will fix any of the problems it is supposed to fix. No one has provided the evidence showing not only that it will solve the problem of incorrect information being inserted into articles, but to what degree. (This feature can be subverted, some times rather simply. I'll leave that as an exercise for the reader.) And then there is the trade-off of discouraging new editors from further contributions: one of the enticements that is often mentioned by new users is the thrill of seeing one's edit accepted for all to see.
There is the advantage that some articles no longer need to be indefinitely semi-protected or fully protected -- but no one has provided evidence whether applying "Pending changes" to certain contentious articles will work better in protecting content than protecting those articles. However, instead of providing a comparison between these two approaches, those for adopting "Pending changes" simply assert that it is the best approach, and that it should be adopted immediately.
I am all for any change which improves Wikipedia's content, but Wikipedia has grown so large and so complex that I doubt anyone understands how it works comprehensively any more. I used to criticize Jimmy Wales because in many of his responses to its problems it was clear he no longer understood either the community or its dynamics; only those in constant interaction with the community, with editting articles and debating policy would have any chance of knowing. He had been out of touch with the day-to-day activities of Wikipedia for years now. More recently, despite the fact I participate in some way daily on Wikipedia, I've realized that I have grown out of touch. And it is clear that many other core members are too, for people will assume the concensus around a given article or discussion page is identical to its state when they last viewed it, when it may have changed radically in a few months or even weeks.
And here is yet another tension present in Wikipedia: between its constant changing and the need to manage that evolution. Its increasing rate of change, most of the time but not always for the better, has left practically all of its members behind and out of touch. There are no more "clued" Wikipedians -- assuming they existed in the first place. Some may see this as a good thing because it is this uncontrolled evolution which is responsible for Wikipedia's success; but if this true, then why does Wales, a philosophical libertarian, feel compelled to intervene when there is a problem? Wikipedia needs some management, and for management to be successful one needs information.
I like to think that the dynamics of Wikipedia can be controlled, through the same approach which existed in the beginning, through open discussion and building consensuses. It through this discussion and consensus-building that information is shared. The problem is that often, to arrive at a timely decision shortcuts needed to be made. Yes, one last tension that exists in Wikipedia! (I guess now that I've seen a few structural tensions The old shortcuts, however, no longer work; we need to either find and create new ones, or accept the need to laboriously build consensuses at every new change. Because the one thing that will kill Wikipedia is have a single person who makes all of the decisions; Wikipedia has flourished because of its "out of control" nature, not despite it.
Geoff
Labels: wikipedia
