Forum:Peer Review

Forums → Hailing Frequencies → Peer Review (Reply | Watch)

Proposal

So, I've been working on the concept of Sourced Fan Fic, and I'll be ready with that in just a bit, but first I was also pondering a Peer review system. There is an awful lot of fan fiction here, and I, for one, would love to be pointed to the "good stuff", instead of having to wade through Ms. Captain Carter's First Sexual Voodoo Experience... So I've been pondering a review system (provided, of course, that this isn't a "Aabh, shut up" sort of thing :) ).

A little note: Things with a ">" are things I'm simply not sure about... Either these were problems I saw and couldn:t find a cute way to fix, or they are alternative rules that I wasn:t sure were good ideas or not...

Nomination, this is rather simple: To be reviewed, a series must be nominated for review.

>Should the nomination be seconded before it can be reviewed?

>Can an individual nominate their own works?

Once Nominated, a group of no less than 3 members of this community must read/watch/listen to and review it for content. Each member would then give it a number value from 0-10, 0 being absolute Voodoo, and 10 being Cannonical. During this time, the original article would bear the "This Article has been nominated and is currently being reviewed for content" label (Or something similar).

>Members who review the work must be registered users.

>Members who review should not be a close friend of the reviewee, though I:m sure in a pinch this rule could be lifted, we all sort of become friends over time...

After the nomination is approved The Sourced Article is then put on the main page as open for members to review under its own section with a plea for reviewers. Because of the nature of this, only Sourced Fan Fiction is allowed to be reviewed.

>I don't know what to do if a mess of Trolls decide that Starship Farragut is complete crap and all give it a unified "0" just to be mean... Should we have a review for reviewers? Should we give more weight to people in "good status"? These are things that keep me awake at night...

>What do we do if only 2 people are available to review it? Should it stay on the "To be reviewed" table until it gets the third? I don't see why not... I guess there wouldn't be a "Nominated Timeout" or anything like that... It is something to think about, though.

>I guess it might be a good idea to put some sort of time limit on reviewing... but this has its own sets of goods and bads: On the good, reviews would actually be completed... in theory... by the folks whom are notorious procrastinators and just need a little push, or they would be weeded out... on the bad, it might discourage folks from reviewing things as they don't think they have enough time to do it...

Once a review is given by the three or more members, the average score is compiled (Rounded to the nearest whole number) and the master article is awarded that number as an STEU Peer Review Rank. It is put in Catergory "Fan Fiction Reviewed by the STEU community" and the article will have its own little mark for being a Reviewed article, complete with its rank, good or bad. (This way we know what articles we have reviewed and the Voodoo articles don't keep coming back up to haunt us for review over and over).

>I think that if you have 5 or more reviewers, then the highest and the lowest scores should be tossed out and the remainder averaged. It's controversial when it happens in sports, but really, it does keep Joe Voodoo from logging in as Voodoo2 and reviewing his own piece and giving it a 10... it also curbs JoeTroll from panning everything he reviews that doesn't fit his exacting idea of FanFiction.

Reviews need not be more than a number from 1 to 10, but Super bad or super good reviews would be very suspect, and it might be better to write a brief "Why I liked/hated it" just to defend your point should the review become suspect... It would keep any subsequent reviewers from throwing out your opinion out of hand during a re-review.

Ranking and awarding If the article has a rank of greater than 5, it appears in the Catergory "Ranked as among the best of STEX Fan Fiction", if it is a 8 or 9, it also appears in the Catergory "Must Read", things ranked as a "10" are considered, "The Best Fan Fiction has to offer" and, I might add, that I will faint dead-away if anything ever achieves a perfect 10... That is almost as suspicious as a Complete "0"...

>Note: This is a very slippery slope. We have to be prepared for people to completely pan Tamerlane or Pendragon or even New Voyages... And we'd have to be willing to put our pride aside and stand behind that review... though I have to admit, if any of the biggies here popped up as a "0" or a "2", I'd call for a re-review and review it myself.

>Here's yet another problem I just thought of; Let's say I read Pendragon and I think it is absolute Voodoo, so I give it a "2"... then Tim comes along and sees that I gave it a "2"... what does he then think of me? Would I, as a user, be a little worried about giving his beloved child a "2" (Tim, thank you for being an unwilling guinea pig for my example :D)? Perhaps I'll give every series a "6" all thie time because I don't want to make anyone mad... it's like a "Reverse Troll"...

Re-review: It's bound to happen, the poor series was reviewed by Troll1701, IHateEverything and FanFicSucks23. Dispite its eloquent writing, and extremely well defined characters, the three reviewers agreed that it is a "2" (And that's only because Troll1701 felt pity for the series and gave it a "4")... The series can then be called up for a re-review, but this would only be something the Admins and Moderators can do (Otherwise, Voodoo would pop up all the time for re-review and it would drive us crazy constantly re-reviewing Voodoo).

Lastly: Guidelines. On the "I want to review something" page, there is a series of helpful guidelines that could help the reviewer think about critical functions of the review. For example: Are the characters original and interesting? Is the story new and different? Does the story fit, in your opinion, the vision of Gene Roddenbery? If not, does it take Star Trek in a new and different direction that is engaging? Of course, these are simply sugestions, but it might help the reviewers look beyond "Oh my GOD I hate the way they portray Shelby!!!!! Her hair was MUCH blonder than that!!! This show is a "0" definitely!"

So this is simply a suggestion, of course, But I figured I'd start with a framework that we can all work from, tear apart, rebuild, throw out completely, etc... but at least we will be chatting about this topic! I think it's a great idea, and I support the concept! :D Thanks Aabh 03:14, 21 June 2007 (UTC)

Reactions

I don't mean to sound mean but... it sounds like a lot of work to me, and very subjective work at that as well. If you are looking at doing this you need a) an assessment sheet with criteria b) a sheet with what constitutes evidence c) judgement statements, i.e. what constitutes a 4, 5, 6, 7 etc., and how marks are allocated d) a team of assessors (the 3 people you point out) who are qualified enough in assessing something in order to make the correct judgement/assessment and e) a team of moderators to ensure that the assessments are fair, valid, consistent and accurate. That's probably just the tip of the iceberg. What about policy and procedure? Quality control? How can you make sure the people who are doing these things have adequate knowledge and experience in doing it? I do this sort of stuff at work all the time, and, to be honest, I get enough of it there; I personally don't need it here. I do appreciate the work you have done on this, Aabh, and honestly, you must've been working on it a bit to come up with so much, but I wouldn't be a part of it. I can see disaster all over it a mile away, because some people will cry foul and it will probably tear up the fabric of this wikia. Being a gay guy who works with some gay guys and also has gay friends, as well as 70-odd female students who, by the end of the year, get really bitchy at the same time each month, I get enough drama in my life LOL --usscantabrian 04:14, 21 June 2007 (UTC)

Actually, that doesn:t sound mean at all. It sounds very valid. I:m not posting THE Answer, I:m posting AN answer. We still have a problem with "What is good FanFic and what is utter crap". Or maybe we don't... I don:t know. But it seems like there has been discussion in the past on this one... And I was just pondering it as I pondered the other things... So... (With the honest-non-challenge-voice) What is your suggestion? :)Aabh 05:39, 21 June 2007 (UTC)

It does sound involved but it sounds like a solid idea to me. The only thing I would suggest is the "review committee" be either active members or only people have earned Member of the month status. Just my two bars of latiumn before I run out the door to work. -- Sneg ^Admin•Talk 11:17, 21 June 2007 (UTC)

Ah, good idea! How do you decide Active Membership here? Aabh 03:13, 22 June 2007 (UTC)

usscantabrian's idea

My idea? Make it somewhat simpler, like for Featured Article.

1. Nominate the series. Have someone nominate a series, most preferably not the person who writes it. When the series is nominated, an announcement goes up in a forum here, much like Featured Article, et cetera.

2. Ask members to answer a series of questions. These would be stock standard, yes/no (non-leading) questions with a why part for the voter to explain his or her rationale. Can this be done in secrecy on this wikia to ensure no hard feelings? If people abstain from some questions (unless those questions are not applicable, and this would need to be defined) then the vote doesn't count either way. The final tally would be a simple "Yes/No": Is this series good enough to get this mark?

3. When 5 members or more vote "Yes", and with good reason, the series gets the mark. 5 is a number I got from the Featured Article and other areas. It could be more or less, but I think 5 is a good mark.

What is the "mark"? Like Featured Articles, perhaps the series page (and even main pages) get an icon. This icon would be placed initially on the series page and those who gain approval could put it on as many pages of theirs they wish (as long as not shared amongst other series). This should simply say, "STEU users have reviewed this series entries and found it is a good example of good Star Trek fan fiction" (or similar).

I think that this really simplifies the process. Getting "grades" or "marks" can disadvantage and will make the process confusing and cumbersome. Also, this is an opt-in feature; with "having to have" 3 people mark could be done grudgingly, or, in cases like me where I have some parts of the year where I have virtually no time to do personal things or when I am away, could be a disadvantage. Having people do it on their own time and under their own steam means they don't feel pressured into having to do it. And... that's my idea to this (hopefully a lot simpler too!) --usscantabrian 22:38, 22 June 2007 (UTC)

More reactions

I like this idea too! :D I only have three problems with it (And they might just be things that I'm overlooking!): Though it does do a good job of telling us what is good and what is not, I fear it might be a little too simple... It could be really easy for me to pop in and give all of my stuff a "yes" and greenlight the Voodoo... then again, using Sneg's idea and restricting it to active members might solve that rather well (Which removes this problem :D).

The second problem I have is; how do we distinguish the really, really good stuff from the good stuff? I mean, say you don't have all the time in the world to read all 5,330 series' that we have all decided were good enough to get the mark... How do you get funnelled into the top 100? Or top 10? I totally hear you on the point of grades being cumbersome, and I'm thinking on a way that is a compromise between the two... I just have no ideas at the moment... any thoughts? Wait... I just got a thought (And yes, I jumped back up here to add this afterward :D); what if we gave it a percentage of people who said "yes" out of a percentage of people who said "no"? For example; 3 out of 5 people said yes, so it gets 3 out of 5 stars, or 60%, or 6 out of 10 Com badges, or whatever we want to do with it... This could be a mathematical formula programmed in somewhere on the Wiki, so all you do it type in the number of respondants, then type in the number of "yes"s... and it gives you the resulting "Com badge" number... or something like that (I picked Com Badges because... really... we shouldn't use that :D). Because it doesn't say anything about who said what about the series, it prevents hurt feelings (It only says the rating was 4 out of 7 or whatever), but still allows a ranking system.

The last thought on this is Re-review... Would we use my re-review concept (retesting with only the admins/Moderators)? Or did your design have a different re-review function? Or no re-review?

Anyway, the thing I like most about your idea is that you could (Theoretically, and if we so desired) have "stats" of a way that some people could look into, for example, 3 out of 5 people thought Farragut (Just picking on it as an example) had a good plotline, but 5 out of 5 thought it had good characterization... So in it's review "Jacket" one could see that it had a 3 out of 5, a 5 out of 5, a 4 out of 5... etc As a for instance. And since people all have different things they look for in Star Trek (Some people think Star Trek isn't good without an epic space battle, some think that's not in the vein of Star Trek, etc), you could search for series' that got a 5 out of 5 in not having (Or having) Space battles... or whatever.Aabh 23:56, 22 June 2007 (UTC)Aabh

The details really need to be worked out, so I just was providing what I feel is a simpler approach.

Make four categories maybe. I don't know how we'd segregate them (maybe, as you said before, a percentage, i.e. 4 out of 5 voted yes in this area where that gives us 80%) or what the criteria would be, but that would be something the community would have to decide. (This is how our international governing bodies grade our students, so this is just an example and not set in concrete.)

Distinction, or Excellent Series: 90% or above, or if we rate it differently, maybe 5 out of 5 people say it's great.

Credit, or Above Average Series: 75% to 90%, or if we rate it differently, maybe 4 out of 5 people.

Pass, or Average Series: 60% to 75%, or if we rate it differently, maybe 3 out of 5 people.

Fail, or Below Average Series: To avoid embarrassment for the author involved, probably would be best not to show this on the page.

Again, I think use the KISS principle: "Keep It Simple, Stupid" (LOL). I don't know if the segregation into plot, characterisation, etc. would be good, although I guess if that's what we asked in the survey, then we'd have that information. I do think, however, using the "3 out of 5 people said" is very simplistic in this area. Perhaps if the author wanted further information and the people who rated it were willing to go further, we could have a separate survey on that where people could rate on a scale.

The real danger (and I keep on saying this) is that characterisation, et al, are really subjective, not objective. This also happens because tastes are very personal. Some people loved Voyager; some people hated it. Some people thought it was mediocre. And I don't think, in all honesty, series on STEU could be rated in these areas due to this.

There could be a re-review (I haven't even thought about it... painting in broad strokes here), and probably this could be visited a year later.

Reviews will most likely also have to be moderated... and that's a bit of work. Moderation ensures consistency, accuracy, fairness and validity. And that would have to be entrusted to trained, level-minded, objective people with a set of guidelines or performance criteria.

So... that's my 2 cents worth and more LOL --usscantabrian 03:54, 23 June 2007 (UTC)

Reactions continued

To chip in w/my own two cents: Aabh deserves an A for effort, but indeed it does sound complicated, and it makes me uneasy. Somewhat like stated above: You can see trouble all over it from a mile away. I agree with the KISS rule, and using just plain common sense: What is junk compared to not-junk should be fairly obvious. Things have been working on Wikias as they've been working for some time now, and like the saying goes, if it isn't broke, don't fix it. There are policies and/or procedures to weed out the crap. So I don't see the need for this system, to simply be honest. – Sasoriza ^{Adm • Tlk} 08:04, 23 June 2007 (UTC)

I agree that this may not be the answer, but I thought the point about this type of policy was to prevent the crap from going up in the first place? So we can have an objective standard to point to, or an "official" means to determine what is good enough and what isn't, instead of simply saying "Sorry, but Captain Sexy and the Space Cadets doesn't cut it." Something like that may be obvious, but it's also obvious that the Homesun's and Zman's of the world will keep coming. And what if we get the FanFiction.net fan girls with their Mary Sue romances and so forth? Something needs to be in place to prevent that from happening. --TimPendragon 19:12, 23 June 2007 (UTC)

Hey Tim and everyone... what Aabh is talking about here is a peer review system so readers can know what the people who contribute to this wikia believe is a good series or not (to read, etc.). The other policy and procedure Aabh was developing is more what you are talking about: sourced fan fiction.

The more I think about it, the more I think that something like a peer review system would be better left to places like Trek Writer's Guild or similar, where the focus is on writing and stories, not encyclopedic entries.

Wouldn't our time be better spent on weeding out bad articles or helping improve them?

As Sasoriza said, there are policies and procedures to weed out the garbage (although I have to admit, I don't feel the stub policy is working 100% with one of my articles getting a stub when, in fact, it has no more information that can be added to it... and still waiting for a reply on that one!!!)

So maybe we should focus on getting the policies and procedures here working right and tweaked to our satisfaction instead? (I'm not bitching as much as I would like to help as best as I can!) --usscantabrian 22:18, 23 June 2007 (UTC)

Erm... What he said :D - Actually, this is indeed a Peer Review System. I'm working on the Sourced Article function seperately. I thought this one would be more controversial (And, I believe I was right on that one), so I went on ahead and posted it before I started making graphics and all that would be changed as soon as folks shot down ideas and all :D

And actually, I prefer usscantabrian's idea over mine, it is simpler, more elegant, and all around easier to use... I don't know if that helps any, but I'd put my vote in behind his over mine anyday. :D

I like the grading concept you have as well, uss... er... can I call you uss? Anyway, I like the grading system you have there, it's good as well! Aabh 14:14, 24 June 2007 (UTC)

When I originally brought up a "peer review system," I meant more along the lines of a mechanism whereby inappropriate/substandard material is brought before a "review board" - a sort of "jury of worthiness" for the unsourced/"wiki original" material. Things like what Zman and Homesun put up here would not pass; heck, probably some of my Pendragon articles wouldn't pass muster. Something objective, instead of us seeming to randomly say to people "this isn't good enough." That's what I meant by "peer review." --TimPendragon 21:28, 25 June 2007 (UTC)

It's not a bad idea, but seems tricky to execute in concept (as detailed). A) It's subjective, B) It could be time-consuming. On point A, Homesun/Zman-type stuff would leap out – granted; but for material a cut above that, sufficiently better in quality, it becomes a matter of opinion (even if it's a majority opinion, it's still an opinion, and opinions are subject to change over time), and you know what they say about opinions... plus what it can lead to: A "clique" or "private club" affair where total objectivity falls to secondary consideration. But then I believe that's already been said.

On the other hand, it may be the only best possible solution, and some material might merit the attention of such a system. But there's still that matter of opinion. (Again) like someone said, if a bunch of ppl simply didn't like something, they could shoot it down, for that reason. Take for example "Mary Sue" fiction, or "Captain Sexy and the Lust Cadets" or whatever you want to call it--technically, how can anyone draw the line and say it's not acceptable? How would that be fair? Such stuff could be meant in a humorous vein, like Hawku's stuff, or not meant to be taken 100% seriously. I'm not advocating anything, just saying. It's a very fine line. Like I said, tricky.

Plus, I don't think 5 ppl or whatever is good enough to constitute a majority. So, rhetorically speaking, how many ppl does it take to make it "count"?

I guess I'm looking for a foolproof factor, with checks and balances incorporated to keep it from being manipulated. If I missed something to this effect, well, that's possible. – Sasoriza ^{Adm • Tlk} 01:48, 26 June 2007 (UTC)

I hear what you are saying, Tim... I wish I had a brilliant brainstorm to solve that problem... but really, I can't think of anything that can fit the bill easily. Saying "your work sucks" causes a great deal of negative emotion. Even this version of peer review has enough negativeness tied to it to cause folks to see this place as slightly cliqu-y.... If we Actively went hunting for stuff to call crap and bump it, that could be construed as downright unfriendly... Maybe the best way to solve this situation (And... boy wouldn't this be interesting) would be to allow, and even Encourage folks to nominate their own piece for independant peer review. You can't really say a site is looking for trouble if you asked for it in the first place.

And Sas... (Er... can I call you Sas? I can't see your whole name in edit mode...And I'd really feel foolish for spelling it wrong), Herein lies the crux of the problem, if you will... Checks and Balances make it complicated. Complicated (As uss has already pointed out) means people won't do it... Simple means everyone can do it, but it also means you lose the ability to have completely unbiased peer review.

Now, (And please forgive me for being long winded on this one), my thought isn't that peer review is mandatory, nor is it a thing where you thumb your nose at people when you have it, nor did I ever think it would be a "Anything that has been peer reviewed is good, all else is crap" sort of thing, indeed, I figure some of the best of Fan Fiction may never be Reviewed (We may forget some of the wonderful stuff that came many years ago and the creator no longer produces... I'm sure somewhere out there is a Radiskull and DevilDoll type Star Trek that is ancient, but lovely) I simply thought it might be a useful tool for people who walk in here to see what we thought was pretty good. There will always be stuff that never gets peer reviewed, but at least you know the ones reviewed are among the best things to read/watch.

I don't know if there is a way to weed out the junk. We can peer review the good stuff (And, by nature, that'll weed out some of it) and we can award articles that have a source (And that should weed some of it out as well)... but I think actively looking for bad stuff and kicking it out is a bad idea... But there might be a compromise... let's percolate on it a bit and see what we can come up with! :) Aabh 05:56, 26 June 2007 (UTC)

Sas is fine. Or you can call me Todd. Yeah, it's still not a bad idea, as long as someone has time for it. – Sasoriza ^{Adm • Tlk} 12:53, 4 July 2007 (UTC)