Streamlining & Improving review process

[sasquatch]

I wanted to carry some comments I had in another thread over to a different discussion devoted to the review process at TR:

The problems I see currently are:
-interior reviews are one of the most major bottlenecks
-lack of early input leads to greater problems after work has been completed
-even worse, lack of input and large-scale planning at the beginning or end of production leads to remaking a large amount of work
-lack of peer input prevents flow of good ideas
-locked claims in review prevent peer discussion
-locked claims in review prevent last minute changes
+this slows work and prevents a natural process of feedback and revision (claims that enter review should stay there for continued minor revision and feedback from all TR members)
-review is too narrow in scope
+ often does not critique other important aspects like suspense, sense of adventure, uniqueness or whether a claim has allowed for creative movement through the space

________________________________________________________

there are other points that I will add as they come to mind
my previous comments in regards to interior reviews below:

Finishing the reviews seems like the most tedious, yet easiest way to make major progress on the mod. Completing the reviews would also make me much happier since none of the dozen or so interiors I've done over the past couple months have had a final review. I'd like to see all this stuff in-game within the year.

I also want to add to this that a fast, yet quality review process will give greater incentive for people to claim and complete quality interiors. At our current rate of progress, people, realizing their interiors wont be reviewed for 6 months or more and that their interiors are unlikely to be added within the next year or two years, are not likely to be as productive.

I'd like to assume we all care about the mod and want to see progress made. Moving items through the review process is the only way that will happen. It also seems to be the most major bottleneck besides a lack of new models and textures, but reviewing doesn't require more assets or people. Reviews only require time and dedication to the mod and the people producing content. Improving this process will go a long way in improving overall workflow and production.

The last most important question is if anyone in leadership is willing and capable of dedicating the amount of time required to properly use the resources and people we have here. If not an alternate method of review and implementation needs to be found, one based on cooperative review and feedback of multiple TR members.

[Thrignar Fraxix]

-interior reviews are one of the most major bottlenecks

Not really but I'll touch more on this later

-lack of early input leads to greater problems after work has been completed

I can't think of specific examples of this, at least not on as dire of a level as you seem to make it out to be. Questions in threads sometimes go unanswered, other times they are answered in IRC or via PM.

-even worse, lack of input and large-scale planning at the beginning or end of production leads to remaking a large amount of work

For the seven thousandth time, this is a known issue that we are working to improve on with the restructuring. Just because you are late to the party doesn't mean it hasn't been going on for a long time. Again though, this is something I'll touch more on later.

-lack of peer input prevents flow of good ideas

I must have missed this lack of peer input while I was busy reading all of the now publicly visible planning and discussion threads you yourself have even posted in. The lack of peer input does not exist nearly as much as you think it does, and the one most guilty of shutting down recent discussions is you. If you think people want/need unsolicited opinions in their int claims, then by all means give them.

-locked claims in review prevent peer discussion

The reviewing forum is locked for normal members for several reasons. First is that I always like to review from oldest to newest, and if you bump the threads you've just moved your int to the back of the line. Second is that the reviewers are capable enough of doing their jobs just fine. I've seen you posting errors in approved ints, that is all well and good and is actually helpful (when they are actually errors), but I will tell you again what I already told you, if you have such a problem with the speed of interior reviewing, maybe you should take the test. As far as peer discussion is concerned, if people cared enough to look at ints in reviewing they would become reviewers, otherwise I think everyone has plenty of other things to do.

-locked claims in review prevent peer discussion

Last I checked the PM function works just fine.

+this slows work and prevents a natural process of feedback and revision (claims that enter review should stay there for continued minor revision and feedback from all TR members)

If more people want to become reviewers, then they will become reviewers. What you seem to be proposing would actually slow the review process's potential speed down DRASTICALLY. You may not have been around during the days when I actually had the time and the mental fortitude to review/approve 60 ints in a day, but reviewing goes as fast as the reviewers make it go.

-review is too narrow in scope
+ often does not critique other important aspects like suspense, sense of adventure, uniqueness or whether a claim has allowed for creative movement through the space

Because a common house needs suspense and a sense of adventure. I will never forget the first time I first went into Dralosa Athren's house in Balmora, the feels I felt stick with me to this day. Uniqueness beyond avoiding cookie cutting is overrated in most ints. Yes, important ints need to be unique and by and large we make them as such, but if we devoted enough creative energy into making every single random shack memorable, we'd have gone mad years ago. Int makers are welcome to add the touches as they see fit, and it is encouraged, hell I've done fancy things in non-fancy interiors myself fairly regularly. I'm not going to hold it against someone though if their random shack in the middle of nowhere that is probably going to be locked due to the owner standing outside, and therefore seen by less than 1% of the player base lacks an artistic vision that makes me question my perception of reality.

As for creative movement through the space, I don't even know what you are talking about there. We do make sure ints can be maneuvered through.

Finishing the reviews seems like the most tedious, yet easiest way to make major progress on the mod. Completing the reviews would also make me much happier since none of the dozen or so interiors I've done over the past couple months have had a final review. I'd like to see all this stuff in-game within the year.

I think your understanding of the flow of progress is not as good as you think it is. Yes it is tedious, but you seem to think the entire future of the mod hangs on the speed of int reviewing, when really it doesn't. It is a piece of the puzzle that is far smaller than you think it is.

I also want to add to this that a fast, yet quality review process will give greater incentive for people to claim and complete quality interiors. At our current rate of progress, people, realizing their interiors wont be reviewed for 6 months or more and that their interiors are unlikely to be added within the next year or two years, are not likely to be as productive.

I think you are speaking for yourself and acting like that is the opinion of everyone. The Map 6 ints we put up years ago went like hotcakes, even though they were so far down the line anyone who knew anything knew they wouldn't be released for years.

I'd like to assume we all care about the mod and want to see progress made. Moving items through the review process is the only way that will happen. It also seems to be the most major bottleneck besides a lack of new models and textures, but reviewing doesn't require more assets or people. Reviews only require time and dedication to the mod and the people producing content. Improving this process will go a long way in improving overall workflow and production.

You don't even know what our bottlenecks are. You think you do, but int reviewing and models and textures are not our current bottlenecks. Planning is our bottleneck, the planning you seem to think isn't even going on. It is, but we are trying to make so many plans that we are getting bogged down. We want to have enough of a basis for progress so we stop making the mistakes we've made so much in the past that you yourself continually harp on us about.

The last most important question is if anyone in leadership is willing and capable of dedicating the amount of time required to properly use the resources and people we have here. If not an alternate method of review and implementation needs to be found, one based on cooperative review and feedback of multiple TR members.

You want to help reviewing, I'll say it again, take the test. The final reviewer position is not something I am going to toss around lightly. It took a great deal of change in my thought to even promote Andres to the position, and he is the best reviewer I've ever known to be at TR.

[Gnomey]

Having been a member of TR for a long time, from my observation the rhythm of reviews is simply different. There might be long periods with no or very few ints getting approved, and then without warning there will be a flood of dozens of ints. I'm not sure whether such a flood has happened since you joined, (there appears to have been a smallish one in February), but it'll happen sooner or later. I have seen the Reviewing section cleared out plenty of times.

I do think new members tend to be somewhat more anxious about seeing their ints finished, so that they know what they are doing wrong and what they are doing right, and it might be rather nice if their ints were given somewhat of a priority. (Reviewers might already take that into account). I think most 'veteran' members are used to the flow and don't mind it, though.

By the look of things, reviewers prefer reviewing interiors in bursts, and I'd imagine I would do the same if I were to become a reviewer. Unless they're paid, I don't think reviewers can be expected to be more regular in their reviews; the same goes for all TR modders. A modder's involvement in the project is based mostly on inclination, and if a reviewer forces himself to drudge through dozens of reviews he'll only tire himself out and may end up taking a break from the project or reviewing, in an extreme case.

[Terrifying Daedric Foe]

My personal feeling is that reviewing happens more quickly now than when I first joined TR. There definitely feels like there are more active reviewers.

Sure, the wait for the final review does sometimes feel like it takes a while. Then, when Thrig does a load in one go his messages usually just say 'approved', which makes it look a bit like he got sick of all the interiors piling up and just bunged them into finished without looking at them. However, that view is incorrect for two reasons. Firstly is that there's usually few errors missed by the first review for him to correct on the second review. Secondly is that all messages posted whilst the file is in review become invisible to non-reviewers when the file is taken out of review. That does cause confusion sometimes and should be altered to reflect the new transparent TR. But it isn't a fundamental problem with the review process, which has been copied by other province mods and works perfectly fine.

[sasquatch]

I first want to say I appreciate your thorough response TF and all the other comments made here thus far. The following will be a response to all points raised by TF, hopefully with additional productive comments to follow:
-------------------------------------------------------

sasquatch wrote:
-lack of early input leads to greater problems after work has been completed

I can't think of specific examples of this, at least not on as dire of a level as you seem to make it out to be. Questions in threads sometimes go unanswered, other times they are answered in IRC or via PM.

I cant really comment on what went into redoing dozens of interiors 2 or more times as I wasn't here at the time, but I think almost any of these would be good examples. My second interior instantly comes to mind though. It had been completed 2 or 3 times before me. I was approved for a 3x3 layout with a fireplace. Midway through I found out it was the wrong dimensions. I then asked several times for the fireplace to be approved, even adding the request into the text for the file so it would be viewed when opened. This was a highly detailed interior with nearly every object painstaking leveled to the .01 decimal on de_p. This thread and file no longer exist on the forum. I don't know how many more instances of this there have been but I can imagine there have been many.
-----------------------------------------------------------

sasquatch wrote:

-even worse, lack of input and large-scale planning at the beginning or end of production leads to remaking a large amount of work

For the seven thousandth time, this is a known issue that we are working to improve on with the restructuring. Just because you are late to the party doesn't mean it hasn't been going on for a long time. Again though, this is something I'll touch more on later.

I'd like to know what the restructuring means so I'm looking forward to learning more about this. I think the input of everyone should be included to make any changes. I have noticed problems in general planning but mostly the interiors reviewing process, because I have only done interiors. For this reason I am bound to have good insight on how to improve workflow in that specific area. I'm sure developers from other departments have more ideas specific to their area of expertise that I cant mention because I haven't experienced them.
--------------------------------------------------------------

sasquatch wrote:
-lack of peer input prevents flow of good ideas

I must have missed this lack of peer input while I was busy reading all of the now publicly visible planning and discussion threads you yourself have even posted in. The lack of peer input does not exist nearly as much as you think it does, and the one most guilty of shutting down recent discussions is you. If you think people want/need unsolicited opinions in their int claims, then by all means give them.

First, while topics may overlap somewhat, this discussion is on the "review" process specifically, not general planning.
Second, I'm not going to continue to insult specific people by bringing up past threads. I will say that if all planning was as thorough as this or held to the same scrutiny as interiors MANY problems that I am in no way responsible for could have been easily avoided. The fact is that ideas, my own and others, were being shut down/ignored in previous threads and discussion was being diverted from the topic at hand. When I called person(s) out on this those threads were shutdown each time (but this is a diversion and almost entirely unrelated).
------------------------------------------------

sasquatch wrote:
-locked claims in review prevent peer discussion

The reviewing forum is locked for normal members for several reasons. First is that I always like to review from oldest to newest, and if you bump the threads you've just moved your int to the back of the line. Second is that the reviewers are capable enough of doing their jobs just fine. I've seen you posting errors in approved ints, that is all well and good and is actually helpful (when they are actually errors), but I will tell you again what I already told you, if you have such a problem with the speed of interior reviewing, maybe you should take the test. As far as peer discussion is concerned, if people cared enough to look at ints in reviewing they would become reviewers, otherwise I think everyone has plenty of other things to do.

First, the problem you mentioned could be resolved by dividing exterior/interior reviews by region. Bumped claims would remain roughly sorted. I can't understand why some interiors should take priority over others if they would remain grouped with claims from the same area.
Second, good point, however I think many people have, or would have, good points of critique other than bleeding/caspering and floating objects.
---------------------------------------------

sasquatch wrote:
-locked claims in review prevent peer discussion

Last I checked the PM function works just fine.

See my above responses.
----------------------------------------------

sasquatch wrote:
+this slows work and prevents a natural process of feedback and revision (claims that enter review should stay there for continued minor revision and feedback from all TR members)

If more people want to become reviewers, then they will become reviewers. What you seem to be proposing would actually slow the review process's potential speed down DRASTICALLY. You may not have been around during the days when I actually had the time and the mental fortitude to review/approve 60 ints in a day, but reviewing goes as fast as the reviewers make it go.

Not everyone has a desire to go check every object for floaters and bleeders. I understand how this could be a drag and so I respect those who are willing to do this. Next, I don't see how any review can reasonably take longer than one month. Our current problem is that they just are not being done. This not an issue that a half dozen reviews actually takes six months to complete.

This brings me to my next point on this. Sixty interiors should not even be available for review in one day. Interiors in review should not go above ten. I don't care if reviews have been irresponsibly handled for the last 10 years or how long absurd numbers of them have been stacking up. I care about what is happening right now and in the near future. This is why I believe it is so important that you help complete a thorough reorganization with all the insight you have gained over your many years here. I'm sure you have many more ideas than I on how to improve things here. Right now our process bottlenecks down to a single person in places. For a project with such a long history and such a massive amount of content I think that is a poor idea in the long-term. Right now nothing is moving in reviews and while they sit, no last minute changes can take place and no discussion can be viewed.

If things are not being reviewed as in-depth as the should be or with proper frequency, mistakes cant be corrected and improvements cant be made on future claims either. This creates further problems for future reviews and at the very least prevents easy improvements from being made. By the nature of this project there is no guarantee the claim holder will even be active in 6 months to a years time. For this reason claims need to be handled reasonably quickly and thoroughly.

Currently there is an average of two or three interiors being completed a week. Once the 90+ claims interiors in review are cleared, reviewing claims at this pace should be manageable. Maybe relative priority should not even be a concern. All claims waiting on review should be high priority and if completing claims isn't a priority then they shouldn't be posted. Doing so is irresponsible.
----------------------------------------------------------

sasquatch wrote:
-review is too narrow in scope
+ often does not critique other important aspects like suspense, sense of adventure, uniqueness or whether a claim has allowed for creative movement through the space

Because a common house needs suspense and a sense of adventure. I will never forget the first time I first went into Dralosa Athren's house in Balmora, the feels I felt stick with me to this day. Uniqueness beyond avoiding cookie cutting is overrated in most ints. Yes, important ints need to be unique and by and large we make them as such, but if we devoted enough creative energy into making every single random shack memorable, we'd have gone mad years ago. Int makers are welcome to add the touches as they see fit, and it is encouraged, hell I've done fancy things in non-fancy interiors myself fairly regularly. I'm not going to hold it against someone though if their random shack in the middle of nowhere that is probably going to be locked due to the owner standing outside, and therefore seen by less than 1% of the player base lacks an artistic vision that makes me question my perception of reality.

As for creative movement through the space, I don't even know what you are talking about there. We do make sure ints can be maneuvered through.

You are assuming something I had not intended here. I am not suggesting indiscriminately requiring elements of suspense, adventure and lore in every outhouse, tool-shed, and shack. These things NEED to be addressed where appropriate and, as far as I have seen, they are not.

Also, I have mainly focused on interiors because this is the area I have been active in. I apparently wasn't clear enough that this discussion was intended to cover interiors/exteriors and any other review process the topics here could also apply to.

I am using the term "creative movement" here to encompass the paths the player has available to them. In our creative process we should allow for the player to use their own creativity in the gamespace. Incorporating "creative movement" into the planning and review process would attempt maximize the enjoyment and freedom the player has as they move through the game. Since so much of Morrowind depends on moving it is an aspect of interior/exterior design that should be more formally recognized.
-------------------------------------------------------

sasquatch wrote:
Finishing the reviews seems like the most tedious, yet easiest way to make major progress on the mod. Completing the reviews would also make me much happier since none of the dozen or so interiors I've done over the past couple months have had a final review. I'd like to see all this stuff in-game within the year.


I think your understanding of the flow of progress is not as good as you think it is. Yes it is tedious, but you seem to think the entire future of the mod hangs on the speed of int reviewing, when really it doesn't. It is a piece of the puzzle that is far smaller than you think it is.

You are taking my comments/critique out of context. I am not claiming the entire fate of the mod rests on an adequate review process, but is one of the most glaring, complete failures in our work process. It needs to be addressed. That is the purpose of this discussion thread.
----------------------------------------------------------

sasquatch wrote:
I also want to add to this that a fast, yet quality review process will give greater incentive for people to claim and complete quality interiors. At our current rate of progress, people, realizing their interiors wont be reviewed for 6 months or more and that their interiors are unlikely to be added within the next year or two years, are not likely to be as productive.

I think you are speaking for yourself and acting like that is the opinion of everyone. The Map 6 ints we put up years ago went like hotcakes, even though they were so far down the line anyone who knew anything knew they wouldn't be released for years.

Developers being willing to claim interiors does not absolve our faulty review system of error. I really enjoy interiors and will gladly do more, but I do not think it is right how many aspects of review and planning are being handled at the moment. The willingness of people to contribute to the mod knowing full well how far off official releases are shows the amount of genuine interest people here have in continuing work on the mod and seeing progress made. Much of this is due to the fact that TR is the largest MW mod project, but this in no way changes the fact that there are MAJOR problems with planning, waste, and review here.
--------------------------------------------------

sasquatch wrote:
I'd like to assume we all care about the mod and want to see progress made. Moving items through the review process is the only way that will happen. It also seems to be the most major bottleneck besides a lack of new models and textures, but reviewing doesn't require more assets or people. Reviews only require time and dedication to the mod and the people producing content. Improving this process will go a long way in improving overall workflow and production.


You don't even know what our bottlenecks are. You think you do, but int reviewing and models and textures are not our current bottlenecks. Planning is our bottleneck, the planning you seem to think isn't even going on. It is, but we are trying to make so many plans that we are getting bogged down. We want to have enough of a basis for progress so we stop making the mistakes we've made so much in the past that you yourself continually harp on us about.

You've raised a great point here. We need an updated agenda, every month, that includes a reasonable timeline of expectations and goal-setting a list of priorities including high priority tasks and subjects for discussion. It should also include proposals that have been shelved for progress at a later date. I think this point alone should have its own thread at a later date (once you have reorganization master plan that has long been in the works has been revealed.) Hopefully there will be room for revision at that time so we can work together to get TR in top working order.
-------------------------------------------------------

sasquatch wrote:
The last most important question is if anyone in leadership is willing and capable of dedicating the amount of time required to properly use the resources and people we have here. If not an alternate method of review and implementation needs to be found, one based on cooperative review and feedback of multiple TR members.

You want to help reviewing, I'll say it again, take the test. The final reviewer position is not something I am going to toss around lightly. It took a great deal of change in my thought to even promote Andres to the position, and he is the best reviewer I've ever known to be at TR.

None of this immediate discussion is meant to be an insult to Andres or his capabilities as a Reviewer. I consider everyone here past and present a valuable asset to the mod. HOWEVER, I think it is reasonable to suggest that there should be more than one final reviewer. I assume everyone here has other obligations and interests. A reviewer may be perfectly capable, but if consistent progress is not being made at pace with completion of claims its a null point. Spreading responsibility would ensure continued steady progress.

IMO no reviewer should even be promoted if they are not held to the same standard. Completing a final review should be an option available to all reviewers. After a preliminary review process of critique by peers and technical review by an official reviewer, a second review should be a sufficient final check. Any two qualified reviewers completing reviews should catch any problems. If they don't, the issue is likely inconsequential or will be caught in alpha as so many others have.
------------------------------------------------------------

On a slightly different but not unrelated note:
The work that has been done, largely by Sload, on the Masterplan thus far is great. The "Everything, Everywhere" work is excellent.

Masterplan
http://tamriel-rebuilt.org/old_forum/viewforum.php?f=226
"Everything, Everywhere"
https://docs.google.com/spreadsheet/ccc ... _web#gid=0

These are the beginning of a framework of information and guidelines that, once completed and if properly followed and maintained, will ensure that future work can continue with a level of quality and consistency worthy of the work done thus far with or without the active valuable insight/vision of core members who have been here for so long. The review process needs a system that will also allow progress to continue without complete reliance on any single person.

The main point I want to make here is that TR needs to have system in place that requires an absolute minimum amount of continued effort required from leaders to function long-term. Their vision and desired processes need to be preserved in a system that will allow things to smoothly continue. I think it imperative that this be done across the board, but especially with reviews and the planning/opening of new exterior/interior claims (which I haven't mentioned here.) Any point in our work process that is holding up progress on multiple counts for months at a time needs to be addressed.

[Thrignar Fraxix]

I'll say at the outset that this post comes after a full 8 hours of incredibly annoying and frustrating work.

I cant really comment on what went into redoing dozens of interiors 2 or more times as I wasn't here at the time, but I think almost any of these would be good examples. My second interior instantly comes to mind though. It had been completed 2 or 3 times before me. I was approved for a 3x3 layout with a fireplace. Midway through I found out it was the wrong dimensions. I then asked several times for the fireplace to be approved, even adding the request into the text for the file so it would be viewed when opened. This was a highly detailed interior with nearly every object painstaking leveled to the .01 decimal on de_p. This thread and file no longer exist on the forum. I don't know how many more instances of this there have been but I can imagine there have been many.

You somehow think that redoing claims is because of a lack of communication between the core and the claimant. Do you have no idea as to the timeline? Those claims were approved, the redos generally don't happen while a claim is claimed, they generally happen far down the road when we look back on our past actions, collectively sigh and lament how stupid we once were. You can't fault claim redos like that on poor communication, they are faulted on changing standards and poor planning. Maybe read the dates on those posts before jumping to conclusions. (For example, the claim you talk about was approved in 2007)

On the thread you mention, just because you can't see something doesn't mean it isn't there. The claim currently sits in Interiors N/A. I am reading the thread right now. While the fireplace could have been mentioned sooner, it should be noted that we are human. You also seem particularly perturbed about its removal but beyond the space and redundancy issues with the fireplace, it was a ravenrock quest scripted item, which couldn't even have simply been replaced with a non-quest variant as that would still constitute style mixing. I am amazed that this wasn't caught sooner, but should never have been reported as anything but an error.

On the Dimensions, Jule said in the review that it was ok as a 3x3, even though it should have been a 2x2. 2x2 ints are exceedingly rare, and generally bumped up to 3x3 as a rule. This was a difference in opinion between the HoI and a reviewer, which is a non-issue. The ranked one superseded the non-ranked one.

On the fireplace specifically, you asked about it once. Only once. Don't lie. I don't see it mentioned in the file comments for any of the WIP files, unless you added it into the actual description for the .esp visible only in the CS for the .3 seconds someone looks at the screen while loading files.

Oh wait, I can download all of those files and look to see. Yeah, none of the files in the thread uploaded by you mention the fireplace at all, unless it was in one of the files we cleared out because you cluttered the top with a ludicrous number of WIP files. Still seems like something you would have left in until you got it ok'd though right? So it should be there in the files up til that point if it was ever there in the first place.

One more thing, the interior had only been approved once before, in 2007 as stated previously. Again, don't lie.

I'd like to know what the restructuring means so I'm looking forward to learning more about this. I think the input of everyone should be included to make any changes. I have noticed problems in general planning but mostly the interiors reviewing process, because I have only done interiors. For this reason I am bound to have good insight on how to improve workflow in that specific area. I'm sure developers from other departments have more ideas specific to their area of expertise that I cant mention because I haven't experienced them.

I have no further comments re-restructuring. You have raised your issues with interior reviewing and I have rebutted the things you've raised on a point by point basis. I am done doing this after this post as I don't believe I can convey reality to you, nor do I think you are particularly interested in it. Still going to finish out this post though so enjoy it while it lasts. I will also say though that 13 interior claims does not automatically mean you have good insight into how interior claims work. Insight comes with time and understanding, and cannot be gained over night.

First, while topics may overlap somewhat, this discussion is on the "review" process specifically, not general planning.
Second, I'm not going to continue to insult specific people by bringing up past threads. I will say that if all planning was as thorough as this or held to the same scrutiny as interiors MANY problems that I am in no way responsible for could have been easily avoided. The fact is that ideas, my own and others, were being shut down/ignored in previous threads and discussion was being diverted from the topic at hand. When I called person(s) out on this those threads were shutdown each time (but this is a diversion and almost entirely unrelated).

From the bottom up, those threads were locked because they had become unproductive and heated, thanks in large part to yourself. You raised non-issues and imaginary slights and turned them into fights. I have told you several times that the Almalexia thread was locked for this reason, and for the reason that it should not have been made in the first place. Fact is, by the logic of heated threads being locked, this thread should also be locked, though for the first time since the rule's inception this won't be your fault. You are right though, this is off topic so lets not talk about it. I will repeat again, if people want to give unsolicited comments on int threads, they are more than welcome to do so while they are in claimed. It is not required though and most people like to do their own thing.

First, the problem you mentioned could be resolved by dividing exterior/interior reviews by region. Bumped claims would remain roughly sorted. I can't understand why some interiors should take priority over others if they would remain grouped with claims from the same area.
Second, good point, however I think many people have, or would have, good points of critique other than bleeding/caspering and floating objects.

Again you demonstrate a complete lack of understanding of reality/circumstances. The claims browser is something coded into the very forums. The sorting by claim map is not something we can change. I mean good lord think about it for a minute would you. People still to this day get mixed up because we still have different claim maps from release maps. Don't you think that if we had the technical capabilities to fix this, we would have? This is not something that can be changed, and we've learned to deal with it.

As for critiquing the ints in review, they are done at that point. The time for critique has passed. Comments like this can be made when the claim is still in claimed.

Not everyone has a desire to go check every object for floaters and bleeders. I understand how this could be a drag and so I respect those who are willing to do this. Next, I don't see how any review can reasonably take longer than one month. Our current problem is that they just are not being done. This not an issue that a half dozen reviews actually takes six months to complete.

This brings me to my next point on this. Sixty interiors should not even be available for review in one day. Interiors in review should not go above ten. I don't care if reviews have been irresponsibly handled for the last 10 years or how long absurd numbers of them have been stacking up. I care about what is happening right now and in the near future. This is why I believe it is so important that you help complete a thorough reorganization with all the insight you have gained over your many years here. I'm sure you have many more ideas than I on how to improve things here. Right now our process bottlenecks down to a single person in places. For a project with such a long history and such a massive amount of content I think that is a poor idea in the long-term. Right now nothing is moving in reviews and while they sit, no last minute changes can take place and no discussion can be viewed.

If things are not being reviewed as in-depth as the should be or with proper frequency, mistakes cant be corrected and improvements cant be made on future claims either. This creates further problems for future reviews and at the very least prevents easy improvements from being made. By the nature of this project there is no guarantee the claim holder will even be active in 6 months to a years time. For this reason claims need to be handled reasonably quickly and thoroughly.

Currently there is an average of two or three interiors being completed a week. Once the 90+ claims interiors in review are cleared, reviewing claims at this pace should be manageable. Maybe relative priority should not even be a concern. All claims waiting on review should be high priority and if completing claims isn't a priority then they shouldn't be posted. Doing so is irresponsible.

Oh my, this is a big one.

Reviews in reality for me take less than 5 minutes a pop. I have a system, that is very fast. I'm sure others take longer, but the point is this is not some time intensive process. It is just something I have to have the mental fortitude to do. Oftentimes these days I don't as I am currently a full time college student in my last full year, working part time at a stressful job, running a student organization hell bent on getting itself disbanded, and trying not to go insane. I admit reviewing needs to go faster, which is why I have promoted Andres to final reviewer and have my eyes open for other candidates. Final reviewers will come up later though.

You need a history lesson, even if you think you don't. When I became Head of Reviewers there were WELL over 300 interiors in reviewing, and yet the department was still running along just fine. It was a terrible situation, and it was during that time that I was able to consistently do ludicrous numbers of reviews in a day. After the backlog was cleared and the forum emptied for the first time since its creation, I would let them build up to reasonable levels about what I could do in a day, then power through them all. I haven't been able to do this in a while, hence changes that are in progress. I was always the only person to do it, but it is only somewhat recently that I've actually become a bottleneck. But hey, every department used to have this problem, thats why we had no ints go up for the saddest amount of time. Guess what though, changes are being made! Stop complaining because stuff isn't happening fast enough. Last minute changes can be PMed, and if discussion needs to happen in an int, then the claimant can request it be sent back, otherwise it is a non-issue.

You also seem to have a misconception about how the review process works. 95% of the time, the reviewer fixes the mistakes in review, and the claim does not get sent back to the claimant. You are the only person who I can think of in recent memory who has had multiple send-backs on the same int as well as mutliple ints sent back.

As for how in-depth the reviewers go, they list the errors, they make comments about the overall int as they see fit. If nothing is said, people assume them to be fine because if something was wrong a good reviewer would say so.

I agree that they need to be handled faster, but it really isn't as big of an issue as you make it out to be. As far as the priority is concerned. The system we have now works just fine apart from review speed. Ints can be marked as high priority and treated as such, and everything else gets done in order. If we didn't put up lower priority ints then every int claim would be a painful rush that many people would be unable to enjoy.

You are assuming something I had not intended here. I am not suggesting indiscriminately requiring elements of suspense, adventure and lore in every outhouse, tool-shed, and shack. These things NEED to be addressed where appropriate and, as far as I have seen, they are not.

Also, I have mainly focused on interiors because this is the area I have been active in. I apparently wasn't clear enough that this discussion was intended to cover interiors/exteriors and any other review process the topics here could also apply to.

I am using the term "creative movement" here to encompass the paths the player has available to them. In our creative process we should allow for the player to use their own creativity in the gamespace. Incorporating "creative movement" into the planning and review process would attempt maximize the enjoyment and freedom the player has as they move through the game. Since so much of Morrowind depends on moving it is an aspect of interior/exterior design that should be more formally recognized.

For the more important ints, we tend to require that they go to experienced modders who we know to be capable of doing those things. It is not something we feel we have to babysit claimants about.

You've only discussed interior review, and no one seems willing to raise any complaints or comments about any other form of review, so this is pretty much just an int reviewing thread.

I still am not fully understanding you on the creative movement thing. I agree that caves should perhaps have multiple paths, but that they shouldn't always because that feels samey. I don't think creative pathing is neccesary in common run of the mill interiors, and really I am not sure it matters intrinsically for what it is even. A good interior isn't good because you can walk both ways around a table. A good interior is good because the table feels like it is positioned in a way that makes sense. Caves aren't good because they are laid out in a creative way allowing for great freedom of movement, caves are good when they tell stories with their paths, items, and statics. When they feel like they are alive, and that though the inhabitants may be gone or dead, that they were there at one point, whatever they might have been, if only plants.

You are taking my comments/critique out of context. I am not claiming the entire fate of the mod rests on an adequate review process, but is one of the most glaring, complete failures in our work process. It needs to be addressed. That is the purpose of this discussion thread.

It isn't as complete as you think it is, you just seem to take personal offense that your claims haven't been approved yet, even when they were reviewed well ahead of other ints that had been waiting far longer. While none of your ints are high priority, they still managed to receive initial reviews before an int submitted in May 2013. I'm going to go ahead and make a plug and say that we could indeed use more initial reviewers. I've already graded one test in the last 24 hours, and I believe this person is going to pass once they make some minor corrections I've already requested. There is a high priority int waiting on an initial review right now.

Developers being willing to claim interiors does not absolve our faulty review system of error. I really enjoy interiors and will gladly do more, but I do not think it is right how many aspects of review and planning are being handled at the moment. The willingness of people to contribute to the mod knowing full well how far off official releases are shows the amount of genuine interest people here have in continuing work on the mod and seeing progress made. Much of this is due to the fact that TR is the largest MW mod project, but this in no way changes the fact that there are MAJOR problems with planning, waste, and review here.

I never said it aboslved the problems that I agree exist. I just said that it meant they weren't nearly the issue you think they are.

You've raised a great point here. We need an updated agenda, every month, that includes a reasonable timeline of expectations and goal-setting a list of priorities including high priority tasks and subjects for discussion. It should also include proposals that have been shelved for progress at a later date. I think this point alone should have its own thread at a later date (once you have reorganization master plan that has long been in the works has been revealed.) Hopefully there will be room for revision at that time so we can work together to get TR in top working order.

Timelines are not something we do at TR because historically when we try to do things by a set time we fail. If you want timelines at TR then give up. We do need a list of high priority tasks, but that should be updated as the high priority tasks change, not some sort of arbitrary timeline like you seem to think would help. TR goes at its own pace and any attempts to hurry it are not taken kindly by the people trying to actually live their lives. When we've established high priority tasks, we label them as such, see the high priority ints. The thing is, our planning is behind, so we don't have nearly as many of them labeled as should be. This is indeed something that will be discussed elsewhere.

None of this immediate discussion is meant to be an insult to Andres or his capabilities as a Reviewer. I consider everyone here past and present a valuable asset to the mod. HOWEVER, I think it is reasonable to suggest that there should be more than one final reviewer. I assume everyone here has other obligations and interests. A reviewer may be perfectly capable, but if consistent progress is not being made at pace with completion of claims its a null point. Spreading responsibility would ensure continued steady progress.

IMO no reviewer should even be promoted if they are not held to the same standard. Completing a final review should be an option available to all reviewers. After a preliminary review process of critique by peers and technical review by an official reviewer, a second review should be a sufficient final check. Any two qualified reviewers completing reviews should catch any problems. If they don't, the issue is likely inconsequential or will be caught in alpha as so many others have.

This is once again a demonstration of a complete lack of understanding on your part. Andres is the second final reviewer, I remain the first. More are coming should we find someone capable. It is a lofty position though that is not handed out lightly.

Your lack of understanding continues with the belief that it is possible to find out if someone is capable of being the elite final lines of defense as easily as with the TR reviewing test, or any simple test administered in a single go. Reviewers are expected to have a good eye for detail, yes. Final reviewers are the reviewers of the reviewers though. They have to have as much better of an eye for this sort of thing than the reviewers, than the reviewers themselves have in comparison to a normal modder. You want to relax standards so things move faster, I get it, but that is not going to happen. The reviewer/final reviewer distinction shall remain, otherwise there is no guarantee that the job is being done properly.

On a slightly different but not unrelated note:
The work that has been done, largely by Sload, on the Masterplan thus far is great. The "Everything, Everywhere" work is excellent.

Masterplan
http://tamriel-rebuilt.org/old_forum/viewforum.php?f=226
"Everything, Everywhere"
https://docs.google.com/spreadsheet/ccc ... _web#gid=0

These are the beginning of a framework of information and guidelines that, once completed and if properly followed and maintained, will ensure that future work can continue with a level of quality and consistency worthy of the work done thus far with or without the active valuable insight/vision of core members who have been here for so long. The review process needs a system that will also allow progress to continue without complete reliance on any single person.

The expansion of the ranks of the final reviewers is a part of that plan that may or may not have ever actually been articulated. It is a slow process though because of how strict my standards are. This is the way it is and I encourage you to just accept that this bit of the change perhaps comes slower than others.

The main point I want to make here is that TR needs to have system in place that requires an absolute minimum amount of continued effort required from leaders to function long-term. Their vision and desired processes need to be preserved in a system that will allow things to smoothly continue. I think it imperative that this be done across the board, but especially with reviews and the planning/opening of new exterior/interior claims (which I haven't mentioned here.) Any point in our work process that is holding up progress on multiple counts for months at a time needs to be addressed.

The main point I am trying to make here is that, yes reviewing is slow, yes there need to be more final reviewers, and yes more are coming as they are found to be suitable. Additionally, there need to be more initial reviewers, and this is something people can actually do something about in the here and now. At this exact moment though, final interior reviewing isn't holding up anything other than my ability to relax after a hard day at work, and even that is just because I've now spent about two hours typing this damnable post. There is a high priority int needing an initial review though (hint hint reviewers).

I implied I would lock this thread, but I will not. Since I have stated previously I will not reply further to this thread I will sum up my statements though:

-Final reviewing distinction is here to stay, though more will be promoted as they are found capable.
-We could indeed use more initial reviewers.
-Interior reviewing is slow, and I feel bad, but it isn't nearly as big of a deal as you think it is.
-Posting will continue to be disallowed by normal members in the review forum as any comments on the int itself should be made prior to completion.
-Re-orging the claims forums to be by section is not currently possible.

Nothing in this post is much of a revelation to those who know what is going on except maybe that we need more initial reviewers. The claims browser being a poorly documented mess is common knowledge for anyone who has any knowledge about it whatsoever.

[sasquatch]

Thanks for replying to my previous post TF. I should make an addendum to make it clear that this was not meant as a personal attack or "complaining." It is a critique of problems I have experienced with our workflow and communication. The review process, not reviewers or reviews specifically, has been a failure. They do not encompass enough of the aspects I have mentioned in the post. I appreciate all the work you have done reviewing interiors for years. As I said, I understand everyone has other priorities and interests outside of this project and for that reason individuals cant be placed with full blame for the lack of reviews, rather the review system, for reasons I have discussed here. Adding more final reviewers will be a big help so I am thankful you are willing to promote more as they become available.

Now, on the interior I specifically mentioned. I used this as an example where I had experienced problems with poor communication which led to issues later. Remember, this was only my second interior and had very little idea of what I was doing or how to make a fireplace. If this had been resolved earlier it would have saved me a lot of time. This is a good example even if it was a relatively small complication because it shows the need for thorough early input it is relevant to much larger tasks as well. On a larger task tens or even a hundred hours of work could be jeopardized if the right conditions and vision are not implemented for a design.

Also. I know for a fact I asked twice. Once in line with text of when it was first added and in an additional post requesting again that it be checked. Also, I DID add a request on the file. If those file versions are not there it is because they were later deleted because there were so many versions (10+) not because it was not uploaded. I cannot view the thread so I cannot prove it myself with a link, but if you check again you will see you are mistaken on this point.

Last, there is an assumption here that I am merely ignorant, troublesome, and assume that I have every solution to all problems that don't exist. This is plainly false. I did not lie, nor am I calling people by name, shifting blame or making personal attacks. This is a critique of workflow which is something I was invited to do and something that is needed. These are constructive ideas that, thankyou, have been appropriately addressed and I appreciate that. I hope that you will continue to consider at least the idea of setting an agenda with topics for discussion each month, and I do think it should be a month. Setting a timetable for a list of high priority discussion and planning topics will help move things forward. All proposals should also be treated with the same level of scrutiny as interior reviews and many future problems will be avoided. I think that is needed.

[Yeti]

I also have access to the interior thread, Sasquatch. As far as I can tell, you only asked once in the thread itself about the fireplace.

I need someone to check this to tell me if the fireplace is passable. The exterior area has a fire pit and boiling pot where the baskets of ingredients are headed. My plan is that the inside grill is used for cooking the questionable meat away from prying eyes.

[Haplo]

I have cleaned out all the posts regarding slander, police, memes, flamebaiting, trolling, etc., as they are derailing the thread from its topic and generally hurtful to the atmosphere of this community. I apologize specifically to Gnomey who posted right as I was doing this cleaning. The posts can still be found in the Archives, as per usual.

I will take this opportunity to remind you all that flaming, flamebaiting, and maliciously trolling users, even with lighthearted intent, etc. are all against the rules. I urge you all to stay on topic, and refrain from boldly labeling other users as liars. If you have personal problems with someone, take them up via the PM system or address a Lead Developer with your problem in private if the situation escalates.

EDIT: Feel free, of course, to continue discussion regarding the reviewing process.