GitHub/Redmine usage? Please read!

Not · Post by **Not** » Sun Dec 13, 2015 3:15 am

Okay, I should have made this post a few hours ago, but horrible things got in the way (mainly sleep.) Anyway, I wanted to go over what was talked about in the Skype meeting that took place earlier today and what plans we came up with as far as the Claims system goes. I would like to first point out though that NONE of this is set in stone; these are just a couple of ideas that we came up with and any input is very strongly encouraged.

As I've stated in the pipeline thread, I don't believe the order of the workload is going to really change much, I don't believe that it's going to make much of a difference one way or the other if we go for a top to bottom approach or vice versa because I never believed that was an issue in the first place. If we're to move forward at a better pace and finally leave that damn gas station in Creuse, we need to become more organized.

To do that, we need to start with the claims system. It's incredibly outdated and we should have phased it out years ago. However, due to circumstances beyond anyone's control, the backlog grew too big and the claims system became a mess. No one really knew where to start anymore and no one wanted to tackle the daunting task of trying to reorganize it because just the thought of it is so overwhelming.

That being said, I talked to our friends over at OpenMW and they told me about github. It's one of the tools they use to stay organized and continue to move forward. I was looking into it and it does seem pretty awesome. Granted, it can be a bit overwhelming for someone like me who's never used it before but it WOULD help with our work flow and amount of progress we're making.

For those of you that have never heard of github, it's a website that helps projects like TR stay organized and on top of things by using a tracking system where we'd put everything that needs to be done, i.e. claims into a sort of "to do" list called Issues. Everything would go under issues in github, be it claims, bugs or technical discussion. Each individual item posted under Issues would then be given a sorting tag to better keep track of it, e.g. bugs, int claims, exteriors, quests, etc. Each "Issue" at this point can then be easily monitored for progress through the use of neat colored graphs that show how far along each claim/issue is.

Keep in mind, we would NOT be abandoning the forums at all. The forums would still be used for the ideas and discussion of claims along with everything else, the ONLY thing that would be on github is the actual technical side of the developers. For example, if someone is working on a quest and a script isn't working, they could post what they have on github and see why it's not working. (It's just a very rough example.)

Picture it like two sides of a coin basically; the planning and discussions would take place on the forums like they always have but the actual implementing and CS work would be done on the github side. That being said, github is entirely transparent and everyone can see what's going on. It's just a means of us trying to stay better organized and on top of things, by no means should anything be considered seperate. If nothing else, think of it as an organizational tool for developers that would enable them to better keep track of things yet also allow planning and discussion to have a better idea of the bigger picture.

I'm going to give a very broad example here of how we might use github, this is entirely hypothetical, but here's how we'd use it.

Let's say that we haven't started on the OE section at all. Nothing has been done, no exteriors, no ints, quests etc. It's just sea. Naturally we'd discuss what needed to happen to make this area on the forums; we'd plan it via skype meetings (although we certainly wouldn't take a month or longer to come up with a decision,)we'd discuss the vegetation and the weather and decide on the coordinates for the exterior cells and whatnot. When we finally decide where the exteriors are going and what specific things we want to see happen with each of them, we'd put them up on github using the Issue tracker. We'd give the necessary tags so people know that these are unclaimed exterior cells. There would be graphs involved that would show progress, we'd know that these particular Issues (claims) for example haven't been started yet and so 0 progress has been made. A developer would then come and click on the issue and accept the claim. At which point the tag would be updated to exterior claimed and the claimant could then post periodic updates to show the progress. Finally, the Issue (claim) is finished, we'd cross it off the issue list and the final file would then go into the github repository.

Again, this is a very rough example, certain steps may change but I do hope this can be used as a general overview. The whole point of github is to better keep track of our workflow because right now everything is such a mess and so disorganized.

Pros
[spoiler]-Using the Issue tracker, coordination between planning and implementing should theoretically become that much easier.
-By having everything we need made in a list and seeing what needs to be done, we can greatly improve our workflow and overall organization.
-This would finally phase out the horribly outdated claims system for something much more efficient.[/spoiler]

Cons
[spoiler]-We can't upload binary data. Binary data is anything that is not considered a text file, i.e. audio, .bsas and meshes/textures. (esps/esms are considered text files and we'd have no problem uploading them)
-Anonytroll brought this up to me, but apparently github is designed to work on one masterfile, not individual files such as claims.[/spoiler]

As you can see, I listed the pros and cons of what it would mean to implement this system. The first con isn't a big deal because audio and that kind of work should stay on the forums anyway. The problem is the second con, because I'm hearing different things, such as we might be able to work around that or we might not. Again, I'm still teaching myself github and don't know everything about that I'd like to just yet. However, from what little I've gathered, our completed work could go into a repository which in theory would pretty much tell us what needs to be merged into the main file upon completion.

Again, all of this is just one of two great ideas and I'd like to have feedback on it. Maybe someone knows of an even better site than github that's geared towards working on multiple files? I don't know. But if it's true that we can't work around that last con, then that's pretty much a deal breaker for github.

mspyerzuka · Post by **mspyerzuka** » Sun Dec 13, 2015 8:55 am

I can ask around who use it and send them here. Maybe get them to write out a response i can edit in this post, assuming they will probably read this post as well. Hey guys.

Sorry, I'm tired and will toast some more fresh info when more rest is contained.

Not · Post by **Not** » Sun Dec 13, 2015 9:53 am

Actually! I just got done talking to a friend of mine from OpenMW. We just went over getting around the binary data issue and multiple file uploads. It's all a green light for us!

Basically, because our files are so small, we shouldn't have any problems uploading esps/esms. Github may not be specifically made for that kind of stuff, but it CAN do it no problem. The other concern that was brought up in IRC is that github is more geared towards projects that use one master file and work is made directly to that rather than multiple esps (claims.) Well, we overcame that problem as well and it's not an issue at all. Github encourages collaborative efforts and this falls into that category. It won't be a problem. I'll give an overview of the general workflow and try to explain.

So once a claim clears the design phase on the forum (what I had posted above about the hypothetical OE discussion for example) we'll move it to github where it'll become an issue.

Once it becomes an issue, anyone can work on it if they so choose, there'll be an option to claim it, at which point it'll be updated with a tag (i.e. interior/exterior, claimed) and then the claimant can work on it. Once finished, said person can submit a Pull Request which tells a github admin (i.e. one of the lead developers) to put it in review. Once it clears review, it'll end up in the github repository which will pretty much become a list of all files that are ready to be merged with the main one. We'll still be merging the files using the same method we always have, but at least this way we have a repository which shows all the files ready to be to go. Once the files are merged, they are removed from the repository and then it's rinse and repeat.

Github also has a milestone system, wherein each finished file that ends up in the repository will show how close we are to our next release through colorful graphs. When we've reached a milestone, it means that we're ready for our next release (pending file merges of course.) In this manner, we can see what needs to be done to push out our next release as well as visibly see how close we are to our next one.

I also think it's worth mentioning that it also shows the amount of contributions that each developer has put in over any given amount of time through line graphs. Personally, I don't think it's really important, nor would I hold it against people that have made little to no progress with their contributions (hobby project and real life after all.) The reason I bring this up though is because there's always the possibility that people may start some friendly competition to see who can push out more content. I don't know, we'll probably never pay much attention to that, but I figured it was worth mentioning anyhow.

Actually, using this same concept, you can see how much progress has been made on specific days compared to others, which I think is pretty cool.

Overall, I need to learn how to work this system, I'm looking for a tutorial right now that'll help me better understand all of this, but I'm definitely liking what I'm seeing so far and really think this could work.

Post by **Seneca37** » Sun Dec 13, 2015 1:13 pm

I've updated the subject for this thread. While 'Kittens!!' may be cute, they certainly do not convey the main ideas in this thread.

My thoughts on Github.
Github was developed for software geeks ( like me ), who want to collaborate on building a piece of software. Unfortunately, I have never had the need to use this system, so I am unsure of it full potential. But here are a few of my concerns: How would maps be incorporated into claims, what about reviewing (is there a review system built in), can images be added to claims, => can it do everything we need it to do? How easy is it for a 'non' software geek to use? Would this be separate from our forums, or could it be incorporated into them? Who is going to implement-maintain this system?

Even though some parts of our claim system are 'broken', the system on the whole is working. I don't see the need to go to something new at this point in time. We finally got a webmaster who knows our current system (Drupal) and I, for one, am not going to ask him to learn something new. He has enough work just trying to get our system up-to-date.

I am not opposed to using something new, but I would need to see a demo of the system in an environment such as ours (OpenMW is a software development site, not a gaming/modding site like TR) before even considering the possibility of using it.

Jet133 · Post by **Jet133** » Sun Dec 13, 2015 5:03 pm

Speaking as an outsider that basically just lurks and waits for the next release, I'm going to throw in my two cents here for the fun of it.

I think something like Github would be very beneficial for all the planning documents such as the Master Plan. I haven't used it before, and this is all based on the descriptions provided above. I know it would be nice to have something like that for the exterior/interior/quests claims system, but I think having a simple one for planning would be good. If you can use it for multiple files, then I see it as a definite boon for that side of the project. It would keep it all in one place, and it would keep track of how much planning work has been done or completed on a particular subject.

For instance, if someone wanted to work on an interior in Kragen Mar, they could look at the planning document for the city and see under a "characters" section that there is planned to be trader that regularly makes the arduous journey to take goods from Morrowind across the border to sell in Cheydinhal. So they finish the simple house claim and it goes off to review. CS work gets done and more of TR is complete and progress is shown on the project in a more visual way. Yay!

So then the modder goes back to do a second claim and looks around the city. They notice the interior for the Fighter's Guild hasn't been started. They look at the planning documents and see a blank section for the Fighter's Guild in Kragen Mar. They then look up the main section of the document and see it is only partially filled out. They see the document is only 40% complete (or however Github measures its progress) and decide to post in the relevant threads on the forums to get the discussion moving on the Fighter's Guild because they want to work on the claim. A week or two of discussion and planning happens on the Fighter's Guild in Kragen Mar and in general. This produces enough ideas about the faction and its relation to House Hlaalu for the modder to get started. They then claim the interior and get to work and finish a first pass (still held up to interior design standards mind you) and send it off to review where its approved and merged, and then the modder goes on his merry way.

In the meantime, more discussion has happened and perhaps someone came up with the idea that an underling in the Fighter's Guild is working with the previous trader to smuggle things across the border to and from Cyrodiil to make an extra coin here or there. A note or flag of some kind would be put on both the Fighter's Guild interior claim and the previous trader's house interior claim to add stolen good to a new basement or other evidence. Then a second modder is searching throughout the interior claims forums looking for something to do. They notice this note for some changes to be made and decide to claim the interior for touch up work (no major re-dos unless a much wider consensus is reached or the quality is somehow really poor), such as adding the basement or some hidden away smuggled goods or a note from the smuggler to the FG member. They complete the work and send the interior off to review where it is finalized and merged.

Work has been redone, but minimalized and accomplished nearly simultaneously with planning, while using an interior modder's desire to work on a claim as a sort of kickstart to more specific planning discussion and implementation with planning.

Not just in game design, but it all kinds of large projects, things are constantly updated as new ideas emerge, but they're not always given major re-dos. This centralization of the planning and quasi-concurrent work would hopefully get things to a more fluid state while hopefully providing incentive based on madder's desire to work to get discussions "resolved" in a more timely manner.

Thoughts from those in the know?

Post by **Anonytroll** » Sun Dec 13, 2015 5:39 pm

The issue of me being overworked by learning how to operate Github are not applicable - if that's what we're going with, I'm not operating it as anything other than another developer.

I figure I'll have enough to do with the website itself (it's not going to be 100% finished in January, and if all else, the problem of the slow end of D7 is not going away and there needs to be an update path in a year or so at average).

But that's also not necessary - Github is a hosting service (the software itself is just plain Git), and they and the people who've been using it for a while know what they are doing.

The problem with Github is essentially that Morrowind's esp files are not like code. Notling got it the wrong way around, actually: Github projects are mostly made up of component source code files or libraries. Morrowind is made up of various monolithic blobs of compiled codes with a clear hiearchy of esm -> esp and all kinds of hell when you try to do esp -> esp.
You can't even really merge and compile them together, that needs painstaking work in the cs (or your doors won't go anywhere or into the void, for example).

That, I think, eliminated Github's greatest strength, and makes me curious for the test run in January. I've no experience with Git at all, so I wonder how and if that can be worked around.

- - -

In opposition to it we have the Claims Browser originally created by Adarnocil, then left to age (like fruit juice).
The basis is roughly the same as anything on Drupal (a node with comments and uploads), but it replaces the forums with a workflow field and tries to implement the claims relations zephy once made (but only somehow work).
I've also taken a look at the feedback and added a queue term (a free-flow taxonomy term), so

The upside is that it wouldn't change the current workflow much at all and would integrate into other things on the website (taxonomy first of all, global and local bookmarks, existing user accounts, same database), when those are done. As it is just a node, it's also extendable as much as we want to and can do (which, admittedly, is not everything).

The neutral is that it's not an actual project or versioning software, because that doesn't mesh with the workflow of TR much. It's more hierarchical and the way that esps work makes me doubt how good a true collaboration versioning control system like Git (as said above) would work.

The downside is that it's all cobbled together by me, it is in comparison to a dedicated versioning system hackish and unrefined and using it would not challenge TR to find a better workflow (which might give it new impulses).

It's also not done, but I'm very positive there'll be a working version for the test run in January with the current kinks if not worked out then with a workaround.

Post by **Gnomey** » Sun Dec 13, 2015 6:35 pm

For my part at this point I'm mostly waiting for the test runs of the two systems to try things out in practice. The dynamic of the two test-runs will probably be somewhat different; in the case of Github, the functionality is all already in place and we need to focus on learning how to use that functionality to our ends, whereas in the case of the claims system the functionality may start out somewhat bare-bones, but can be extended, as long as the extension is within the bounds of Drupal's and Anonytroll's abilities. Basically the difference between buying a suit off the rack and buying a tailor-made suit.
If the suit off the rack turns out to fit or can be altered to fit, it will certainly be a very viable option. Even if it doesn't, we may discover some functionality we might want carried over to the claims system.

I find Jet133's example especially interesting, and if we can get something like that set up on either system I'll be quite happy.

psi29a · Post by **psi29a** » Mon Dec 14, 2015 4:16 pm

Hello! I was bright bulb that suggested github before realizing that Anonytroll (or anyone) was working on a way to improve TR project management.

I firmly believe that shoehorning TR's workflow to work with github is not ideal. I'm not sure the gains from a management perspective outweigh the additional work necessary to handle the new github based workflow, such as learning github and/or git itself which.

That being said, I can see how github could be something temporary until Anonytroll's system is a fully operating death star!

I've worked with Drupal (version 5 and 6) in the past, so I can imagine the amount of work it would take to get something like this up and running.

As a possible third option, we (I can help too if necessary) can take some of the nicer aspects of github from the management side and add that on top of Anonytroll's system.

In theory, the taxonomy could be used to doing view filtering to make searchable lists. It would be possible to have progress tracking and milestones as a result of proper tagging.

Any thoughts?

Not · Post by **Not** » Tue Dec 15, 2015 8:03 am

Alright so, psi29a and myself went over github and potential workflows extensively last night....err, early morning for most of you :p

Anyway, we realized that github isn't really a viable option for us at this point, because there'd be too much going back and forth between this website and github. Ideally what I wanted to see happen was ideas and designs for claims would start on this forum, then when they're ready to be implemented they'd be carried over to github where developers like myself would work on them. The problem with this however is that while it sounds good in theory, there's too many drawbacks that we couldn't find an easy workaround for. We'd more or less have to take certain uploads back to the forums if say, a claimant could no longer work on his claim and whatnot and it would just become a mess.

Nevertheless, I do think it was a good lesson because it showed a really amazing idea for great taxonomy and workflow. If we could somehow incorporate something similar to that process into what Anonytroll is doing, then we'd be all set. So all in all, I think this just gives some ideas of what we could be using for the new and improved website.

psi29a wrote:As a possible third option, we (I can help too if necessary) can take some of the nicer aspects of github from the management side and add that on top of Anonytroll's system.

In theory, the taxonomy could be used to doing view filtering to make searchable lists. It would be possible to have progress tracking and milestones as a result of proper tagging.

I couldn't agree more with this statement. Anonytroll's system already allows us to view parent/child claims and that's going to be super useful, if we could expand on some other ideas, I think we'd be all set.

Sorry if this post doesn't make much sense...I'm literally half asleep as I write this ><

Post by **Gnomey** » Tue Dec 15, 2015 5:27 pm

Is the mismatch to the extent that a trial period testing out Github would not be viable/too much trouble? Because I still think that it would be good for us to get a bit of hands-on experience with Github and its feature set, even if we won't use it in the long-term.
I think we'd be better able to judge which of Github's features would be worth carrying over -- in what form and for what parts of production (QA, writing, modding, project organization...) -- with hands-on experience as opposed to just theoretical 'Github has this cool feature that maybe will be useful to us'.

Edit: not to mention that I think it will be more difficult for us to think of ways to improve our current system if the current system is the only one we're used to and familiar with. We might treat weird peculiarities of our system as necessities.

NathanJ · Post by **NathanJ** » Tue Dec 15, 2015 5:38 pm

A little idea by the way: wouldnt it be useful to let some people from the very outside of the project look at our current system and the experiences TR has made with it and let them contribute their judgement to the discussion? Because people working with a system sometimes become blind for obvious possibilities for improvement after a while.

Post by **Gnomey** » Tue Dec 15, 2015 5:47 pm

It would be, yes; the difficulty is mainly in finding people willing to take the time to get acquainted with our convoluted system.

psi29a · Post by **psi29a** » Tue Dec 15, 2015 6:08 pm

Gnomey wrote:It would be, yes; the difficulty is mainly in finding people willing to take the time to get acquainted with our convoluted system.

I'm a willing victim...

Not · Post by **Not** » Wed Dec 16, 2015 1:50 am

psi29a wrote:I'm a willing victim...

That can be arranged! No but in all seriousness, I can't thank you enough for your continued support and willingness to help out. If you're willing to familiarize yourself with this convoluted system, then so be it. It'd be nice to have an outsider's point of view, since the rest of us have been so accustomed to it for years that it just seems natural.

Gnomey wrote:Is the mismatch to the extent that a trial period testing out Github would not be viable/too much trouble?

Well the main issue I have with this is that I don't really see the point of a trial run when just the theoretical run through that psi and I discussed the other night started to show lots of holes. I mean I suppose we could do a trial run to show others, but the main issue is that we'd have to teach everyone a brand new system that's very different from our current one, and I just don't see the point of that when we're not going to keep it anyways, it's just not worth the effort.

Gnomey wrote:I think we'd be better able to judge which of Github's features would be worth carrying over -- in what form and for what parts of production (QA, writing, modding, project organization...) -- with hands-on experience as opposed to just theoretical 'Github has this cool feature that maybe will be useful to us'.

Well that's the thing, I've had enough hands on experience that I could already tell you what should definitely be carried over and what probably shouldn't. Granted it's only one user's opinion, but unless you're wanting to teach yourself github and learn the basics of it (in theory shouldn't take that long, but it's up to you whether or not you want to put in the effort to learn it or not,) I could simply just go over some of the basic concepts we could use anyway.

The main purpose of github and why psi and I even suggested it was because the workflow, progress monitoring and organization was very impressive. However, psi knew all about github, I didn't. I knew all about TR's process, he didn't. The more we started to acquaint ourselves with the other system, the more we realized that the two just wouldn't mix. However, we also knew about Anonytroll's new website design and having seen a demo of it first hand during the last skype meeting. If we could combine the positive aspects that github offers into the design of the website, we'd be all set.

The number 1 thing that I wanted from github most of all was the organization and progress monitoring that it offered. It had a milestone system: the more claims that are completed, the more the graph is updated and shows how close we are to our next big release. It showed how close we are, what was still left that's needed to be done before release and was a lot easier to keep track of when you use the graphs to see how everything is doing.

With our current system, you have to check numerous threads to see what is done and what still needs to be done, and even I myself couldn't begin to tell you what needs to be done for our next big release because I have no clue where to even start. I was hoping that github could help fix this. That being said however, Anonytroll did mention that he could implement a somewhat similar milestone system to the website, and I think if psi is willing and able to help out with that, then why not?

Ah shoot, once again I got carried away and this post dragged on. Anyway, I guess what I'm trying to say is, github isn't a viable option, but we should definitely look into integrating some of it's key components into our new system.

Post by **Anonytroll** » Wed Dec 16, 2015 6:31 am

Sorry if I'm not typing much about this at all, but I'm a bit at a loss at what to say. I thought a Github solution would get off the feet more and challenge the current workflow into something better, even if I didn't think it would be ideal for the esps. But then, what is? They make everything complicated.

That said, it turns out real-time coordination with psi29a is going to be a bit difficult as we're both on opposite timezones (I pretty much get off work when he stops being available).

So until we hopefully chat on the weekend, I'm going to post a hopefully heling large-ish post of how I think the solution I've been dabbling with is going to work and how it differs from the current one (and it's at its core still Adarnocil's browser, so I assume he knew why he was doing it this way), but this week so far I've been in a perpetual state of exhaustion.
There's probably not going to be fancy graphs, though.

Not · Post by **Not** » Wed Dec 16, 2015 7:38 am

Anonytroll wrote:Sorry if I'm not typing much about this at all, but I'm a bit at a loss at what to say. I thought a Github solution would get off the feet more and challenge the current workflow into something better, even if I didn't think it would be ideal for the esps. But then, what is? They make everything complicated.

I'm almost wondering if I should post the dry run that psi and I did together and see if you guys can't fill in the possible downpoints that we weren't able to. Although I think I'll hold off on this for now because psi did mention to me that he had an idea but he asked me to wait a little while. So I'mma just wait and see

So until we hopefully chat on the weekend, I'm going to post a hopefully heling large-ish post of how I think the solution I've been dabbling with is going to work and how it differs from the current one (and it's at its core still Adarnocil's browser, so I assume he knew why he was doing it this way), but this week so far I've been in a perpetual state of exhaustion.

Nah that's fine, there's no rush. But I do hope this weekend you and psi can work something out...or something. It's time's like these I WISH I knew Drupal so I could help out too x.x

There's probably not going to be fancy graphs, though.

We don't even need 'em! I was just mentioning them because that's how github helps keep track of everything. While they're really useful, they're not vital and I'm not worried about it one way or the other.

psi29a · Post by **psi29a** » Wed Dec 16, 2015 10:27 am

Hello again, I may have a forth option available to us as well. More on that in a bit, but first about github.

I'll leave it up so that people can play with it if they want.

I love github and use it for several of my FOSS (coding) projects, its project management tools are slick.

That being said, the short-comings in the system became apparent when we started talking about various work-flows and scenarios.

I have solution that gives TR the project management tools it needs plus the ability to upload/download files. "Redmine" is collaboration tool that isn't repository-centric like github is.
https://en.wikipedia.org/wiki/Redmine
https://www.redmine.org/

I created a demo on a free hosting site, but it is something that we can host ourselves like phpbb and drupal. Speaking of which, it also has ways of integrating with drupal and phpbb. So that opens up a few doors for us as well.

Have a look around:
http://www.hostedredmine.com/projects/tamriel-rebuilt

To get a feel for it, click on Activity to see things as they happen, Roadmap to see Releases, open issues/tasks/deliverable/etc. and the overall status.

Look at the Gantt chart, as more data comes in... people uploading files (esp/textures/assets/whatever), marking their own progress in tasks, you'll start to get an idea of when a release will be ready.

In Issues, you can track down (with filters) the issues (be claims, bugs, requests) you want to look at... filter on any number of things, like target released, or un-claimed claims.

We have a place to put up documents, modeling guidelines for examples. There is also a place for files... the latest and greatest TR_Data for example.

As for 'issues', you can have parent-child relationships. Say you have a particular city you want completed, make it a task, fill in a description, upload a base esm to work off of... then create all the sub-tasks (issues) you need to complete them. Call them say, Deliverables... or claims!

In those claims, you again go into detail about how much time you put in, percentage done, upload your work, post photos, give comments.

So please, everyone, give it a good hard look and ask questions! At the top right, you can register. Please send me your username so I can add you in as a user.

Keep in mind, this is a demo and has limitation such as 20MiB upload limit per issue. If we host this ourselves, we wouldn't have an upload limit, just the size of the harddrive. It also means we can customize the fields to match TR's terminology/taxonomy.

What do you guys say?

Update:
For a working example of a real real project using redmine in action, have a look here: https://bugs.openmw.org/projects/openmw

Not · Post by **Not** » Wed Dec 16, 2015 10:54 am

Honestly, I love what I'm seeing from the demo, especially knowing that we can reword certain things to better match our terminology. That's great!

psi29a wrote:What do you guys say?

I say you're a ****ing Godsend if this pans out!

Post by **Seneca37** » Wed Dec 16, 2015 11:52 am

I really want to check this out. It does look/feel much better than github. Unfortunately, I'll be away for about 2 weeks. I can test this out more when I get back (sometime around the 30th).

Not · Post by **Not** » Wed Dec 16, 2015 12:08 pm

Seneca37 wrote:I really want to check this out. It does look/feel much better than github. Unfortunately, I'll be away for about 2 weeks. I can test this out more when I get back (sometime around the 30th).

Wow, we finally get a (potential) new system that can get things moving again and you decide NOW to have a life!?? Rood!
I kid! :p

But I do think Anonytroll should take a look at it and test it out, especially if it's compatible with phpbb and Drupal, which is awesome.

Post by **Seneca37** » Wed Dec 16, 2015 2:52 pm

Not wrote: But I do think Anonytroll should take a look at it and test it out, especially if it's compatible with phpbb and Drupal, which is awesome.

Of course Anonytroll needs to look this over, but he and I are coming at this from different angles. He needs to look at this from a Webmaster point of view, where as, I'll be looking at it from a Lead, Developer and general user point of view. All of which need to be considered.

Also, any and all comments made in this thread by 'Everyone' will be taken into consideration. So I'm hoping other Leads and our vast development team will take some time to comment as well.

- just a little comment. The whole setting deadlines philosophy in these systems (github/redmine), goes against the general philosophy of TR. TR is supposed to be 'FUN' and I always tell people - 'do the work when you can'. When we do finally have a new system in place (whatever system), deadlines should not be used to force people to get stuff done. I will certainly not harass 'littlesuzyblue' to get her interior done just so the system can mark it as done, nor will I yank it away from her. We are all fans of the game, and are here to help TR succeed in being a great mod. Real life does happen to most people and does cause delays.

psi29a · Post by **psi29a** » Wed Dec 16, 2015 3:41 pm

While you can set dates in redmine and github, I don't use them in my projects.

OpenMW as an example, we just use % finished per issue to track things. Often, if it is trivial... we go from 0% to 100% and mark it as closed/finished/done/put-a-fork-in-it.

A release is ready when a release manager and developers say it is good enough. With OpenMW, we have a week or two to say to everyone to "wrap it up". If it can't, no worries, it is for the next release.

Post by **Anonytroll** » Wed Dec 16, 2015 6:18 pm

Well, I'm certainly looking at it and will try it out on the weekend (Saturday evening/Sunday, because I'm already seeing too much code in my day as it is).

Also, real-time coordination will have to wait until then. I certainly do appreciate the support and he momentum you guys are building. Don't let me slow you down!

Jet133 · Post by **Jet133** » Thu Dec 17, 2015 4:07 am

Something like Redmine is exactly what this project needs I think. Even if used purely as a tracker and a place to hold files and view completion progress. For example, if working on Uld Vraech, and a complete count of all cave/ruin/city doorways and entrances can be identified, a list of all the possible interiors can be listed as part of an Uld Vraech region or a Velothis district group of claims or tasks. Seeing the amount of work done and yet to be done could spur people to work more on a particular issue if they see it as bringing the project further to completion. I don't believe this would cause the quality of work to diminish, as I feel that has always rested on the amount of effort put into an interior/quest/etc. by the individual creating it. I feel that the developers around here are dedicated enough to keep to the higher current standards and keep lore in mind while working. I see this as a crucial link between planning and implementation.

In the link to the Redmine demo for TR posted by psi29a, I saw a link to a wiki of sorts, but it gives an error 404. If this is a wiki similar in style to say, the UESP, then that could be one of the greater benefits to planning that could come out of it. Having nicely organized searchable information would probably help bring the project out of the rut and get it moving again. This would make it easy for people looking for claims to see what planning has been done for them, and be able to see various applicable decisions and lore for said claim. If it had been decided on the forums or in a Skype meeting that a certain character lived in Kragen Mar, or only a few certain rug meshes are allowed to be used in interiors in Dunkreath, or that there are not to be any orcs used in all of Narsis for some reason (obviously just hyperbole), then there can be notes made on the respective section or pages for those subjects.

Now that I'm rambling on once more, I was looking at the interactive map for TR posted here:

[url]http://tamrielrebuilt.uesp.net/trmap/[/url]

I noticed that when clicking on a location, it will bring up a link to the respective mod page on the UESP, such as this one for Rhanim:

[url]http://www.uesp.net/wiki/Tes3Mod:Tamriel_Rebuilt/Rhanim[/url]

If there are already placeholder pages for these locations, then why not fill it with placeholder information? Use the UESP as a repository for all information about each place, its inhabitants, unique items, new spells, quests, and so on. A lot of information on it can be outdated quickly, but that's the point, it is something that can be edited relatively quickly and updated as soon as new ideas or plans emerge. If the people running UESP aren't happy with that idea, then something with a similar structure could be utilized in its place. Perhaps that wiki on Redmine?

As for the lack of information and the blank pages, you can hold a "brain dump" week and invite all developers to fill the pages with as much as they know about a subject with information that has been created or decided on so far. Just have them pick a page they would enjoy writing about or happen to know a lot about, and have them go at it. From the broadest region pages, such as Roth Roryn or Velothis District, to the most refined such as a specific river or the NPC section of the page on the Redoran village of Nhalrum. Spend a week or two doing that, just dumping all information possible, and another week organizing it, removing inconsistencies and errors. I know that may sound like a replacement to the planning and faction documents idea (which I guess it could be) but I think it could work in conjunction with them. The planning documents could then be used as more of cohesive picture of everything.

After this mass collection and organization of data, I feel that it will become clear which subjects are more planned than others. It might come to light that say, House Hlaalu are far more fleshed out than people might think and that implementation of their holdings might be possible sooner than realized. It might also possibly show that there is a complete dearth of info and lore regarding Kogotel (or Maar Veloth or whatever its name is now). This can all then be coordinated with Redmine and linked up with claims and progress charts.

Thanks for reading and I hope people give the Redmine and wiki ideas some consideration as I don't believe setting up the basic framework for those will be too much work in the long run and I really don't want to see this project die. I may not be an editor or member at UESP or a developer here, but I'd help with it if I could. Even with my limited time as is.

psi29a · Post by **psi29a** » Thu Dec 17, 2015 9:30 am

Jet133 wrote:In the link to the Redmine demo for TR posted by psi29a, I saw a link to a wiki of sorts, but it gives an error 404. If this is a wiki similar in style to say, the UESP, then that could be one of the greater benefits to planning that could come out of it.

Yeah! I just created the first wiki-page, which is the reason why you got a 404... no had created it yet.

http://www.hostedredmine.com/projects/tamriel-rebuilt/wiki

I think this here is also great for documentation, for example, what works in Morrowind and in OpenMW. Status of particular sub-projects.

There are quite a few useful 'modules' in redmine, including forums. I mean, in theory, we could use redmine's forum ability... not sure how well phpbb's content will port over to redmine, nor how much work that would be either. Just saying what is possible, not that we _have_ to use it. We can stay with phpbb and drupal.

Post by **Seneca37** » Sat Dec 19, 2015 11:27 am

Remember psi29a and Not, that we are just looking for a DEMO. We just need to see how things work with Redmine. Please do not go overboard and create a whole system at this time. Just a claim or 2 for interiors, exteriors and quests should suffice, and whatever else you think would be nice.

I still think that the best approach would be for psi29a and anonytroll to create a custom claims system using Drupal and whatever additional tools we have (php, cms ...).

Also, remember, the claims system is not holding us up at the moment. Getting the planning done for the regions and the factions is.

Post by **Anonytroll** » Sat Dec 19, 2015 9:13 pm

Okay, time for a large effort post.

Up until last week, I was basically flailing around in a vacuum. So, what was I thinking and what was my plan?

The code is broken
I think this is pretty much universally accepted. Zephyr implemented the claims system for phpBB2 back in 2004 or thereabout and it has never been update since. Attempts to upgrade to phpBB3 and retain the claims system were promised and/or attempted twice, but nothing became of it aside from some dead tables in one of the databases.

The system basically runs a couple of claims table in parallel to the thread and post tables and identifies several forums as "claims" forums.
If a post or thread is posted in one of those forums, additional features are unlocked if the thread is stickied: additional input fields (some of which don't work anymore or cause errors), links to the flash map on the old claims browser (long defunct, but still somewhat working), link to a group map (same), links to a parent claim, and the collection of all attachements in the op, now ordered by reverse date.

Occassionally, this process of running two structures side by side bugs out and threads are lost - threads in the claims forums need to be stickied to trigger the code that fills the claims tables and sometimes even the entries in the claims tables disappear afterwards and the thread is effectively rendered inaccessible. It is no longer a claim, but not actually a thread. It is a mystery.

One last attempt at an overhaul was made by Adarnocil a few years ago (same time as when the second attempt to move to phpBB3 happened), but while the main page was moved to Drupal, the forum was a large point of contention and essentially where both attempts fell apart. All people involved eventually disappeared from view.

Since then the entire site pretty much ran on autopilot and blind hope. Since last week, with psi29a around, there's two people who can understand roughly how this system works. There wasn't one when I became webmaster (and a good long while afterwards, as I'm working a full day job and I am not well-versed in modern web development even now). Every solution has to take into account that we are basically alone up here.

That said, bitrot aside...

The workflow is sound
Tamriel Rebuilt has the following workflow:

a claim is spun off from a larger discussion as a seperate thread by a lead developer,
a claim thread gets posted,
a member of a designated user group (interior, exterior) developer, who has previously created a showcase and earned their progress from a plain user, claims that quest by comment (and/or button)
that developer is added as maintainer and uploads versions of files, slowly lurching to completion,
if that developer calls it quits, the claim is put back to unclaimed status (forum) and another developer picks it up,
if the developer is done, n reviewers pick up the claim,
they post successive corrections, or task the developer to finish it,
finally, the claim is finished,
it then sits around until it's merged into a section file,
transitive issues (connecting one claim to another) can only be done in section files, it is absolutely impossible to do those before merging exterior and interior claims,
section files are merged into TR_Mainland or TR_Preview as possible.

Claims are merged as they are finished, it's possible (even likely) that an exterior is merged before an interior is for example.

What makes this workflow (which is not too dissimilar from other software project up until now) peculiar are three things:

claims need other claims (and a master file), they cannot exist conceptually on their own,
claims do not need to be finished in a linear but an onion matter,
claims are somewhere inbetween individual source files and full projects, they are neither, which makes them difficult to place in a traditional PM process.

To elaborate:

Certain types of claims (quests, for the most obvious example) can only be done when exteriors, interiors, NPCs and dialogue have been finalised. They then need additional changes to interiors, exteriors, and NPCs (at least the dialogue database(!) and item placement), which are done in the section file, but reference existing claims as basis.
Additional quests changing the same NPCs need to be developed with existing changes in mind, otherwise they are incompatible.
This makes it necessary to treat claims as more than issues but less than projects - an ongoing process perhaps. They are never truly finished until all development ceases.

The section file is something that cannot be properly implemented into a standard workflow. It's not a full project on its own (it's a compilation of several and needs to be combined with others of its own hierarchical level into one project) but it is required for others, it constantly changes shape and half of the development work takes place in it.

Keep in mind that Project Tamriel uses the same system, without any additional coding. It is, however, comparatively tiny and inactive.

Basing it on forums removes context

The organisation in forums makes it tremendously difficult to categorize an existing claim - while there are forums for the different workflow states, there is no further categorisation, such as regions, group map, or first released version viewable outside of the topic itself, as the claims structure is just a semi-integrated addon to the existing way the forum works.
This is of course not a problem that only partains to claims.

It is however an easy thing to understand for people who make up most of TR's modders - casual users, who like to play games and are enamored with Morrowind. They also post on forums or are at least familiar enough with them to easily understand them.

Tamriel Rebuilt has too much unquantifiable history

Currently, everything is a thread. A thread contains a body, and comments (which are structured the same way). There are also file attachments (those were modded in) and a half-working claims system (dito). The forum has been active and unchanged for so long (styling aside) that it has been a constant in over a decade.

This is excellent for discussions, this is horrible for data retention and categorisation. All the plans are hidden all over the place, decisions were made and then forgotten because the people who made them are no longer here and nobody knows why and how they said things. It's not only the planning ahead that's a big issue, it's turning the data that's already there into a coherent something and retaining the ability to keep it as seperate, categorizable data.

Right now, the interim solutions are two-fold: a series of stickies so that the most important data is clearly visible and coordination via Google Docs.

The merit: structure
Everything in Drupal is a node. A node is also definable as "a content" or a content type. A forum thread is a node, but so is a claim, a page, an article, a book.

Each node has a definable number of fields. Forum threads merely have a title, an url, a body, and an attachment. There are also comments (which are attached to nodes) which can be similarily defined.

By defining both n types of categorisation according to taxonomy vocabularies (which means that those are not just items in your dropdown list, but actual terms that can be interacted with) and being able to reference other nodes directly (although there is still work to do), slowly but surely a network of interconnected claims can be built, where by clicking on an NPC the references to quests, interiors, exteriors that uses it can be found (and vice versa).

In short: thread-based steams of data become a node-based, categorised network. If we need further categories, we can just add them later (albeit with greater difficulty the later it is)! Freely enterable terms such as queues can allow arbitary grouping of nodes and impromptu to-do lists or multi-layered tasks. While this is not as sophisticated as a full PM system, it should make no difference in practice to an amateur.

The claims browser is one - but a very important - part of the whole. It will need to have refined access control (I think I know now how to do this) a refined workflow (implemented), but the fundamental workflow of giving n developers/reviewers access to the node and having them edit and comment is the most basic form of PM and is the part which least needs an overhaul.

I am also right now thinking that defining all types of claims as their own nodes instead of a general "claims" node will give more categorisation possibility, including the ability to distinguish between "linked interior", "linked exterior", "linked NPC", and "linked quest". This will take a moderate amount of time to implement.

Additional views (a collection of parameters and filters to generate lists) can be made available to show something similar to the current claims forums for users who are so used to the forum-based view that it should make little difference to them. This is not yet implemented and will possibly take a while.

This of course does not deal with some of the issues arvisrend and TF have mentioned a while ago - it cannot force discussion to seperate between the claim as a whole and an uploaded file. It's possible to change discussion to claims to a threaded view (like Reddit has), but that's all the crutches that it can offer without making the whole thing a lot more complicated (putting claims files as their own node and injecting them into the claim similar to how the current claims system injects the attached files).

The interim solutions (stickies and Google Docs) can be changed to collaborate editing on given nodes in Drupal (with forced revisions and additional diff and revision capabilities, similar to a wiki page) and a series of books or pages linked in the menu structure or given their own view.

Since this would all refer to the same user- and permission base, it would be a natural growth, usable even if I get run over by a bus tomorrow.

The alternative: workflow
The Drupal-based solutions assumes that the current workflow is fine as is and needs little changing. It should be obvious that I can be very wrong. I've never seen the development process from one end or the other, only as an outsider (I wasn't made developer until earlier this year and have never watched an actual claims process go from beginning to end) and now with a vague idea of how the scripts behind it work.

Unless implemented (and right now I do not think it is necessary or desirable) it does not have a to-do list, or a bugtracker, or a fully-fledged revision system.

But by ditching the way TR assumes claims forward to traditional PM paradigms, changes are introduced into the whole current development cycle. The forum structure finally fully gives out and TR will find out if it was a crutch all along or if it was necessary.

More to the point, Redmine can also act like a forum and a wiki. Going that route and dropping Drupal entirely or mostly would encourage more process-oriented thinking, as the project management is the heart and soul of Redmine (or Git, as it might be, even though it has been discarded as an option), while for Drupal it's managing content.

I'm not sure on Redmine yet.

psi29a · Post by **psi29a** » Sun Dec 20, 2015 8:59 am

Thank you for posting that and getting at least me up to speed.

From a _really_ new guy perspective I see phpBB2 and think of the possible attack vectors that could probably give TR headaches. This eventually needs to dealt with one way or another.

From what I read, the current claims system is 'good enough' as in, it has worked so far. However in upgrading phpBB2 to say phpBB3 we'll be breaking the system and causing additional work to get back to working again.

Is it my understanding that the Drupal system being worked on ties into phpBB? Are we going to have problems when upgrading to phpBB3? If not, then was it the intention on keeping phpBB as 'the forums' and Drupal for 'the rest: website and claims'?

I consider Redmine a raw demonstration. It may turn out that people don't like it, but we can learn from it and move on.

However, from a maintenance and just 'getting shit done' perspective... I rather not reinvent the wheel and get back to doing what is fun.

A case against Drupal would be that someone would have to maintain it. TR has a history of losing its institutionalized knowledge when people vanish, leaving others to hold the bag which is really unfair. This would be less of a problem if the software being used was not home-grown.

Take [Evil_Genius] for example, we use Wordpress, phpBB3 (was 2, we upgraded without problems) and a c-based torrent tracker for file distribution. The workflow was handled inside the forum (finding raw, cleaning raw, translation, type-setting, proof/QC, release) and we used ftp to handle files between each of these steps by moving the file from one directory to the next. When finished, we uploaded to our tracker directory and it was released. http://www.evil-genius.us/

This is just to give another idea how the workflow of others work.

I see a pretty good mapping between the workflow of TR and that can be used via Redmine:
* Regions are our milestones, no deadlines.

* Each Region has sections, the initial 'issues' in redmine parlance. These parent issues then have child (sub)-issues.

* Each sub-issue under a section can be classified as either: interior, exterior, npc, monster or quest (or anything else that is necessary).

* Using role-based-permission we can allow developers to self-assign (or un-assign) claims. Moderators can always assign/unassign claims if necessary.

However we'll have to see if this is workable and feels right during the demo.

Speaking of which, when will the demo begin and how long will it last?

Post by **Anonytroll** » Sun Dec 20, 2015 9:46 am

My intent is to move everything to Drupal and keep the phpBB2 forums as archive, maybe remove anything that allows user input instead of merely deactivating registration and posting.
There's already an indevelopment forum over on the main site, but that's an ongoing process and I kind of stopped working on it after Redmine was brought up, since that can potentially manage everything as well (forums included9.

I'm hestitant to migrate the existing forum data over (which would be possible with a bit of work, since there were phpBB2->D6 migration tools and they would just need some tweaking) because it's so much and that gives too much incentive to just focus on the old forum threads above everything else (and starting fresh). That's certainly an option should they irrevocably break, however.

Drupal would be easier to handle than a mish-mash of other software, I feel (same with Redmine), since the technical expertise to handle several interlinked solutions just isn't there. There should be one configuration and administration center and once you properly understand the basics you can find out the rest.

I'm not sure about how issues will work out in praxis, that's why I'm looking forward to seeing it in action.
This networking of claims is also not implemented yet, so I think we might have roughly equal amounts of work to do in either case.

Beginning the demo will have to wait for next year, since a lot of people are away over the holidays, sorry.

psi29a · Post by **psi29a** » Sun Dec 20, 2015 12:54 pm

Anonytroll wrote:I'm not sure about how issues will work out in praxis, that's why I'm looking forward to seeing it in action.
This networking of claims is also not implemented yet, so I think we might have roughly equal amounts of work to do in either case.

Beginning the demo will have to wait for next year, since a lot of people are away over the holidays, sorry.

The redmine demo is ready for data-input now and I've done relatively little in way of configuration to get it to do what I described above. There are certain things not under my control however with it being hosted for us on a multi-project setup. We should be able to theme it how we like if we hosted it ourselves to make it look integrated into the rest of the site.

I don't mind having a separation of concerns when it comes to software, I would consider phpBB3 to be superior to phpBB2, Drupal and redmine because that is what it was made for.

I'm not ignoring your other valid points however, they stand and I can't argue against them.

psi29a · Post by **psi29a** » Sun Jan 03, 2016 7:46 pm

Ping...

psi29a · Post by **psi29a** » Tue Jan 05, 2016 4:11 pm

To answer myself, I just found out that whatever we enter into the hosted redmine "demo" we can get out to put on our own private instance.

"Free database export for backups and migration into a private Redmine instance.

This exporter allows users of HostedRedmine.com to regularly export their database contents for both backup and migration purposes.
You can therefore use this utility to keep a local copy of your data on your computer.
The exporter format is a zip file containing both the database SQL and all of your uploaded attachments.
You can use these files to import directly into a new private Redmine instance with minimal disruption."
http://www.hostedredmine.com/

I just tested it, it is a zip file of a SQL file. The system seems to be running MySQL5 on FreeBSD.

-- MySQL dump 10.13 Distrib 5.1.73, for portbld-freebsd10.0 (amd64)
--
-- Host: 174.136.100.195 Database: hostedredmine
-- ------------------------------------------------------
-- Server version 5.6.26-log

I'm assuming we're using MySQL for both phpbb and drupal?

Post by **Anonytroll** » Tue Jan 05, 2016 4:27 pm

That's excellent news. We won't have to enter things thrice this way.

Indeed, it's mysql. Different databases however.

psi29a · Post by **psi29a** » Thu Jan 07, 2016 10:11 am

For shits and giggles, I setup redmine on my own system.

http://23.95.113.25/

I'm using apache/passenger and sqlite3 as the backend at the moment. Feel free to create an account.

Anonytroll, please create an account and I'll give you direct admin access.

You can see that we have a LOT of things we can tweak to make it very TR centric.

We thing we should probably discuss is how we plan on developing... it seems there is some disconnect between TR devs as to how to organize development. It could also just be what people find best to wrap their head around development.

For me, I'm ok with whatever works!

Update: I've tweaked the new redmine instance to be more inline with TR.

1) There is one main project Tamriel-Rebuilt, there is were we can keep the master files and have a strategic overview of the whole project.

2) There are sub-projects, these will be the regions (sections) that are going to be worked on. In this case, it will be Old Ebonheart.

3) Creating an issue (new issue) gives you a page that you can fill out. The tracker defines what kind of issue it is with the following options (for now):
Asset, Bug, Exterior, Interior, NPC, Quest.

Subject will be whatever name you give the issue, and description.

The status of the issue is as follows:
New, In Progress, Resolved, Closed, Rejected

There is a priority field that should almost always stay 'normal', unless there is a particular reason to make it more or less urgent.

Assignee is the person who is currently working on the issue.

Target version is which version this issue falls under, such as TR14.12, TR15.12 or TR16.01. Please note that the original idea was to have a region/section as a target, but that was to fit into the hosted redmine's model. We now have more control of things so we'll get version as version which works out for bugs and claims alike.

Claimed field has been added, a simple No (default) Yes list to indicate that it is currently claimed or not. This works with the Status field, so while a claim can be "In Progress" as in, no longer new and has some content, it was unclaimed by the previous assignee or manager.

You can choose files to upload, estimated time to work on and percentage done.

You can also chain issues, by assigning a task a parent task, you are saying that the parent issue depends on the new issue you are creating. This allows us to get a better overall picture what needs to be done for a particular quest which perhaps needs additional NPCs and go to particular Interiors.

I highly suggest you sign up to http://23.95.113.25/ and give it a try. When you create an account an admin will come by to give you access.

Note: the old hosted redmine will be left as-is, but will not be worked on further.

klep · Post by **klep** » Fri Jan 08, 2016 2:11 pm

Unfortunately I won't be able to make the Skype meeting tomorrow, but my vote goes to phpbb3 and redmine sharing the same database backend. For the sake of functionality, efficiency and the ease of operating/administrating.