As we all know, Romhacking's primary translation route has been through text files in its entire history.  However, text files don't lend themselves well to data hiding, organization of information, nor tracking changes.  On the account of others such as Carnivol and satsu, I propose that we should move our text storage system to spreadsheets.  The process will work like so: Dump Text -> Format Script (ie, Atlas commands) -> Convert Script to Spreadsheet -> Translate Script Using Spreadsheet -> Revise Script Using Spreadsheet -> Convert Spreadsheet to Script -> Insert Script.  A utility would be developed for the text<->spreadsheet conversion.  However, before such utility can be made, there are a lot of design considerations on the side of the Spreadsheet.

• Which spreadsheet program(s) will be supported?
• What is the feature set for the spreadsheet?  (Word count for a column, % of translation complete, etc)
• If the target spreadsheet doesn't support these features, will an external program provide them?
• Will the spreadsheet enable better translation collaboration?
• Will the spreadsheet have user access control?
• Will the spreadsheet have a way to track revisions?

The goal is to keep things simple.  A standalone program will handle the conversion to and from the spreadsheet.  Either spreadsheet embedded code or the standalone program will implement a set of translation-specific spreadsheet functionality.

I first did an evaluation of MS Excel, since it's the most commonly used spreadsheet program.  There are three methods that I'm aware of that could be used.  The first is actual integration of VBA code into the spreadsheet.  The second is creating an Excel Add-In using C/C++.  The last is creating an external program in C#/.NET and using Excel Automation.

VBA would allow some sort of user access control (since it's embedded within the spreadsheet) whereas the others don't.  It would contain a separate worksheet that contains a UI for useful translation functions.  An Add-In would keep the interface components within Excel, and may allow user access control with a clever solution.  Excel Automation would be a last resort as the functional environment would be outside of Excel.  None of these solutions contain built-in collaboration.

The option most promising to me so far: Google Spreadsheets.  I haven't looked much into it (just discovered it a few days ago), but an owner can upload a spreadsheet (after conversion from text file), share the spreadsheet to other users (collaboration and user access control), edit it online (free, web-based software), tracks revisions, supports multiple sheets per work book, and supports scripting functionality.  This method is also secure from a known translation evil: crashing hard drives.

The Google Spreadsheets option means that I need to create a script conversion program that generates a spreadsheet , possibly a file containing various script code to add to Google Spreadsheets, and the ability to turn it back into a text file.

Opinions or suggestions on this?

I'm not too interested in adding full support for Open Office.  As the initial exported format will be some sort of XML spreadsheet, it will be usable in it.  I don't have any plans in adding the extra functionality for more than just one target app.  You do bring up a good point about web-based.  Some users may have to download a copy from Google Spreadsheets due to a lack of internet where they do their work at.  I'm not sure how cleanly the reuploading of the spreadsheet would be, but it wouldn't be a problem unless several people were working on it at the same time.

I was going to say I didn't really understand the payoff from the seemingly large load of work, but Satsu cleared that up.

He also hit my other point. For this to work as nicely as it sounds, the tools aren't there. You'd need more than just a conversion from text<->spreadsheet. Satsu is using an in-house tool suite. None of us have that luxury. We're left with what? MS Office, Open Office, Google Spreadsheet? First, I'm not sure any of them are great script editing tools. Second, one is costly, one is more obscure but free, one is online. Picking just one of the three to target doesn't seem like it will really make the impact you're trying to make and reach as far.

Next, it complicates the whole process. You're adding an extra conversion utility in the mix. The dumper should output directly to whatever format is used for editing the script. The inserter should be able to then work with that format as well. It doesn't make sense to try complicating the process further to me, unless the reason is so anyone with a dumper that can at least make text files could potentially use the spreadsheet format. But, there's going to be rules for the converter, so if they're not met ahead of time, that's out anyway.

I can see the benefits, but there seems like some big hurdles to really get it to take off. Dumping script to an XML based format to begin with might be the better part of the concept presented here. Of course, it's more difficult to edit that way without a tool to interpret and edit with. Circle back to the tool suite issue.

This leads me back to my thoughts on a movement to try to standardize as many things as possible from the ground up (such as the table file, so we can start developing compatible tools and building a suite a tools to use and not sacrifice freedom and choice of tools. Standards exist everywhere else for these reasons, yet we as a ROM Hacking community largely ignore the issue. Judging by the reaction to patching format standards the last few years, I see little hope for achieving this goal.    But maybe there will be some things the majority can agree on elsewhere. Or, maybe it's the few that can deliver the goods make the decisions and everybody else lives with them.  I'd definitely like to start seeing interchangeable/compatible tool suites being made.

Actually, quite a lot of the work that I've done has been in Excel files. Many developers use generic spreadsheets.

Additionally, what is a great script editing tool? In the context of a game translation, I would say that Excel/spreadsheet is certainly a better script editing tool than Notepad/text file due to what I've discussed above. I did find translation and editing in spreadsheets a bit strange at first, but now it's second nature to me.

If theoretically Klarth were to update Atlas to support dumping to XML/inserting from XML, that'd already be a huge boost. I'm pretty sure most modern spreadsheet programs would be able to handle it.

As far as I know, profesional translation memory software takes a similar approach on data representation to the user. Take a look if you can to DejaVu X, it is the software of choice of one of my professional translator friends and if something half powerful exist in a non commercial license it will be an interesting try to just go directly for that instead of plain spreadsheets.

I'm not sure why you'd ever use text files, except that they are the simplest things - you don't need anything special to open them.  But a spreadsheet is definitely more logical, from the translator's point of view, and I'm pretty sure the dumping program uses arrays/lists or whatever anyways.

Did you consider the CSV table format? Seems the easiest to do, and is pretty much supported everywhere since its literally a text file. It should also be relatively simple to convert a text file to a CSV table.

And then, if someone wants collaboration, version control and user access control, they can use Google Docs, or just edit it with their favorite spreadsheet editing program. Then on top of that you can create excel add-ons, OpenOffice add-ons, google doc add-ons, standalone programs, etc.

If XML would be the direction all this ends up going, I think it would make sense to start talking about the details of XML file standard soon. XML is pretty broad. It would seem what's been established so far is any Atlas compatible text script can be converted to and from this new format. But the details of the new format should be established as soon as Klarth provides a basic example to look at as a basis for discussion. The idea of course would be so future utilities could use or be compatible with this format.

There is no formal definition of CSV files. Hence, Excel exports using ANSI code page, but will be able to import UTF formats as well. You can write a small script that exports it in csv, or just save as "unicode text file" which saves as TSV and should be processable as easy as CSV.

If you think about coining a general xml format for this, maybe start a community poll or something. I know that I would like to include references in there, like one piece of text pointing to the same text, therefore only dumping it the first time and referencing it all other times. I remember some games also call text inside of strings...

cYa,

Tauwasser