My apologies for chiming in now, but if it is stored like a compressed video stream, than it would not be the first time Konami games have messed with people. This seems to be a near-daily occurrence for them; Kitsune hates them, and we had a fellow working on something else a while back, the credits were stored differently (and I mean completely differently) than anything else. Do not let them fool you one bit =p

Some experimental results just in. Firstly I changed the very first byte that I expected served as the first literal, at address 0x698 in Nanami's file. I figured that if the compression algorithm was using a dynamic dictionary, it would add the 0x00 in as the first dictionary value, but now I'm not sure:


Basically changed that very first value to a couple test values, and found that the exact test value made no difference; 0x02 and 0xDD in place of the 0x00 both seemed to pull in an adjacent chunk of VRAM...!? Could this be some kind of reaction the game engine just does when it doesn't know what's going on? At first I thought it might be a command to locate and pull tiles from within VRAM, but it looks like the game is just reacting allergically to an alteration of the first byte. If I switch Nanami around in this experiment, she'll always pull an adjacent section of VRAM just like this.


The second experiment, which you suggested Tauwasser, seemed to have more promising results as far as revealing some secrets:


Looks like 0xFF might be a dictionary value standing for a run of 0x00s, and when that's changed, it throws off the rest of the sheet for which that dictionary value serves as the first building block. The major thing I'm getting out of this is that the affected section of VRAM is indeed the first uncompressed sprite sheet, so we know the storage method finally! :beer:  Even ONESHEET was too wide. The reason why the the change didn't propagate further is probably that there are separations between the first four compressed runs -- in fact, that 0x00 byte I changed could be a terminator instruction, because I remember seeing that between the compressed sprite sheet files. Now that I know how the sprites are stored I can finally, finally get to work on determining dictionary values with confidence that I'm looking at the proper decompressed output.

If you're getting anything else out of these experimental results Tauwasser, let me know. I'm not sure whether the second experiment conforms to your theoretical results, for example.

Also, I've been leaning in the direction of dictionary compression now, but I'm wondering if the dictionary would be static and stored somewhere or dynamic and built by the algorithm during the compression phase? It seems to me that a dynamic dictionary built on-the-fly would have to start with a literal, and this obviously isn't unless I'm misinterpreting the first experiment.

In any case, now that I know how the output is segmented I can give the sliding window compression possibility one last shot with some comparisons. If it's not sliding window, dictionary would be the logical alternative I guess, but I'm really just basing that on David Holmes' observation on page 1.

Update: I think another opcode is identifiable: 0x01 plus a multiple of four writes (Byte Value - 1)/4 bytes. For example, the value 0x7D writes (0x7D - 1)/4 = 1F bytes! This accounts for why 1F bytes were taken out of the uncompressed stream in one of the experiments when 0x7D was zero'd, and why 19 bytes were taken out of the uncompressed stream when 0x65 [(0x65 - 1)/4 = 19] was zero'd.

The opcode is apparently in the format xx yy, where xx is the aforementioned multiple of four-plus-one and yy is an address-based director that tells xx where to start ripping its bytes from. Experiments suggest the following:

7D 8A @ 6BB writes 1F bytes from 0x1AC.
69 90 @ 6BD writes 1A bytes from 0x1B5.
71 8E @ 6C3 writes 1C bytes from 0x1B0.

I thought at first that it would make the most sense for yy to be anchored at either the next highest or the next lowest 256-byte boundary in the decompressed output, but now I'm not sure. That would hold for the first two examples the change between 8A and 90 equals the change between 1AC and 1B5, but the third example throws that trend off. I might be making a mistake in my source address observations, so I need to do more experiments to make sure I'm absolutely right on the 0x1AC, 0x1B5 and 0x1B0 observations. Note that these addresses could actually be 0x2C; 0x35; and 0x30 because I just found out that I may have included an extra 0x180 bytes' worth of 00s at the beginning of the 128x128 pixel sprite sheet.

Each run seems to have an exogenously determined minimum length, which seems to interrupt the multiple-of-four runs. For the bright green and sky blue runs the minimum length is 6 bytes, but for the pale green run the minimum length is 5 bytes. Zero them all out and you'll get the minimum length.

The multiple-of-four-plus-one opcode is confirmed to work for all values up to 0x7D, but it obviously cuts out at a certain value since 0xC1 is also a multiple of four plus 1, and it does something very different. (EDIT: Actually, I've found at least one exception so far. Rats.)

EDIT: Actually, it's also possible the yy references for the model opcode in question are relative to the start or end of the previous uncompressed run or otherwise something that changes from one run to the next. That might explain the change discrepancy between the first & second examples and the third.

---

In any case, using the 128x128 pixel sheet that got screwed up during the second experiment definitely eliminates the odd jumps we were seeing before (so 0x34 isn't a jump command or anything). In other words, the 0xC1 commands are all following in order now when compared to the output. There's a ton of literals strewn throughout though, in addition to the ones generated by the 0xC1 commands.

If anyone's interested in checking out patterns with me, these files should provide more accurate comparisons than what we've been doing until now. I should have done an experiment like that to determine how the sprite sheets were stored a long time ago, agh. The spreadsheet contains color-coded run matches and literal strings for a portion of the compressed and uncompressed files. I thought at first that it would be easy to identify library entries marked 0x02 and 0x20 since they appear in a lot of compressed runs, but now I'm thinking that they're just literals too.

One thing that's interesting is that the compressed file gets into an "add 0x22 to the next entry" at the very beginning. Values like 0x7C21, 0x7C43, 0x7C65, etc., one after another. Not sure what it means yet.

---

Aha! New experimental results in:


This is what happened to the data when I changed the 0x7D @0x6B4 in Nan 120 to a 0x00 value. The green section is shortened by 0x1F bytes in total (meaning the 0x7D was responsible for writing either 0x1F or 0x20 bytes depending on whether the 00 in its place now is treated as a literal). More importantly, it's now become clear what the compressed runs are doing. It doesn't look like a dictionary after all -- all the compressed runs just seem to be acting on past data with the exception of the 0x2E 0x1E compressed run in yellow, which is actively writing 00 bytes or else it would have propagated the errors introduced previously. Note that the errors do get propagated throughout the sprite sheet, so runs further on must be reaching back into the garbled up runs seen in the example.

I guess this rules out dictionary-based compression, or at least reduces the range of values in the compressed runs that could be associated with dicionary-based compression. It seems more like some version of LZSS from hell now.

---

More experimental results in. The first compressed green run shows the nature of each experiment (just zeroing out certain bytes in that compressed run):




Fascinatingly, there seems to be no purpose to the second and fourth bytes in that first compressed run, or at least they have the effect of producing one zero each. Also, zeroing the second and fourth bytes one at a time had absolutely no effect on the file. Also, it seems the compressed green run can produce six zeros with just four...somehow.

EDIT: Updated the experimental results with an address grid of the decompressed output. The algorithm can't be copying previous runs relationally, otherwise the 0xD0 21 bytes wouldn't be corrupting the sky blue uncompressed run in some results but not others. Whether the 0xD0 21 is copied into the sky blue uncompressed run depends completely on the raw address at which 0xD0 21 appears, it would seem...

Never seen anything like this before, that's for sure.  

Maybe this would be a good time to start accumulating RAM address functions for Suikoden II, and make a sort of Memory Map.

Is 8010E8C8 a RAM address that specifies which character takes the last place in the inactive party menu, or how is this working exactly? Did the forced change from a value o 0x52 to 0x53 cause the game to overwrite another character with Tir McDohl?


I'm obviously behind on figuring out the graphics decompression routine, but I plan to keep working on it. It seems to be analogous to LZSS and I think I've got one of the most important opcodes partially figured out as explained in my previous post (Tauwasser figured out the opcodes for literal strings too, so that's two down already), but there's still some kinks that need to be figured out through sheer experimentation.

I guess part of me is also waiting to see if Konami's "unnamed PS3 RPG" is a main series Suikoden, since a continuation of the main series might kill interest in modding the already-existing games.

Might be a tad late, but you didn't happen to come across the script while you were tinkering around?

Interesting. I did like you said and I've managed to locate the script to the VA0X.bin files. Thanks!

It's pretty garbled though but at least I know where it is.

Too bad my attempts at changing the script around a bit just caused the game to crash though.

EDIT: Aha... turns out UltraISO just wasn't up to the job.

Using CDmage instead I was able to get the game to change Nanami to Jackie in the beginning.

EDIT: Working out better than I expected. Now I just need to figure out how to put in names that are longer than the original without crashing the game 

Well, as far as I can tell it's impossible for me to recompile the game after fixing the scripts. All methods I tried result in an endless black screen.

Not to say it's impossible, just that I don't have the skills needed.