Author
|
Topic: New patching format, UPS, debuts today (Read 5 times)
|
Karatorian
Guest
|
|
« Reply #75 on: April 03, 2008, 06:56:54 pm » |
|
In theory, if you have X and whatever widget toolkit byuu used (looks like GTK to me) on OS X, you should be able to compile and use the Linux version with few or no modifications. Of course, X is a pretty heavyweight dependancy, and it won't be as nicely integrated as a native port, but it should work.
|
|
|
|
RadioShadow
Guest
|
|
« Reply #76 on: April 04, 2008, 10:42:29 am » |
|
Okay, UPS is very good is your creating patches for big roms. But is it worth using over IPS if the files are below 16MB?
Unless UPS does something better than IPS such as making the Patch smaller, you might as well stick with IPS. Unless I'm over looking something.
|
|
|
|
nomit
Guest
|
|
« Reply #77 on: April 04, 2008, 11:50:20 am » |
|
Okay, UPS is very good is your creating patches for big roms. But is it worth using over IPS if the files are below 16MB?
Unless UPS does something better than IPS such as making the Patch smaller, you might as well stick with IPS. Unless I'm over looking something.
- automatic bi-directional patching. The same patch can both patch and unpatch a game. - CRC32 checksums on the original, modified and patch files guarantees patches will not apply to the incorrect games. We use CRC32 as this is designed to test integrity, not to prevent malicious checksum conflicts. Odds of a false positive are 1:4 billion with CRC32, and CRC32 is 100x easier to implement than eg MD5SUM, etc. Very important for others implementing this algorithm. [...] - Windows / Linux GUI patchers, core library written in ISO C++9x. - all of this is public domain
And besides, why use IPS on smaller files if both would be equal good, it's easier to just have one format.
|
|
|
|
hanhnn
Guest
|
|
« Reply #78 on: April 04, 2008, 11:58:21 am » |
|
why use IPS on smaller files if both would be equal good, it's easier to just have one format. Maybe because there is no emulator support UPS yet ?
|
|
|
|
Deathlike2
Guest
|
|
« Reply #79 on: April 09, 2008, 04:47:40 pm » |
|
byuu, there are a few issues with your patcher that needs to be addressed. I'll PM you about it.
|
|
|
|
DaMarsMan
Guest
|
|
« Reply #80 on: April 09, 2008, 05:49:13 pm » |
|
Just because things support IPS, we should not stick with it. If you really need an IPS you can create one after patching with UPS. Emulators will and should be adapted to work with the new superior format.
|
|
|
|
byuu
Guest
|
|
« Reply #81 on: April 10, 2008, 04:20:30 pm » |
|
I've removed my patcher for now, sorry for the inconvenience. Please see my website here for a detailed explanation. Essentially, there are too many bugs in it. I don't have time to address them all right now, and I'm under way too much stress at home and work to handle the constant bug reports by multiple people. Nach's patcher will be out very soon, it's in beta testing right now. I'll link to it as soon as it is ready.
|
|
|
|
DaMarsMan
Guest
|
|
« Reply #82 on: April 10, 2008, 04:27:11 pm » |
|
Don't worry about it man. We've waited over ten years for the format...what's another few months. =)
|
|
|
|
Darkdata
Guest
|
|
« Reply #83 on: April 10, 2008, 04:59:00 pm » |
|
And we will still be picky then, but hey, that is human nature.
|
|
|
|
byuu
Guest
|
|
« Reply #84 on: April 10, 2008, 05:12:59 pm » |
|
I took a moment to hex compare Nach's patch, I'm starting to see that the problem is more communication than bugs. Don't get me wrong, my patcher has some bugs.
But take the DL patch for instance, our patchers output very different files. Turns out all of those differences are in the extended portion of the file. The top portion, first 2MB of patches and checksums are identical.
Basically, I specified that data after the end of the smaller file should be read in as "0x00". The value was picked semi-arbitrarily, semi-strategically. If a file were padded with mostly 0x00s, then UPS would consider those portions to be identical to the original file, and not output the data blocks.
Nach took a different approach and appended the modified data as-is as soon as you reached the end of the file. This is how NPS worked, and UPS is basically NPS renamed and with a new pointer encode method. So the confusion was only natural.
We're going to work this out the right way, clean up the patcher, and get out a proper release with all of this stuff 100% documented, along with a test suite that can be used to validate new patchers as 100% compliant, so that nobody has this problem in the future. I don't think it'll take two months, probably a week or two.
|
|
« Last Edit: April 10, 2008, 05:21:27 pm by byuu »
|
|
|
|
IIMarckus
Guest
|
|
« Reply #85 on: April 10, 2008, 10:33:16 pm » |
|
byuu, does this mean that your patcher is still usable? Gonna submit that to the main site, or is it okay to link others directly to it?
|
|
« Last Edit: April 10, 2008, 10:42:48 pm by IIMarckus »
|
|
|
|
Deathlike2
Guest
|
|
« Reply #86 on: April 10, 2008, 11:03:04 pm » |
|
byuu, does this mean that your patcher is still usable?
"Usable"? Probably. Would the UPS patch be compatible with the new patchers with the issues fixed? There's a damn good chance it won't. Byuu's current UPS patch of Der Langrisser has a good chance of being replaced with a corrected version when both Nach's and byuu's separate patchers are released. In other words, just hold off because you will be wasting your time. Nach's patcher (same guy who brought you NSRT for SNES ROM checking and header removal/adding) will also include a IPS->UPS converter (for those of you fearing radical change).
|
|
« Last Edit: April 10, 2008, 11:39:04 pm by Deathlike2 »
|
|
|
|
byuu
Guest
|
|
« Reply #87 on: April 11, 2008, 03:01:33 am » |
|
It depends on whether or not Nach and I decide to go with my method of extending files, or his. If we go with mine, then as long as you can apply the patch in both directions, yes it will be fine. I made a UPS patch with Nach's beta and it matches 1:1 on everything except the extended portion of the file.
I'd say wait for the time being, shouldn't be too much longer.
|
|
|
|
dshadoff
Guest
|
|
« Reply #88 on: April 11, 2008, 10:49:37 am » |
|
Byuu (or anybody else who wants to create a new patching format), please don't take this as criticism - but rather a bit of input for the next time a new patch format is produced:
If you want to create a patch format that I'll use *during development* as well as at release time, it should: - contain "replace data" rather than "xor" data - not care (or not necessarily care) about checksums - support >16MB ROMs
IPS does 2 of the above 3. Unfortunately, none of the "new" patch formats covers those same 2 points.
I use a patch utility dozens of times more during a development cycle than I do at release time. If you want to make a patcher that becomes used by the community, I think it's important to consider the development cycle as well.
Let me explain the reasons I need these: I create my patches as a series of smaller patches, each with specific function. For example, one may be graphic, one may be an item menu, etc. I like to keep these separate, so they are pretty much self-documenting. I find that layered patches are much easier to maintain and retain version-control over, than an all-in-one BLOB.
I apply them in layers, so CRC checks would cause a domino crash of failures if even one of the patches changed. I sometimes need to check which version of a subpatch is which, or whether a crucial phrase is already included, so having it at least somewhat human-readable is important.
Most PC Engine games (my favored system) are actually CDROMs, and the data track is >16MB in probably 90% of the cases, so IPS fails here. On the other hand, most existing or completed projects are for the FC/SFC, and most ROMs in those worlds are less than 16MB, so I can see that few people in the creation side of this scene are motivated to move to another format.
I realize that the CRC check is there to prevent some issues which occur in the SFC world, but those same issues aren't so prevalent in the PC Engine world. And it makes UPS worthless to me for use during development. If I'm only going to find a particular patching format useful at release time, then I have no reason to pick one over another. But if I use it during development as well, I am "bought in".
If somebody wanted to make a format that sticks, they should find a way to extend IPS, and perhaps add those extra features as options.
Anyway, something to keep in mind...
|
|
|
|
byuu
Guest
|
|
« Reply #89 on: April 11, 2008, 11:29:09 am » |
|
"Usable"? Probably. Would the UPS patch be compatible with the new patchers with the issues fixed? There's a damn good chance it won't. Okay, this all turned out to be FUD. I was making the patcher fail if the patch was <20 bytes, assuming there would always be at least one patch block. Turns out, just a file resize of all nulls will create an 18 byte patch as the smallest possible, since you don't need to store a change when the extension is 0x00. I can pass all tests I've been provided now. There may be some edge cases that have not been provided to me that I still fail, but it should be simple enough to fix. Basically, too busy worrying about extreme edge cases (though important), that didn't really affect any real-world use scenarios. The DL patch is 100% identical in the old and new versions now, and Nach's outputs the same code up to the file extension. Since he converted the file extension to my syntax, ours are pretty much guaranteed to produce identical output now. The only way a new DL patch will be needed is if we change something up before the next release. When Nach is online again, I'll check with him on the last bits and post a new version. I'm using mmap / MapViewOfFile, now. It makes the patching algorithm twice as small, and now the same code can be used for in-place memory patching (for emulators) as well as for file patching using a simple wrapper. IPS does 2 of the above 3. Unfortunately, none of the "new" patch formats covers those same 2 points. That's because those same two points are the cause of most of the problems with IPS. I get the feeling nobody is reading what I'm writing ... but I'll say it again: With stacked patches, you cannot verify CRCs, because they can be applied in arbitrary order. The best you could do would be to keep a large list of valid outputs CRCs, which is just tacky, error prone, and can never cover every possible scenario. Because of the lack of CRCs, you can never know if you applied a patch successfully or not. This causes countless problems with applying patches to headered vs unheadered ROMs, etc. Further, applying multiple patches is absurdly dangerous. If they both modify the same data, that's very likely to cause problems. And if they do not, then UPS will already work just fine if you simply ignore the bad CRC match. Further, you can chain UPS already (eg A+B=P1 and then (A+B)+C=P2), just not the way you want (A+B=P1, A+C=P2, eg stacking.) You basically want a patching tool to be a development tool, which really isn't what it was designed for. You can write programs in DOS edit binary mode using alt keycodes, but should you? Nach is opposed to adding both unidirectional and bidirectional support to the same file format, and I have to agree. If only because doing so would mean you'd end up with UPS files and have no idea from a glance what they were supposed to do (eg if I apply this patch, will I be able to reverse it?). Plus it complicates the format. But I know you all want to eat your cake and have it, too. I wish you all would have been more vocal the last two years I've been discussing the idea of XOR patching. About the only thing I can say is to come up with another unique file extension, and we can keep that as the unidirectional mode. Unfortunately, we've already chosen UPS for bi-directional, so we can't use BPS+UPS, the current extension stays as is. With a different extension, we can just make minimal changes. Store Y instead of X^Y, and store a length value after the pointer value. Accept CRC32s of 0 as "irrelevant."
|
|
« Last Edit: April 11, 2008, 11:36:37 am by byuu »
|
|
|
|
|