Author
|
Topic: New patching format, UPS, debuts today (Read 4 times)
|
Shadowsithe
Guest
|
|
« Reply #105 on: April 12, 2008, 03:30:38 pm » |
|
And my axe support. It's a secret to everyone.
|
|
|
|
byuu
Guest
|
|
« Reply #106 on: April 12, 2008, 05:46:56 pm » |
|
So why do you differentiate between "new files" and "existing files"? Wouldn't it be much more convenient to just have the program see if source = traget and then decide what to do or maybe give a warning with a "continue anyway" option? Two reasons: one, to remove the "bypass checksum" option for direct patching, and two, so that the user doesn't have to pick the same file twice. EDIT: first emulator to support UPS soft patching was released today :D ... yeah, I'm sure you guys were all really surprised that one would be first, right? :P
|
|
« Last Edit: April 13, 2008, 09:36:19 am by byuu »
|
|
|
|
Deathlike2
Guest
|
|
« Reply #107 on: April 13, 2008, 09:57:10 am » |
|
You are shameless byuu. Shameless.
Anyways, my only gripe with your patcher is that when you select any of the individual options (other than Quit), that you have a really huge insta-quit button way too ready to quit. Replace that with a "Back to Main Menu" option and it would not be as annoying.
|
|
|
|
byuu
Guest
|
|
« Reply #108 on: April 13, 2008, 11:01:27 am » |
|
You are shameless byuu. Shameless. And you are vague, as usual. Please elaborate what you mean by this. Anyways, my only gripe with your patcher is that when you select any of the individual options (other than Quit), that you have a really huge insta-quit button way too ready to quit. Replace that with a "Back to Main Menu" option and it would not be as annoying. I like having a quit option on each window (yes, I know there's an X at the top right of the window), but I guess that doesn't hurt much. Alright, then.
|
|
|
|
Deathlike2
Guest
|
|
« Reply #109 on: April 13, 2008, 11:15:00 am » |
|
You are shameless byuu. Shameless. And you are vague, as usual. Please elaborate what you mean by this. EDIT: first emulator to support UPS soft patching was released today ... yeah, I'm sure you guys were all really surprised that one would be first, right? I like having a quit option on each window (yes, I know there's an X at the top right of the window), but I guess that doesn't hurt much. Alright, then. There's the chance that an option is made in error... in which getting back to the starting menu involves quitting immediately and reloading the app is... not UI friendly, ever. It's not like the user has gone 10 menu/screens deep... just one.
|
|
« Last Edit: April 13, 2008, 11:20:32 am by Deathlike2 »
|
|
|
|
Ryusui
Guest
|
|
« Reply #110 on: April 13, 2008, 07:28:45 pm » |
|
OH-EM-FREAKIN'-GEE. For the hell of it, I just tried making a UPS patch for Death Note: The Kira Game. It came out to around 55MB.
Tried again, this time making sure the input and modified files were the same size (the reconstituted WIP ROM is about 52MB, so the first test patch was actually larger than the game itself). It shrunk down to around 45MB, which is still way too large.
Any idea what's going on?
I fully intend to continue to support UPS, but if this is happening for the same reason that UPS is apparently not ideal for PSP translations, I doubt I'll be able to release Death Note: The Kira Game as a UPS patch. Of course, my plans to release Breath of Fire 2 as a UPS patch are unchanged...:3
|
|
|
|
Deathlike2
Guest
|
|
« Reply #111 on: April 13, 2008, 07:59:35 pm » |
|
If the differences between two files are significant, the greater the chance the UPS is bigger than the intended files. There's no compression being used IIRC...
You are going to put this into a compressed file anyways right (question is not directed at byuu)?
|
|
|
|
Ryusui
Guest
|
|
« Reply #112 on: April 13, 2008, 08:37:29 pm » |
|
Still 32MB zipped. No dice. 7z'd, it's 25MB; better but still not really usable. (And yes, I did tweak the settings.)
|
|
|
|
Aerdan
Guest
|
|
« Reply #113 on: April 13, 2008, 08:39:20 pm » |
|
You have to remember, UPS was not designed for use with filesystem'd data, so if any files changed their positions that's going to result in a bigger patch.
|
|
|
|
byuu
Guest
|
|
« Reply #114 on: April 13, 2008, 09:17:31 pm » |
|
Ah, that's a DS game ... never bothered looking into it, but if that system has a filesystem, then yeah, you don't want UPS. UPS is designed for pure ROM patching of a few bytes, eg NES, SNES, Master System, PC Engine, N64, GBC, GBA, etc.
If you insert a byte in the middle of a file (rebuilding a filesystem will almost definitely do this), every single byte after will appear to UPS as though it were modified, and you'll end up with an astronomically huge patch.
And some fun math: uncompressed, a UPS patch in the absolute worst case scenario (every other byte different) will become 1.5x bigger than the largest file, but compressed it will always be smaller than the largest file. Still not good in this case. You want something like xdelta or bsdiff for this one, sorry.
|
|
|
|
Ryusui
Guest
|
|
« Reply #115 on: April 13, 2008, 10:09:59 pm » |
|
Thanks for the recommendations.
EDIT: I'm probably a flaming moron, but come to think of it, doesn't it seem like a bit of an oversight? I mean, the reason why UPS is needed is because now we have games that are way over 16MB in size, but most of them - DS, PSP, myriads of other platform ISOs - contain filesystems that can be unpacked and shuffled around, resulting in gigantic patch file sizes. And seeing the numerous benefits of doing so (and occasional necessity) for translation and the like, I suspect this issue is going to be a tremendous turn-off to those with ambitions beyond the GBA. I'm probably missing something obvious, though...
In any case, I'm back from perusing xdelta and bsdiff - how the hell does one use these things? I don't mind command-line apps (and, frankly, I laugh at anyone who does), but where the hell's the documentation? Running xdelta with no arguments produces a cryptic error message rather than helpful instructions, and bsdiff seems to need some program called bzip2 "somewhere in my PATH", which I'm assuming means "plop it down in the same directory", but I somehow doubt I'm that lucky. These programs are downright user-hostile.
|
|
« Last Edit: April 15, 2008, 03:00:11 am by Ryusui »
|
|
|
|
BRPXQZME
Guest
|
|
« Reply #116 on: April 15, 2008, 07:08:12 am » |
|
That’s because they’re all Unixy and expect you to read the man pages: xdelta(1)bsdiff(1)Not really user unfriendly so much as shifting paradigms without a clutch.
|
|
|
|
Aerdan
Guest
|
|
« Reply #117 on: April 15, 2008, 07:52:22 am » |
|
Ryusui: Use 'echo %PATH%' in cmd.exe; that's the list of directories the shell will look in for executable code.
|
|
|
|
DaMarsMan
Guest
|
|
« Reply #118 on: April 15, 2008, 08:21:18 am » |
|
Thanks for the recommendations.
EDIT: I'm probably a flaming moron, but come to think of it, doesn't it seem like a bit of an oversight? I mean, the reason why UPS is needed is because now we have games that are way over 16MB in size, but most of them - DS, PSP, myriads of other platform ISOs - contain filesystems that can be unpacked and shuffled around, resulting in gigantic patch file sizes. And seeing the numerous benefits of doing so (and occasional necessity) for translation and the like, I suspect this issue is going to be a tremendous turn-off to those with ambitions beyond the GBA. I'm probably missing something obvious, though...
In any case, I'm back from perusing xdelta and bsdiff - how the hell does one use these things? I don't mind command-line apps (and, frankly, I laugh at anyone who does), but where the hell's the documentation? Running xdelta with no arguments produces a cryptic error message rather than helpful instructions, and bsdiff seems to need some program called bzip2 "somewhere in my PATH", which I'm assuming means "plop it down in the same directory", but I somehow doubt I'm that lucky. These programs are downright user-hostile.
This is why you need to create custom prepatch program that will prepare your file for patching. That's what we are doing with DQ5. It moves your files in the filesystem and expands them. That way you make the patch against that.
|
|
|
|
Karatorian
Guest
|
|
« Reply #119 on: April 15, 2008, 04:20:13 pm » |
|
This talk of filesystems, rom expansions, and custom prepatch tools got me thinking. What if, rather than every project that needs advanced patch capabilities writing a custom patcher or using an obscure avanced patcher, we, the ROM hacking community, developed a standard format for pre-patching. According to byuu, the reason UPS doesn't have any such advanced capabilites is to keep the format simple enough that anyone with programming ability can understand the spec and impliment a diff or patch util easily. I'm thinking that with a simple spec, we could achive at least half of that goal (the patching part) for advanced patching. What I have in mind is pretty simple. Basically the format would have three commands, add, delete, and move. Each would specify a starting offset and a size, perhaps in a similar format to UPS. Additions would simply add zeros to the file. I choose zero becaus that is what UPS uses when patching to a larger file size, so it's consistant. Delete would simply remove the specified number of bytes. Move is a bit more complicated. Conceptually, it's like a delete at the source location and an add at the destination location (except that the added bytes aren't zeros). But while it's simple in concept, it's complicated in practice. The basic problem is that a move requires editing the file at a point other than the "current" one. This leads to issues with addresses changing on you. I'm not entirely sure how to handle this, but with all the brilliant people here I'm we could work it out. With such a simple spec, implimenting a patcher would be pretty easy. Other than the move command, it's dead simple. Furthermore, if the spec had the exact symantics of the move command clearly specified, that wouldn't be very hard either. Implimenting an automated diff generation program for such a format would be more complicated. Finding insertions and deletions is a rather complex problem known to computer science as the Longest Common Subsequence. Detecting moves is even more complex. However, dispite the complications in the diff side of implimenting such a tool, I still think it could have great advantages. For instance, anyone implimenting a custom patcher or pre-patch utility could instead use the format. Unless such a project has actually written an advanced delta detection tool of their own (unlikely), they're going to have to hardcode any such changes. Given the data required to write such a tool, they could simply impliment this proposed format instead. Additionally, while move generation in a tool is a complex task. It can be dispenced with at the cost of some effiency. In this case, a tool need only solve the LCS problem, which is well documented. In this case, any moves are instead simply an insertion and a deletion. This produces a larger patch, but would be fully compatible. I'm not that familiar with Ninja Patches or the upcoming Ninja 3 project, but I wonder: would a Ninja 3 container would be a suitable format for packing such a prepatch format and a UPS patch together? Anyway, what do you think of this idea?
|
|
|
|
|