Author
|
Topic: What are your current gripes with hex editors? (Read 2 times)
|
dshadoff
Guest
|
|
« Reply #15 on: June 12, 2008, 09:01:12 pm » |
|
Also, it would be really neat if the editor had a bookmarking feature with a sidebar (ala Acrobat Reader) that allowed for bookmarks to be imported/exported, thus allowing people to document the ROM within the bookmark file. Then people who were interested in hacking a ROM could load up a bookmark file to find the values they're wanting to hack. People could even work on community bookmark files, adding in information as it is discovered. If it caught on, I could even see a separate section on RHDN just dedicated to such bookmarks.
It would also be nice to have a button within the hex editor that would automatically launch the currently open file in an emulator for testing.
-TEEHEE! I like your interest in bookmarks. I also like that you're headed in a similar direction to what I envisioned with them. It's part of the idea behind this thing. -Oh yeah, command-line execution of files would be a big help. I had toyed around with the idea around this; I was going to write an Eclipse plug-in, but that turned out to be too much research, and with an interface which was likely to change too. But my idea of bookmarks was to document *how to display* various portions of the ROM. Let me elaborate a bit: For example, 0000-0FFF is a jump table (2-byte pointers); 1000-3FFF is 65C02 ASM; 4000-4FFF is graphics, etc. This would sort of be a descriptor file, and the main hexedit display would obey its configuration and show data in the relevant format, line by line. As more is learned about the ROM, the descriptor could become more and more elaborate (perhaps a wiki could exist for each game which tracks these updates) - but most likely, only the interesting areas of a ROM will ever really be located and described like this. Such descriptions could even include commenting the assembly (disassembly) of a key function, although that may be a low-priority feature. Next, I felt that modifications to the base ROM should not directly overwrite the ROM itself, but rather, would be held as a revision to the base rom, and the editor could output a new ROM revision (based on the base ROM). Further, these revisions (patch descriptions) would be in layers such as: (1) hack to print function, (2) monster names, (3) text in area 1, etc. These patches would be loaded together, but held separately, say in tabbed sections, which could also contain comments. Each layer would show the same view as the base ROM, but with its revisions highlighted and indexable (toolbar would include 'jump to next/previous revision'). When you choose to create a patched output, you could configure which patches to apply (or more likely, which ones to disable for this ROM output). And moreover, if these "bookmarks" were to written to disk, I would like to see them written as a text file of some sort, so that a developer could store it in a revision control system such as CVS or SVN, and track the history of his changes in case something causes an undesired effect. Back on topic: For the record, I use UltraEdit-32 (paid) in Japanese locale (via Microsoft's Applocale). Good editor, but the hex edit is a bit short of features. But it does notice if the file changes due to outside influence. And thankfully, it can handle files which are 100's of megabytes in size, like some of the CDROM data tracks I work with. For relative search, I write my own custom code (which is a bit of a pain). And I use Mednafen for debugging emulator. - Dave
|
|
|
|
Disch
Guest
|
|
« Reply #16 on: June 12, 2008, 10:49:42 pm » |
|
I thought about making a hex editor in the past as well. I had a big list of features I was going to include but I forgot most of them. Anyway I never got anywhere. I can think of 2 features... one of which isn't really a feature though:
not really a feature) Handle large files gracefully. This was actually the biggest obstacle with my project attempt. I ended up splitting files into 'blocks' and only loading blocks as needed rather than loading the whole file at startup. Blocks were stored in a tree type structure.
a feature) offset translations with a customizable string. You could have some defaults for common types like SNES HiROM/LoROM. Would be something like...
"{2:X>>16}:{4:X&0xFFFF}" <-- an example format string for HiROM. Would translate offset 0x12345 to "01:2345"
Would be more useful for other systems, like NES where CPU addresses and offsets often differ greatly
|
|
|
|
Tauwasser
Guest
|
|
« Reply #17 on: June 13, 2008, 12:50:33 am » |
|
File change notification: Should the program keep a watch on the file and ask if you want to refresh on changes, like Visual Studio, or simply wait until saving to notify the user? I'm guessing a nice feature would be a temporary save + binary comparison here, although it'd be tricky to implement. Can't really think of another way to "merge" files.
Another feature I just remembered that would really had saved the ass of a ton of us in the past...
When saving a file, check for edits since the file was opened and warn the user about it. I can't count how often I accidentally killed changes I made due to a hex editor that had the file opened for hours while I worked on the file with other programs in the meanwhile...
HexWorkshop does that already. All the other things mentioned except for address translations are also already done, so that`s not a big deal, I think. Also, mostly SnesEdit does already what everybody here is looking for, so...err...yeah :| Also, you don`t have to set up the character map in hex workshop, just go to the folder and do it there in a new file and it will work... cYa, Tauwasser
|
|
|
|
Lleu
Guest
|
|
« Reply #18 on: June 13, 2008, 01:25:26 am » |
|
I notice some people said cross-platform, and while I'm going to chime in with them, I also think you should be open-source. That way other people can contribute if they want to.
Current plans are to provide source once the code is written and documented to my personal satisfaction. I'm working in unfamiliar territory with some of the things I'm trying to implement, and from time to time I'm likely to experiment. Crappy hacks may enter into the picture, and I know better than to release such into the internet jungle. People read that stuff. Plugin architecture is planned from the beginning (working on the interfaces right now), so as long as this base gets written properly the project won't be outdated, regardless of what happens to me. I had toyed around with the idea around this; I was going to write an Eclipse plug-in, but that turned out to be too much research, and with an interface which was likely to change too. But my idea of bookmarks was to document *how to display* various portions of the ROM. Let me elaborate a bit:
For example, 0000-0FFF is a jump table (2-byte pointers); 1000-3FFF is 65C02 ASM; 4000-4FFF is graphics, etc. This would sort of be a descriptor file, and the main hexedit display would obey its configuration and show data in the relevant format, line by line. As more is learned about the ROM, the descriptor could become more and more elaborate (perhaps a wiki could exist for each game which tracks these updates) - but most likely, only the interesting areas of a ROM will ever really be located and described like this. Such descriptions could even include commenting the assembly (disassembly) of a key function, although that may be a low-priority feature.
(Emphasis added) Once again, TEEHEE! Documenting the file is another key concept of what I'm working on, and the primary reason that I'm taking a plugin approach to this utility. So I agree wholeheartedly with that concept. I've had some good ideas for it, but any disassembly functionality is way after I get the initial bases done. I don't want to bite off more than I can chew. I also considered extending an existing hex editor, but then I tried to compile it, failed, looked at the source, and shuddered. And here I am. I'm liking that a lot of other people have these ideas, too. Next, I felt that modifications to the base ROM should not directly overwrite the ROM itself, but rather, would be held as a revision to the base rom, and the editor could output a new ROM revision (based on the base ROM).
Further, these revisions (patch descriptions) would be in layers such as: (1) hack to print function, (2) monster names, (3) text in area 1, etc. These patches would be loaded together, but held separately, say in tabbed sections, which could also contain comments. Each layer would show the same view as the base ROM, but with its revisions highlighted and indexable (toolbar would include 'jump to next/previous revision'). When you choose to create a patched output, you could configure which patches to apply (or more likely, which ones to disable for this ROM output).
And moreover, if these "bookmarks" were to written to disk, I would like to see them written as a text file of some sort, so that a developer could store it in a revision control system such as CVS or SVN, and track the history of his changes in case something causes an undesired effect.
Back on topic: For the record, I use UltraEdit-32 (paid) in Japanese locale (via Microsoft's Applocale). Good editor, but the hex edit is a bit short of features. But it does notice if the file changes due to outside influence. And thankfully, it can handle files which are 100's of megabytes in size, like some of the CDROM data tracks I work with.
For relative search, I write my own custom code (which is a bit of a pain). And I use Mednafen for debugging emulator.
- Dave
-I have to say, I hadn't considered using a Photoshop "layer"-style approach to the hex application. Although I've been considering change tracking for a while, loading and displaying all of that dynamically seemed that it would take an awful amount of resources. I know a lot of people work on older machines, so I'm trying to keep resource usage low where I can. And while I understand that we don't have to code for 640K anymore, I'm trying to reserve resources where possible for some future features and multiple views. It's an interesting idea, for sure, but it could be troublesome once the project grows in both hours and size. Diff files would be in a text format containing binary blobs, if/when implemented. Maybe UPS support, if it meets my needs. -Also noting a vote for large file support. I thought about making a hex editor in the past as well. I had a big list of features I was going to include but I forgot most of them. Anyway I never got anywhere. I can think of 2 features... one of which isn't really a feature though:
not really a feature) Handle large files gracefully. This was actually the biggest obstacle with my project attempt. I ended up splitting files into 'blocks' and only loading blocks as needed rather than loading the whole file at startup. Blocks were stored in a tree type structure.
a feature) offset translations with a customizable string. You could have some defaults for common types like SNES HiROM/LoROM. Would be something like...
"{2:X>>16}:{4:X&0xFFFF}" <-- an example format string for HiROM. Would translate offset 0x12345 to "01:2345"
Would be more useful for other systems, like NES where CPU addresses and offsets often differ greatly
-large file handling is indeed important. And I know that it's easy to implement wrong. I'm not sure how I'm going to resolve it, but I do see the need. -Please clarify this feature a bit more for me. I have absolutely no idea what this is intended to do. File change notification: Should the program keep a watch on the file and ask if you want to refresh on changes, like Visual Studio, or simply wait until saving to notify the user? I'm guessing a nice feature would be a temporary save + binary comparison here, although it'd be tricky to implement. Can't really think of another way to "merge" files.
Another feature I just remembered that would really had saved the ass of a ton of us in the past...
When saving a file, check for edits since the file was opened and warn the user about it. I can't count how often I accidentally killed changes I made due to a hex editor that had the file opened for hours while I worked on the file with other programs in the meanwhile...
HexWorkshop does that already. All the other things mentioned except for address translations are also already done, so that`s not a big deal, I think. Also, mostly SnesEdit does already what everybody here is looking for, so...err...yeah :| Also, you don`t have to set up the character map in hex workshop, just go to the folder and do it there in a new file and it will work... I know I might step on some toes by saying this (sorry), but I can't work with console hex editors like that. The colors always seem grating, most of the time the windows are modal, etc. I'm not saying that they're bad. Just that I can't get any serious work done in a console hex editor. I blame myself. I'll roll around in the prompt all day, but for hexing I prefer a more modern interface. But thank you for pointing it out to me.
|
|
|
|
Tauwasser
Guest
|
|
« Reply #19 on: June 13, 2008, 04:43:21 am » |
|
-large file handling is indeed important. And I know that it's easy to implement wrong. I'm not sure how I'm going to resolve it, but I do see the need. -Please clarify this feature a bit more for me. I have absolutely no idea what this is intended to do. He`s saying that there should be address translation depending on the different roms. E.g. SNES roms have many addressing modes. Because oevery bank in SNES games has 0x8000 bytes, Bank 0 is noted as follows from Byt 0x000000 to 0x007FFF in the rom as 00:8000 to 00:FFFF. It`s a programming perspective thing, because that`s how the cpu is addressing the file. Similarly, for gameboy it would be Bytes 0x000000 to 0x003FFF (0x4000 byte banks) adressed as 00:0000 to 00:3FFF whereas offsets 0x004000 thru 0x007FFF are 01:4000 to 01:7FFF etc... Oh yeah, he`s proposing a single line system that would be readable by the program and then interpreted for address translation. I don`t think this is a good approachh as for different hardware (SNES vs. GB/C for instance), the addressing is not always quite the same. In GB the 0th bank is constantly available at 00:0000 to 00:3FFF while all other banks are at XX:4000 thru XX:7FFF [XX>00]...hence a problem :-/ cYa, Tauwasser
|
|
|
|
KaioShin
Guest
|
|
« Reply #20 on: June 13, 2008, 05:02:50 am » |
|
Tauwasser: Yes, if you look hard enough you can find all features in this or that program. From what I understood, the point of this editor was to get it all together into one program that works, is stable and moreover (this is a HUGE plus over an unbearable console editor like Snesedit) to have a strong interface that makes it comfortable to work with. And that's something really needed and that I'll support 100%. Lleu: - Relative Searching: Inverse pretty much nailed all the problems and important aspects. Many relative searchers were written by people who translate from English to other languages, so they all work only with the latin alphabet. They just search using words only and not values, so you can't search comfortably for Japanese.
- Table Files: It's very easy to support a basic table file, but it gets more tricky when you dive deeper into it.
A table file can encode one byte entries, but also two or even more byte entries, all in the same file. For encoding the file, SJIS or Unicode should be possible, as well as ASCII of course. And yeah, searching for data should be possible using the values from the table file.
- Inserting Raw Data:
I meant a data file. Just a binary file, shouldn't be hard to implement. - Corrupting:
Corrupting is changing an area of data in a ROM to systematically find where specific data is stored. Let's stay you want to find where the level data is stored in a jump and run game, but you have no idea where or how the data looks like. With corrupting, you can just change areas of the ROM to all zeroes or anything, until you find the data. Depending which data you overwrite the game might crash, the sound might stop, some graphics may be messed up or... the level layout might have been messed up. There, you just found it's location
- Byte fill: Should be possible at any given location. Corrupting is the same, just with different intentions. Changing the file size will automatically fill the new space with some data of course. It's just needed to expand ROMs. Should be simple.
- Searching: I don't know about the interface details yet, haven't thought about it. Sorry if I just ask for things without a clue how to implement them best
- File change notification: It would be kind of cool if it would notice it immediately. It should then ask if the file should be reloaded or not.
HxD has another method to circumvent accidentally changing the data with other programs... it locks the file completely. You can't open the file with anything else as long as you have it opened in HxD. Please don't follow that route... it's majorly annoying. [/list]
|
|
|
|
Disch
Guest
|
|
« Reply #21 on: June 13, 2008, 09:31:25 am » |
|
Oh yeah, he`s proposing a single line system that would be readable by the program and then interpreted for address translation. I don`t think this is a good approachh as for different hardware (SNES vs. GB/C for instance), the addressing is not always quite the same. In GB the 0th bank is constantly available at 00:0000 to 00:3FFF while all other banks are at XX:4000 thru XX:7FFF [XX>00]...hence a problem :-/
It wouldn't have to be simple -- I just used a simple formula as an example. That GB format you described could be handled with something like: {2:X>>14}:{4:(X&0x3FFF) + (X>0x4000) ? 0x4000 : 0} Which would use address 0000-3FFF for the first bank, then 4000-7FFF for every bank thereafter. @ Lleu: Basically the idea is instead of printing basic file offsets, you "translate" the offset into another number based on a given formula. This way the hacker doesn't need to do any offset<->CPU address conversions like for pointers or anything... since the CPU addresses will all be printed in the hex editor rather than the file offset. It would also help when working in a hex editor and debugger in tandem -- since debuggers and tracers generally don't print raw file offsets. What a traditional hex editor would look like: 00007FE0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00007FF0 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 00008000 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02
What a translated offset display (SNES LoROM, in this example... hopefully my recollection of it isn't all wrong) would look like: 00:FFE0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:FFF0 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01:8000 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02
The idea is the format string is customizable so that the user can write one for whatever game/system they're working on.
|
|
|
|
Neil
Guest
|
|
« Reply #22 on: June 13, 2008, 09:46:01 am » |
|
I'll admit it, I only skimmed the thread, some of these might be duplicates. - Adjustable byte spacing. It is occasionally helpful to display data in two byte or three byte or four byte clusters for data analysis
- Display binary conversion for selected bytes in the status bar at the bottom of the screen, or somewhere else easily visible. I am occasionally too stupid to do the conversion in my head.
- Second only to table support, searching. Like everything said above and more. Wildcards, hex search, text search via table, etcetcetc.
|
|
|
|
seirj
Guest
|
|
« Reply #23 on: June 13, 2008, 09:53:27 am » |
|
I just remembered something I'd really, really like to see in a hex editor. I want to be able to color code chunks of data and then zoom out really far so that the whole file is on screen and I can see and get my bearings on where data is and how big it is compared to the entire file. If I could do this, I could see what areas that I know what are and can ignore and what areas I don't know what are and can search in. I hope this makes sense.
|
|
|
|
Tauwasser
Guest
|
|
« Reply #24 on: June 13, 2008, 09:54:23 am » |
|
{2:X>>14}:{4:(X&0x3FFF) + (X>0x4000) ? 0x4000 : 0}
Ah! I see it now! It took a while though... is this standard regular expressions? Somehow I doubt this, but I was never really familiar with that either... Also,... if this feature were to be incorporated, I would advise to have the option of showing both, the rom file offset and the cpu address at the same time as well as both alone. Also, in each mode (if you work with a address translation, regardless of what "offset view style" you are in), it should feature a boundary check (that is configurable turn off/on in the options - not annoying that way), so people don`t put stuff across banks when they don`t think for a minute or two. @KaioShin: There is a windows version of snes edit, though it feels like the command line it is... I don`t use it much/like it either, however it has many features, regardless of how you look at it (psx tim viewer and other gfx stuff, too!). Also, for me Hex Workshop has never ever failed to function properly... It only ever crashed when I accidentially opened 20 GB and compared files or something... which I had created on the fly for...er...some unbeknownst purpose lol I`m not against this or anything, I`m just saying where some good features are, so one could kinda copy from those hex editors cYa, Tauwasser
|
|
|
|
DarthNemesis
Guest
|
|
« Reply #25 on: June 13, 2008, 09:55:49 am » |
|
Also, it would be really neat if the editor had a bookmarking feature with a sidebar (ala Acrobat Reader) that allowed for bookmarks to be imported/exported, thus allowing people to document the ROM within the bookmark file. Then people who were interested in hacking a ROM could load up a bookmark file to find the values they're wanting to hack. People could even work on community bookmark files, adding in information as it is discovered. If it caught on, I could even see a separate section on RHDN just dedicated to such bookmarks.
You know, I've had a similar idea rolling around in the back of my head, but for use when viewing/editing the memory inside the emulator. I love trying to pick apart how the game stores its data, so something like this would be a great step in that direction. :thumbsup:
|
|
|
|
ReBirFh
Guest
|
|
« Reply #26 on: June 13, 2008, 10:53:17 am » |
|
I like the idea of color coding chunks of data.
e.g: I have three chunks of data labeled "Text1", "Text2", "Text3" with the color "Blue".
The Hexeditor would need the ability to: -Move to (Place that chunk of data beginning in a specific offset). -Dump/Extract (Dump the chunks of data from a specific color in separate files with their labels as file names) -Insert from
And in a dream...
Your hexeditor would have the ability to link a "Pointer" to a chunk of data.
e.g: "Text1" linked to "Pointer1":
When tou drag or move that chunk of data (Text1) to a specific offset the linked pointer would recalculate itself to match the new offset of Text1.
the different types of pointers would be choosen from a drop-down menu and the user could add new types of pointers.
|
|
|
|
Disch
Guest
|
|
« Reply #27 on: June 13, 2008, 11:54:28 am » |
|
{2:X>>14}:{4:(X&0x3FFF) + (X>0x4000) ? 0x4000 : 0}
Ah! I see it now! It took a while though... is this standard regular expressions? Somehow I doubt this, but I was never really familiar with that either... Everything inside the braces except for the "2:" and "4:" is a standard C expression. The 2: and 4: are to represent that the number is to be padded to 2 or 4 digits. Everything outside braces would be printed "as-is" (ie: that colon that seperates the bank number of address within the bank) This is just the format I thought up of -- it could probably be simplified if it's too confusing. Maybe instead of C style ?: conditionals there could be multiple format strings -- each used for a different range of offsets. That would actually be a lot more flexible and probably much simpler to impliment, now that I think about it. Also,... if this feature were to be incorporated, I would advise to have the option of showing both, the rom file offset and the cpu address at the same time as well as both alone. Agreed.
|
|
|
|
Lupus Erectus
Guest
|
|
« Reply #28 on: June 13, 2008, 12:03:40 pm » |
|
If I guessed what this is trying to accomplish, a system to handle pointers is of key importance. This isn't the first time I see the idea of an editor configurable through a descriptor, but doing it as an enhanced hex editor is very ambitious. Hope to see this finished
|
|
|
|
creaothceann
Guest
|
|
« Reply #29 on: June 13, 2008, 12:51:47 pm » |
|
Handle large files gracefully. This was actually the biggest obstacle with my project attempt. I ended up splitting files into 'blocks' and only loading blocks as needed rather than loading the whole file at startup. Blocks were stored in a tree type structure.
-large file handling is indeed important. And I know that it's easy to implement wrong. I'm not sure how I'm going to resolve it, but I do see the need. http://en.wikipedia.org/wiki/Memory-mapped_file (doesn't have to represent a file, despite the name) Main drawback for you would be the fixed size, making increasing/decreasing the file size a bit more complicated. Just dynamically allocate a regular block (with malloc etc.) for the spill-over.
|
|
|
|
|