Author
|
Topic: What are your current gripes with hex editors? (Read 2 times)
|
Lleu
Guest
|
|
« on: June 12, 2008, 10:32:52 am » |
|
So, I've got an idea that's been sitting around for a bit. Yon "Swiss-army knife" for hacking files. I've been pretty dissatisfied with how many times I have to switch from editor to editor for their different features (or to avoid their specific flaws). So I've decided to try my hand at making my own. I'm asking for input because not everyone hexes in the same way. My way is not perfect. If you find a particular feature to be helpful for a task that you're willing to share, I'd like to know about it. If you've got an idea that you want implemented, let me know, even if it's crazy, and I'll honestly consider adding it. I've already got in mind a rather non-standard approach to editing files. The details on what makes it different are super-SEKRIT (shh!) at the moment, but it is to be a free utility, and I do want it to be useful in hacking console games. If you had the perfect editor, what would or wouldn't it do, and why? Since the interface will be somewhat key to the application, details on how you use the feature and what it is for would be important. Standard disclaimers: -I will not be making a game-specific editor. The focus is on reusable ideas and concepts. -I will not touch your Pokemans. -It's a side-project. It's done when it's done. -I'm not promising that I'll implement your idea. But if I can see the merit of it, I'll at least leave a way for it to be implemented by someone down the road. Mind you, I'm only in the planning stages right now, so don't expect a quick release. This project is meant to be a useful application of some of my recent studies (and some recent technologies) to help me internalize the concepts better. I personally like to have a few spare-time tasks (such as this) on hand so I can switch when I get bored/frustrated with one or the other. Once I've got something to show, I'll post it and turn this into a feedback thread. (UPDATE)Current planned featuresetData organization:- Four data levels:
- Project Data (For multi-file support, a wish for later. This'll be left out, at first)
- Group Data (Large chunks of objects)
- Object Data (One chunk of raw binary data that is, well, an object)
- Raw Data (to allow plugins uninterrupted access to all of the data, if they wish it. Also corresponds to individual bytes in the hex view)
- Name, comment and user-defined color support for all layers with the possible exception of Raw data.
- Links between data types, to designate pointers and file tables. (later)
Plugin support:- Autodetection (implemented, but considering extension for use by plugins)
- Allow synchronized views with other plugins.
- Populate data changes to all other views at the same point in the file.
- Allow plugins to control data, display, and selection of the data representation to some extent (e.g. for search plugins)
- Possibly allow plugin to interact with each other. I haven't fully considered all of the ramifications, yet.
- Windowed, Toolbar and dockable interfaces, eventually.
- Transparent overlays, if I can swing it. Modal dialogs will be reserved for things like saving files. Hopefully I can get the program following the Human Interface Guidelines a bit better this way.
- Implement data layer as plugin. Intended to leave room for better features down the road such as networked hacking sessions and/or source control access.
- Abstracted data views, allowing future plugins to support graphics, sound and video data if needed. Also, the same data can be visible in multiple views at once, allowing you to look at the data and its presentation simultaneously.
- Allow access to configuration methods.
- Versioned.
- Open and documented, once completed.
Logging:- XML logfile support (implemented, may change format)
- System event logging support (implemented, may add pre-defined error messages)
- Possible YAML log support
Configuration:- Profile support
- USB and Vista compatible (implemented)
- XML Configuration (implemented, refactoring)
- Accessible by plugins
Current list of suggestions from this threadSearching improvements: (Planned ) - Relative Searching, both by text and value (KaioShin, Inverse, dshadoff, Spikeman)*
- Search using different text encodings (KaioShin, Neil, Spikeman, DarknessSavior)*
- Search raw data (KaioShin, Neil)*
- Wildcards (Neil)*
Bookmarking:- Basic support (KaioShin)
- with sidebar display(InVerse)
- supporting import/export (InVerse)*
File Undo/Redo support:- Session-long undo/redo(Lenophis)
- extending past saves. (Lenophis)
- Undo in all editing views, text and hex (frantik)
Copy/paste support:- in all views, supporting both insert and overwrite (akadewboy, DarknessSavior)*
- Raw Data Insert (KaioShin)*
Custom encodings/table files (Kaiosin, seirj, Spikeman):* - MTE support in custom encodings (Seirj)-for investigation after all system encodings are supported.
- Allow display of MTE to cross line boundaries
- Fast table file/custom encoding creation (seirj)
Other- Goto/Jump function (Spikeman)*
- Multiple/split views (Dragonsbrethren)
- Binary file comparison (KaioShin)
- Data Corruption feature (KaioShin)
- Don't crash (seirj)*
- Shell Integration (seirj)
- Byte fill (KaioShin)
- Cross-platform (Shadowsithe, Kiyoshi Aman) - to be implemented using Mono
- Open-source (Kiyoshi Aman)
- Check for recent edits, but don't lock the program (KaioShin)
- Execution option (Inverse)
- Automatic pointer recalculation (ReBirFh. Lupus Erectus)
- Large file support (dshadoff, Disch, creaothceann)
- Simultaneous display of a translated file offset (Disch, Tauwasser)
- Custom data displays(dshadoff)*
- Color-coding (seirj, ReBirFh)*
- File overview (seirj)*
- Photoshop layer-style approach to hexing (dshadoff)
- Diff files instead of editing the main file (dshadoff)
- Adjustable byte spacing (Neil)
- Binary conversion of selected data for quick reference (Neil)
- Textual/XML configuration (ETG)*
- Compression support (Gnome)
- ISO support (gnome)
- Memory viewing/editing (DarthNemesis, Lindblum)
Sorry if I missed anyone's input in the list. As far as I know, we're caught up to here. Some of them correspond to the quick feature list at the top. My entire checklist wasn't included, as I'm reinstalling the OS on that system right now. Any features similar to something planned for early adoption have been marked with an asterisk. Not all features will make it in, I'm just listing everything. </disclaimer>
|
|
« Last Edit: August 06, 2008, 10:10:33 pm by Lleu »
|
|
|
|
Shadowsithe
Guest
|
|
« Reply #1 on: June 12, 2008, 11:13:27 am » |
|
Cross platform for obvious reasons.
|
|
|
|
MathOnNapkins
Guest
|
|
« Reply #2 on: June 12, 2008, 11:31:17 am » |
|
Well.... first of all what do you consider "flaws". I use XVI32 most of the time and I'd say its only flaw is that it could use a more robust searching feature.
|
|
|
|
KaioShin
Guest
|
|
« Reply #3 on: June 12, 2008, 11:58:42 am » |
|
Features most general purpose hex editors are lacking that you need for romhacking:
- Table support (compatible with Japanese!) - Relative Searching (with values, not just letters so it's working for Japanese too)
Features that shouldn't be missing:
- Compare files - Inserting raw data
Features that would be very nice:
- Built-In corrupting functionality - Bookmarks - Byte fill + Ability to change file size (to expand Roms) - Lot's of data searching options (Raw bytes or text using the table - should also work with Japanese)
|
|
|
|
seirj
Guest
|
|
« Reply #4 on: June 12, 2008, 12:14:18 pm » |
|
Hex Workshop is my favorite editor right now. Only problems I have with it are: Crashes randomly after saving a file Setting up your own character set is slow Character sets are byte for byte and games like Final Fantasy are harder to hack
Features about Hex Workshop I like: Right clicking any file to hex edit Bookmarks Byte filling Overall interface
That's all I can think of.
|
|
|
|
KaioShin
Guest
|
|
« Reply #5 on: June 12, 2008, 12:58:48 pm » |
|
Another feature I just remembered that would really had saved the ass of a ton of us in the past...
When saving a file, check for edits since the file was opened and warn the user about it. I can't count how often I accidentally killed changes I made due to a hex editor that had the file opened for hours while I worked on the file with other programs in the meanwhile...
|
|
|
|
Lenophis
Guest
|
|
« Reply #6 on: June 12, 2008, 01:17:37 pm » |
|
Program session-long clipboard to do seemingly infinite undo/redo. It's because of this Visual Studios is my primary hex editor, with WindHex32 my secondary.
|
|
|
|
Lleu
Guest
|
|
« Reply #7 on: June 12, 2008, 02:34:47 pm » |
|
Responses for everyone in one post. Yay! Cross platform for obvious reasons.
I'm not adverse to aiming for multiple platforms. However, would using Mono be alright with you? Some BSD-folk dislike the ramifications of it. And I'm targeting C# because I really want to speed development (both for myself and for later extensibility by others). Well.... first of all what do you consider "flaws". I use XVI32 most of the time and I'd say its only flaw is that it could use a more robust searching feature.
It's entirely open-ended. I'm asking for changes that people think should be implemented to make the work easier. Are you looking for RegEx support, or what? Detail roughly what you would like the search mechanism to support. - Table support (compatible with Japanese!) - Relative Searching (with values, not just letters so it's working for Japanese too) - Compare files - Inserting raw data - Built-In corrupting functionality - Bookmarks - Byte fill + Ability to change file size (to expand Roms) - Lot's of data searching options (Raw bytes or text using the table - should also work with Japanese) - When saving a file, check for edits since the file was opened and warn the user about it. I can't count how often I accidentally killed changes I made due to a hex editor that had the file opened for hours while I worked on the file with other programs in the meanwhile...
Let me check my understanding of your terms and how to implement them. Table file: This is a multi-line text file that has a hexadecimal number followed by the character it represents. Example: 15=L You want this to be able to handle Japanese text, so I'd need to be able to parse a unicode text file with the table information. I can also assume that table support implies displaying characters in the hex view using the characters designated in the table file, right? Would I need to have table file editing/creation capability in the application, or would loading table files suffice? I do plan on internally implementing Unicode, ASCII and SJIS viewing support as well as user-defined encodings. Since I'd planned on working with .NET, I should theoretically be able to allow the user to specify any valid system encoding and use that, as well. Theoretically... Another thought that came to me from the relative search comment was that you'd probably like to be able to search for text using the encodings of the table file, right? Relative search: Searching for an array of values (such as a text string) based on the difference between the provided values, as opposed to searching for the absolute value. Very useful for finding known text/values in a game. As I understand the concept, it uses the numerical difference between the provided values. Is there another definition of relative searching that does not, where it's easy to only implement English support? I assume with this you'd like to use * as the wildcard, Like in Relative Searcher, right? Should I ignore whitespace? File comparison: You mean binary comparison of two files, right? This is one of my big frustrations with hex editors, as the ones that support it often have bad interfaces. Or crash a lot. Inserting raw data: Would this be covered by 1) allowing the user to edit the hexadecimal display, and not just the text representation and 2) allowing the user to switch from Insert to Overwrite mode? Or are you wanting the capability to insert a file at a given location? Corrupting functionality: Not very familiar with this. I checked the utilities section, and it looks like it allows you to specify a range of data and "corrupt" (or replace) that data using various methods. Is this correct? Is the corruption method used important? Any additional details would be appreciated. I'll check the utilities for any readmes on what they do later. Bookmarks: Are you wanting something simple like specifying a point in the data and associating a name with it, or something else? Would you want a dedicated menu or a sidebar for going to those bookmarks, or some other navigation method (like keystroke combinations for skipping around)? I'm particularly interested in bookmarking, as I think it can boost productivity if done right. Byte fill + size adjustment: Just fill in a specified place with zeroes? Should it always be at the end, or configurable? Also, are you wanting a mode to lock in the file size? Or a warning if you're about to change the file size? Because allowing data insertion in the first place will require adjusting the file size. Search options: I guess I answered my table searching question. Basically, allow for raw searching, relative searching, text searching, and allow the user to specify which text format they're using for the search. Should relative searching have its own dialog box, or should it be integrated in the same window? Should search dialogs be modal or not? File change notification: Should the program keep a watch on the file and ask if you want to refresh on changes, like Visual Studio, or simply wait until saving to notify the user? I'm guessing a nice feature would be a temporary save + binary comparison here, although it'd be tricky to implement. Can't really think of another way to "merge" files. Hex Workshop is my favorite editor right now. Only problems I have with it are: Crashes randomly after saving a file Setting up your own character set is slow Character sets are byte for byte and games like Final Fantasy are harder to hack
Features about Hex Workshop I like: Right clicking any file to hex edit Bookmarks Byte filling Overall interface
That's all I can think of.
-Alright, so stability is a must. Understood. Is auto-saving required, or should it just be careful saving? Is it alright to sacrifice HD space for safer options (such as saving to a temp file first, before overwriting the original, incremental files, etc)? -Why is setting up your own character set slow? Can you think of ways that you would speed it up? E.g. instead of providing each individual character, allowing a range of characters to be set at once. -Not sure that I understand the "byte for byte character set" issue. Please elaborate. -Please review prior comments on bookmarks and byte-filling and see if you can clarify what you're wanting further. -I can't make any promises on the interface, as that's where I'm likely to make a lot of changes. Let me know if there's something specific that you like about it, though. Program session-long clipboard to do seemingly infinite undo/redo. It's because of this Visual Studios is my primary hex editor, with WindHex32 my secondary. As long as it implemented undo/redo from the last save like Visual Studio (as opposed to unlimited), you'd be alright? Would it be allowable to maintain this on the disk, sacrificing HD space and speed for RAM? If done this way, it could be transformed into a change log based on options. Everyone feel free to drop opinions on any of this, as I'd like to see any differences in styles, or features that are considered universally important. Any (calm) debate is welcome. I think feedback would be useful on search capabilities and sacrificing HD space for functionality, as these features are likely to have long-term effects on how the program works.
|
|
|
|
Lenophis
Guest
|
|
« Reply #8 on: June 12, 2008, 02:43:57 pm » |
|
As long as it implemented undo/redo from the last save like Visual Studio (as opposed to unlimited), you'd be alright? Would it be allowable to maintain this on the disk, sacrificing HD space and speed for RAM? If done this way, it could be transformed into a change log based on options. No. In order for the emulators to see the changes, I need to save. If saving wipes out the recent history of the clipboard, it's not going to do me any good if the changes don't work (and hence can't undo).
|
|
|
|
akadewboy
Guest
|
|
« Reply #9 on: June 12, 2008, 04:10:36 pm » |
|
WindHex32 is my main Hex Editor because it supports table files. What I don't like about it is I can't highlight text and copy it so I can paste it into Google Translate. In order to get a copy of the text I have to go to Edit->Dump Text. This is time consuming if I am merely translating a few sentences here and there. Lately I've been using MadEdit because the game that I'm editing has it's text encoded in Japanese Shift-Jis (this Hex Editor supports standard encodings, but it would be better if it supported custom encodings). This Hex Editor allows me to quickly highlight text and paste into whatever I want. The only thing I don't like about it is when you paste something back INTO the hex editor it inserts it rather than overwriting the text. This is bad for ROM editing because the filesize cannot change. To Compare files I use mirkes.de Tiny Hexer. This Hex Editor is awesome for comparing two files ( see screenshot), I just wish it supported Table Files. Now if there was a single Hex Editor that would replace all these I would be very happy.
|
|
« Last Edit: June 12, 2008, 04:36:35 pm by akadewboy »
|
|
|
|
seirj
Guest
|
|
« Reply #10 on: June 12, 2008, 05:49:13 pm » |
|
-Alright, so stability is a must. Understood. Is auto-saving required, or should it just be careful saving? Is it alright to sacrifice HD space for safer options (such as saving to a temp file first, before overwriting the original, incremental files, etc)? -Why is setting up your own character set slow? Can you think of ways that you would speed it up? E.g. instead of providing each individual character, allowing a range of characters to be set at once. -Not sure that I understand the "byte for byte character set" issue. Please elaborate. -Please review prior comments on bookmarks and byte-filling and see if you can clarify what you're wanting further. -I can't make any promises on the interface, as that's where I'm likely to make a lot of changes. Let me know if there's something specific that you like about it, though. -Hex Workshop actually saves the data, it just crashes at the same time. -Have you used the character filter in Hex Workshop? It's under Preferences>Display>Character Filter It's slow because instead of quickly making a table file, I have to go down a list, click a box, type in my character, click "Reverse Map" then click "Apply." It's slow when you have to do that 50 times. -Hex Workshop shows the bytes on one side and the character set on the other. So if in ASCII, on the left it shows "54" and on the right it shows "T." You couldn't, say, have "54" show "Th." One byte shows one byte, one byte can't show 2 bytes. Final Fantasy is compressed. In that game, it's set so that one byte can show more than one character. I think it's even set up so that one byte can display entire words in the game. For example, "40" could be displayed as "world." But now I'm talking game specifics, and that's not important, it's the feature that's important.
|
|
|
|
frantik
Guest
|
|
« Reply #11 on: June 12, 2008, 05:55:23 pm » |
|
I use ultraedit for almost all my text and hex needs.. the only thing it doesnt have is undo in hex mode which is kind of a pain in the ass sometimes :banghead:
|
|
|
|
InVerse
Guest
|
|
« Reply #12 on: June 12, 2008, 06:05:59 pm » |
|
Relative search: Searching for an array of values (such as a text string) based on the difference between the provided values, as opposed to searching for the absolute value. Very useful for finding known text/values in a game. As I understand the concept, it uses the numerical difference between the provided values. Is there another definition of relative searching that does not, where it's easy to only implement English support? I assume with this you'd like to use * as the wildcard, Like in Relative Searcher, right? Should I ignore whitespace?
I know of several experienced ROM hackers who use the relative search utility within Translhextion to do their relative searching and then switch to another hex editor that doesn't make them vomit to do all other work, once the table values are found, so you might want to take a look at it, if you haven't already. It could probably use a few tweaks to the setup and a few added features (none come to mind at the moment, but I'm sure some others will have suggestions) Also, someone recently released a relative search utility under the name Monkey Moore that seemed to work extremely well. Unfortunately, it currently only has the abilty to relative search for words using a standard order Romanic alphabet. The author said he intended to implement something similar to Translhextion's 'Value Scan Relative' feature sometime in the future, but it's not his main focus, so it could be some time before it ever occurs. However, the source code to Monkey Moore was also released and I believe it was written in C++ or C# or something along those lines, so it might be of use to you. (I believe the source code is archived on this site, if not, there's a thread about it in the Personal Projects forum.) As you can probably guess, a really useful relative search utility is the primary thing I believe is lacking. I'd be happy with a standalone app, but if it were a feature of a top notch hex editor, I'd be ecstatic. Also, it would be really neat if the editor had a bookmarking feature with a sidebar (ala Acrobat Reader) that allowed for bookmarks to be imported/exported, thus allowing people to document the ROM within the bookmark file. Then people who were interested in hacking a ROM could load up a bookmark file to find the values they're wanting to hack. People could even work on community bookmark files, adding in information as it is discovered. If it caught on, I could even see a separate section on RHDN just dedicated to such bookmarks. It would also be nice to have a button within the hex editor that would automatically launch the currently open file in an emulator for testing.
|
|
|
|
Lleu
Guest
|
|
« Reply #13 on: June 12, 2008, 07:33:59 pm » |
|
As long as it implemented undo/redo from the last save like Visual Studio (as opposed to unlimited), you'd be alright? Would it be allowable to maintain this on the disk, sacrificing HD space and speed for RAM? If done this way, it could be transformed into a change log based on options. No. In order for the emulators to see the changes, I need to save. If saving wipes out the recent history of the clipboard, it's not going to do me any good if the changes don't work (and hence can't undo). Hmm. I can see your point. It sounds like session-based unto streams might be more useful. I'll think about revert capabilities in addition to this. WindHex32 is my main Hex Editor because it supports table files. What I don't like about it is I can't highlight text and copy it so I can paste it into Google Translate. Lately I've been using MadEdit because the game that I'm editing has it's text encoded in Japanese Shift-Jis (this Hex Editor supports standard encodings, but it would be better if it supported custom encodings). This Hex Editor allows me to quickly highlight text and paste into whatever I want. The only thing I don't like about it is when you paste something back INTO the hex editor it inserts it rather than overwriting the text. This is bad for ROM editing because the filesize cannot change. To Compare files I use mirkes.de Tiny Hexer. This Hex Editor is awesome for comparing two files ( see screenshot), I just wish it supported Table Files. -Alright, so each individual view (hex, text, etc) needs to be copy/pasteable. Even though the views should have synchronized selections, the actual data copied should come from the view selected. -Another vote for custom encodings/table files. -Insert/Overwrite support for pasting. Sounds logical, but I'm glad you pointed it out. -Ooo! Tiny Hexer has a rather good binary compare interface. That's way better than the current program I've been using. /me takes notes. -Hex Workshop actually saves the data, it just crashes at the same time.
-Have you used the character filter in Hex Workshop? It's under Preferences>Display>Character Filter It's slow because instead of quickly making a table file, I have to go down a list, click a box, type in my character, click "Reverse Map" then click "Apply." It's slow when you have to do that 50 times.
-Hex Workshop shows the bytes on one side and the character set on the other. So if in ASCII, on the left it shows "54" and on the right it shows "T." You couldn't, say, have "54" show "Th." One byte shows one byte, one byte can't show 2 bytes. Final Fantasy is compressed. In that game, it's set so that one byte can show more than one character. I think it's even set up so that one byte can display entire words in the game. For example, "40" could be displayed as "world." But now I'm talking game specifics, and that's not important, it's the feature that's important.
Thanks for the clarification on the crashing. -I haven't used the character filter in Hex Workshop, but it does sound cumbersome. Apparently, table file support will meet your needs, then? -Ah. You need support for multiple-character encoding in the table files. Gotcha. I use ultraedit for almost all my text and hex needs.. the only thing it doesnt have is undo in hex mode which is kind of a pain in the ass sometimes :banghead:
-Ok, all views need to support undo/redo. I know of several experienced ROM hackers who use the relative search utility within Translhextion to do their relative searching and then switch to another hex editor that doesn't make them vomit to do all other work, once the table values are found, so you might want to take a look at it, if you haven't already. It could probably use a few tweaks to the setup and a few added features (none come to mind at the moment, but I'm sure some others will have suggestions)
Also, someone recently released a relative search utility under the name Monkey Moore that seemed to work extremely well. Unfortunately, it currently only has the abilty to relative search for words using a standard order Romanic alphabet. The author said he intended to implement something similar to Translhextion's 'Value Scan Relative' feature sometime in the future, but it's not his main focus, so it could be some time before it ever occurs. However, the source code to Monkey Moore was also released and I believe it was written in C++ or C# or something along those lines, so it might be of use to you. (I believe the source code is archived on this site, if not, there's a thread about it in the Personal Projects forum.)
As you can probably guess, a really useful relative search utility is the primary thing I believe is lacking. I'd be happy with a standalone app, but if it were a feature of a top notch hex editor, I'd be ecstatic.
Hmm. I'll take both Translhextion and Monkey-Moore into account when working on relative search. I like the interface of Monkey-Moore. Maybe if it wasn't too much trouble I could allow for creation of a partial table file from the relative search window... Also, it would be really neat if the editor had a bookmarking feature with a sidebar (ala Acrobat Reader) that allowed for bookmarks to be imported/exported, thus allowing people to document the ROM within the bookmark file. Then people who were interested in hacking a ROM could load up a bookmark file to find the values they're wanting to hack. People could even work on community bookmark files, adding in information as it is discovered. If it caught on, I could even see a separate section on RHDN just dedicated to such bookmarks.
It would also be nice to have a button within the hex editor that would automatically launch the currently open file in an emulator for testing.
-TEEHEE! I like your interest in bookmarks. I also like that you're headed in a similar direction to what I envisioned with them. It's part of the idea behind this thing. -Oh yeah, command-line execution of files would be a big help.
|
|
|
|
Aerdan
Guest
|
|
« Reply #14 on: June 12, 2008, 08:22:29 pm » |
|
I notice some people said cross-platform, and while I'm going to chime in with them, I also think you should be open-source. That way other people can contribute if they want to.
|
|
|
|
|