He`s saying that there should be address translation depending on the different roms. E.g. SNES roms have many addressing modes. Because every bank in SNES games has 0x8000 bytes, Bank 0 is noted as follows from Byt 0x000000 to 0x007FFF in the rom as 00:8000 to 00:FFFF. It`s a programming perspective thing, because that`s how the cpu is addressing the file. Similarly, for gameboy it would be Bytes 0x000000 to 0x003FFF (0x4000 byte banks) adressed as 00:0000 to 00:3FFF whereas offsets 0x004000 thru 0x007FFF are 01:4000 to 01:7FFF etc...
Oh yeah, he`s proposing a single line system that would be readable by the program and then interpreted for address translation. I don`t think this is a good approachh as for different hardware (SNES vs. GB/C for instance), the addressing is not always quite the same. In GB the 0th bank is constantly available at 00:0000 to 00:3FFF while all other banks are at XX:4000 thru XX:7FFF [XX>00]...hence a problem :-/
Also,... if this feature were to be incorporated, I would advise to have the option of showing both, the rom file offset and the cpu address at the same time as well as both alone. Also, in each mode (if you work with a address translation, regardless of what "offset view style" you are in), it should feature a boundary check (that is configurable turn off/on in the options - not annoying that way), so people don`t put stuff across banks when they don`t think for a minute or two.
cYa,
Tauwasser
It wouldn't have to be simple -- I just used a simple formula as an example. That GB format you described could be handled with something like:
{2:X>>14}:{4:(X&0x3FFF) + (X>0x4000) ? 0x4000 : 0}
Which would use address 0000-3FFF for the first bank, then 4000-7FFF for every bank thereafter.
@ Lleu:
Basically the idea is instead of printing basic file offsets, you "translate" the offset into another number based on a given formula. This way the hacker doesn't need to do any offset<->CPU address conversions like for pointers or anything... since the CPU addresses will all be printed in the hex editor rather than the file offset. It would also help when working in a hex editor and debugger in tandem -- since debuggers and tracers generally don't print raw file offsets.
What a traditional hex editor would look like:
00007FE0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00007FF0 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01
00008000 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02
What a translated offset display (SNES LoROM, in this example... hopefully my recollection of it isn't all wrong) would look like:
00:FFE0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00:FFF0 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01
01:8000 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02
The idea is the format string is customizable so that the user can write one for whatever game/system they're working on.
Alright, so, as I understand it, this need would be satisfied by allowing the Hex Editor to display a formatted string instead of (or alongside) the normal addresses. As I understand it, the string would need to be displayed differently for different CPUs/systems. So the program would need to take in the file's address, apply the formula, and display the new format. Assuming I get a decent enough algorithm it will be reusable.
Does this only need to be displayed, or will it need search/Go to address capabilities? I'm not quite sure how I could enforce boundaries, other than a visual display. I'll give it some thought. Any other features that might need to be integrated?
Tauwasser:
Yes, if you look hard enough you can find all features in this or that program. From what I understood, the point of this editor was to get it all together into one program that works, is stable and moreover (this is a HUGE plus over an unbearable console editor like Snesedit) to have a strong interface that makes it comfortable to work with.
And that's something really needed and that I'll support 100%.
Lleu:
- Relative Searching: Inverse pretty much nailed all the problems and important aspects. Many relative searchers were written by people who translate from English to other languages, so they all work only with the latin alphabet. They just search using words only and not values, so you can't search comfortably for Japanese.
- Table Files: It's very easy to support a basic table file, but it gets more tricky when you dive deeper into it.
A table file can encode one byte entries, but also two or even more byte entries, all in the same file. For encoding the file, SJIS or Unicode should be possible, as well as ASCII of course.
And yeah, searching for data should be possible using the values from the table file.
- Inserting Raw Data:
I meant a data file. Just a binary file, shouldn't be hard to implement. - Corrupting:
Corrupting is changing an area of data in a ROM to systematically find where specific data is stored. Let's stay you want to find where the level data is stored in a jump and run game, but you have no idea where or how the data looks like. With corrupting, you can just change areas of the ROM to all zeroes or anything, until you find the data. Depending which data you overwrite the game might crash, the sound might stop, some graphics may be messed up or... the level layout might have been messed up. There, you just found it's location 
- Byte fill: Should be possible at any given location. Corrupting is the same, just with different intentions. Changing the file size will automatically fill the new space with some data of course. It's just needed to expand ROMs. Should be simple.
- Searching: I don't know about the interface details yet, haven't thought about it. Sorry if I just ask for things without a clue how to implement them best
- File change notification: It would be kind of cool if it would notice it immediately. It should then ask if the file should be reloaded or not.
HxD has another method to circumvent accidentally changing the data with other programs... it locks the file completely. You can't open the file with anything else as long as you have it opened in HxD. Please don't follow that route... it's majorly annoying.
-That answers the majority of the questions I had on your suggestions. I may come back to some later.
-Yes, I'd have to agree that HxD's file locking is obnoxious. It's so easy to forget it's there.
I'll admit it, I only skimmed the thread, some of these might be duplicates.
- Adjustable byte spacing. It is occasionally helpful to display data in two byte or three byte or four byte clusters for data analysis
- Display binary conversion for selected bytes in the status bar at the bottom of the screen, or somewhere else easily visible. I am occasionally too stupid to do the conversion in my head.
- Second only to table support, searching. Like everything said above and more. Wildcards, hex search, text search via table, etcetcetc.
-Adjustable byte spacing is a new one, but a good idea.
-Number conversion displays are helpful. Probably would also be helpful to include endian conversions, data type displays. This would probably be pushed back until later/implemented in a plugin. It's one of the things that the new structure is supposed to simplify.
-Table support +1
-Searching +1
I just remembered something I'd really, really like to see in a hex editor.
I want to be able to color code chunks of data and then zoom out really far so that the whole file is on screen and I can see and get my bearings on where data is and how big it is compared to the entire file.
If I could do this, I could see what areas that I know what are and can ignore and what areas I don't know what are and can search in.
I hope this makes sense.
TEEHEE! Drat! I might as well give it up. I can't accept feedback honestly and keep it a secret, anyway.
The premise of the thing is around data mapping with customizable data views. So technically it's a hex editor second (as a plugin, although it will be included), and a file documentation/organization tool first. This way people should be able to represent the data in the most suitable format, and fall back to hexing (or whatever the default configured plugin is) when the data structure is uncertain. It will require some interesting architectural decisions.
@KaioShin: There is a windows version of snes edit, though it feels like the command line it is...
I don`t use it much/like it either, however it has many features, regardless of how you look at it (psx tim viewer and other gfx stuff, too!).
Also, for me Hex Workshop has never ever failed to function properly... It only ever crashed when I accidentially opened 20 GB and compared files or something... which I had created on the fly for...er...some unbeknownst purpose lol
I`m not against this or anything, I`m just saying where some good features are, so one could kinda copy from those hex editors
cYa,
Tauwasser
I'll keep features in mind, but I'm trying to avoid source-peeking on related projects. Even if I did, I'd have to rewrite the code significantly. Besides, DOS hex editors always make me think of BBS door games, like Tradewars and LoRD...
Also, it would be really neat if the editor had a bookmarking feature with a sidebar (ala Acrobat Reader) that allowed for bookmarks to be imported/exported, thus allowing people to document the ROM within the bookmark file. Then people who were interested in hacking a ROM could load up a bookmark file to find the values they're wanting to hack. People could even work on community bookmark files, adding in information as it is discovered. If it caught on, I could even see a separate section on RHDN just dedicated to such bookmarks.
You know, I've had a similar idea rolling around in the back of my head, but for use when viewing/editing the memory inside the emulator. I love trying to pick apart how the game stores its data, so something like this would be a great step in that direction. :thumbsup:
Memory editing would probably be awesome indeed, but it's a bit out of scope for now. Hopefully you'll find the file features useful enough when it's done.
I like the idea of color coding chunks of data.
e.g: I have three chunks of data labeled "Text1", "Text2", "Text3" with the color "Blue".
The Hexeditor would need the ability to:
-Move to (Place that chunk of data beginning in a specific offset).
-Dump/Extract (Dump the chunks of data from a specific color in separate files with their labels as file names)
-Insert from
And in a dream...
Your hexeditor would have the ability to link a "Pointer" to a chunk of data.
e.g: "Text1" linked to "Pointer1":
When tou drag or move that chunk of data (Text1) to a specific offset the linked pointer would recalculate itself to match the new offset of Text1.
the different types of pointers would be choosen from a drop-down menu and the user could add new types of pointers.
Since the secret part is basically not anymore, I'll elaborate. To me, it seems like just color-coding and bookmarks by themselves are insufficient. It doesn't really facilitate a proper representation of the file and the data it contains. Basically, plans include four different classifications/methods of accessing data through the plugins:
-Project Data (For multi-file support, a wish for later. This'll be left out, at first)
-Group Data (Large chunks of objects)
-Object Data (One chunk of raw binary data that is, well, an object)
-Raw Data (to allow plugins uninterrupted access to all of the data, if they wish it)
Besides organizing the data, user-defined names may be assigned to each data classification.
Plugin basics:
-Allow synchronized views with other plugins.
-If the plugin supports reading and writing, updated data will be populated to all views.
-Allow plugins to control data, display, and selection of the data representation to some extent (e.g. for search plugins)
-Windowed, Toolbar and dockable interfaces, eventually.
-Transparent overlays, if I can swing it. Modal dialogs will be reserved for things like saving files. Hopefully I can get the program following the Human Interface Guidelines a bit better this way.
I hope to allow establishing links between the individual data types, but this part hasn't been prototyped. And any automatic calculation of pointers, checksums and the like, although considered, is definitely put off until everything else is stable.
Basically, I want the base system, data interface, dynamic plugin interface and basic hex capability (in a plugin) completed first. But I want to avoid coding myself into a corner, which is why having y'all share ideas definitely helps me. I'm hoping I can use them while prototyping to ensure that there's room for growth where people want it.
It is more ambitious than most things I've put out, so it'll be a big learning experience for me. Expect lots of delays. Once it's completed, though, I will have won at everything ever, forever.
If I guessed what this is trying to accomplish, a system to handle pointers is of key importance.
This isn't the first time I see the idea of an editor configurable through a descriptor, but doing it as an enhanced hex editor is very ambitious. Hope to see this finished
I'll have to mull on how to properly support pointers. Deflected until late stages...
To creaothceann and Disch:
Thanks for the input on this. I know large-file support is going to be a big obstacle once I get there. The approach I've been considering for now is a Model-View-Controller approach using plugins. That way I can separate the data layer from everything else. And then I can probably cheat by implementing basic file I/O and substituting it out later.
I know it could lead to problems down the road, but I'm not afraid of rewriting things. I think I'll have to, anyway, as I don't quite know the full depth of what I'm planning.
I second the gripe about being able to copy text. I think you should be able to do that.
Also, WinHex has the ability to support DTE/dictionary characters for scripts. That's wonderful. But when a multi-byte character starts on the end of one line (say, 0x0000000F) and ends on the next one (0x00000010) it doesn't display it properly. Granted, most of the time you can understand what it should be. It's annoying, though.
~DS
- Copying data representations from all views +1
- Hmm. I hadn't considered how to handle MTE values that spread across lines. I'll add that to the think tank.
Hah! I KNEW this was coming when I saw it linked on Reddit early this morning. Yes, I understand the dangers of cave coding. However, I personally have some own arguments for it in this case:
1. Stability. I worked in a software company where they occasionally did 0-day fixes for larger clients. I've personally spent four hours overtime on a Friday night tracking down an issue with credit card authorization. It turns out the problem came from another coder having checked in some code that made the program to pull the date from the SQL Server instead of the local machine. Yeah, a 5-minute search-and-replace that no one thought was going to hurt anything destabilized one of the main features and crashed the program.
Yes, it should have undergone testing. And it did, as I found it. But as the software had to be released that day it changed the situation a bit. Sometimes you don't control the release circumstances. In this case, I will.
But we all agreed at that point that software shouldn't be checked in until it's certain to not go Godzilla on everything else. It needn't be perfect, but bare minimum, it should be functional and ready for testing.
2. Personal improvement. I'm used to working mostly with Windows Mobile devices in C# 1.0. There's a lot of workarounds and gimmicks needed for that, which don't take place on desktop machines. It's had deleterious effects on my code, although ingenuity has increased. I've already substantially rewritten most of what I've made, and I expect further rewrites. It's a good learning experience for me, and I'm a greedy bastard in wanting to fly solo on it for a bit. If I do all the work myself, I'll end up learning how to do it right.
3. Compatibility. Anticipating these rewrites, I don't want anyone becoming familiar with the old code. Until it's publicized I don't have to worry about breaking compatibility with older plugins.
I'm not saying that being open with code or mistakes is bad. I've gone to other coders loudly proclaiming myself to be an idiot when I needed help. I just don't want to apply it here, yet.
Whew! Back to work.
Author
