Author
|
Topic: The New Romhacker's Bible (King Gideon Edition) (Read 2 times)
|
Nightcrawler
Guest
|
|
« Reply #30 on: March 05, 2007, 09:30:34 am » |
|
People who stumble on compression often do the mistake of jumping right in without much other experience and wonder why the learning curve is too steep. I haven't dealt with compression myself before, so this might seem a bit hypocritical, but I know it's no big deal if you really understand ASM. Trace the routine, see how it works, emulate that in a programming language of your choice - voila, decompressor. What do we need for this? The ability to trace a routine, being able to make sense of ASM code and general programing skills. (Cyberman: I know it's a bit different for PC since you don't have so easy debugging options, but pretty much everyone interested in that document will want to hack console games) Yes, it's very hypocritical. If it's that easy to do why can so few do it? Spikeman is a great example. He knows assembly. Why can't he just trace the routine, see how it works, emulate that, and viola? There's a bridge that is missing to correlate and bring these things together. Even Gideon himself is another example. How many of his projects has he gotten technical help on when it has come to compression? I know of quite a few. Perhaps Gideon is in need of something to help bridge the gap on some application issues. I know he's only somewhat recently gotten much assembly experience under his belt. If you're trying to write a new be all end all document, you can't ignore these things or your document becomes useless and/or no more useful than any of the others we already have. Why does it become useless? Because it needs to cover things that aren't already covered. The same topics have been rehashed in hundreds of documents, but there are still things that you cannot learn from ANY of them. And this is one of these issues. Knowing how to read lines of assembly and application of that knowledge. And it's a bit ridiculous to assume that just because you can read and understand a few assembly routines, that you can just magically understand what another more complex routine does that addresses a concept of something totally different. To identify what an assembly routine does, you need TWO pieces of knowledge. ONE, You need fluent understanding of the instructions used and what they are doing, and TWO, you need some sort of conceptual knowledge of how what you're looking for is being done. Example, you want to expand a menu. How do you do it? If you don't understand the concept of how the menu is drawn on the screen and what you should be looking for, a trace log is going to do you little good. You're going to have hundreds of lines of code and have no idea where to look. The magic is LINKING the two together. I specialize in helping individuals who have learned most of what they can from the available documents and who want to move beyond that. For each person that I have helped, I see many of the same stumbling blocks and the same patterns emerging. And on top of that, I myself have experienced many of the exact same obstacles in the past. All things considered there is a clear trend to me that there missing areas of information. Maybe I'm missing the picture here, but I thought the idea behind this document was to make a new up to date sort of be all end all (this is not possible in totality of course) that we can refer people to. If that's the case, areas that aren't already addressed by the many documents we already have should be taken into consideration. Otherwise, really, what does it have to offer? The utilities referenced will be updated from Nesticle, some concepts may be presented from a different angle or two, but beyond that, it's the same thing over again. Sorry for the long post, but I feel pretty passionately about providing better learning resources for ROM hackers. Founding of RHDN case in point.
|
|
|
|
Metal Knuckles
Guest
|
|
« Reply #31 on: March 05, 2007, 10:31:12 am » |
|
Book 2 - Fonts and tables
-Bitplane font storage -Japanese table files
Book 3 - The Door to Serious Hacking
-Headers -Addressing modes -Pointers
Book 4 - Handling scripts
-Text dumping -Text insertion
Book 5 - Assembly
-Assembly
Perhaps switching books 3 and 4 would work well. Changing pointers and such isn't completely nescesary in every translation, such as with minor menu translations, and seem a bit more advanced then simply dumping and reinserting the text. Of course, most projects do require that before anything else, so perhaps not.
|
|
|
|
KaioShin
Guest
|
|
« Reply #32 on: March 05, 2007, 11:28:43 am » |
|
The problem is this - you can't point the finger on the magical missing link people need. Unless we know what it is, how can we try to fill that gap? I just look at it from a practical point of view, what should and what can he write about? You are right, an important part to understand some code is to know the basics behind what it's doing, in our case this are the common compression schemes. That's why I said to link to wikipedia articles. I know them quite well, they are excellent in explaining how the compressions work. What could Gideon write better about it? If he will cover the topic I'll be more than happy and read it myself, since I always like to broaden my technical horizons. I'm just sceptical on what to expect realistically from such a chapter. It's so easy to say, "yo, write something about compression please". But doing this, and in a way so that everyone understands it, so that all the subtopics are covered, so that it isn't just a rehash of countless other documents is a completely different thing. It's just that I'm surprised that so many people are so fixed on this topic. I'm sure Gideon expected something different from this topic than 2 pages of asking for a compression doc
|
|
|
|
RedComet
Guest
|
|
« Reply #33 on: March 05, 2007, 12:40:02 pm » |
|
Except it IS possible in some cases to finger the link. Take compressed graphics for instance. Let's say, for the sake of argument and practicality, a font in an SNES game.
Staring at a string of text displayed in-game, what's the first thing you know about the font or at least the characters being displayed? They're in VRAM. Okay, what good does that do? Well, you need to have some idea how the graphics get to VRAM to being, usually DMA if the console allows it or via hardware ports. I'll go the DMA route for simplicity's sake.
Now, there's usually two methods for DMAing data: ram to vram and rom to vram. Since we're thinking it's compressed in the rom, we can rule out rom to vram for the time being, which leaves us with ram to vram. We need an easy way to scope ram: savestates!
Load up the rom in an emulator that utilizes an uncompressed savestate format (Gens or ZSNES for instance) and create a savestate with the string on-screen. I usually try to create the savestate about midways through when half of the string's been displayed if possible. Of course, if the game doesn't draw one character at a time, instead opting for drawing whole lines or even entire windows, you'll have to create the savestate with the string in its entirety on screen.
After that, open the savestate up in a graphics editor and check the areas of ram (you'll need to know how the particular emulator you're using lays savestates out). If you find the text, jackpot! From there you set a write breakpoint for that area of ram and then work backwards until you find the calling jump and you'll have found your decompression routine. This is even easier if you've located the text read routine, since you can ignore everything before the character to bedrawn is read AND you've got something you can follow through and see how the character's hex code is turned into its corresponding font tile(s). From there it's just a matter of studying the code, stepping through it in a debugger if possible or look at trace data.
It's the same principle for all compression. If you know what the end result is, find it and then work backwards using the tools at your disposal.
The above is a little rough and could use some more in depth explanation and examples, but it should get the point across. And that's just one method of doing it, without going in to any compression schemes at all., which isn't really necessary, imo. You can worry about applying a name to the scheme after you're finished back tracing and figuring out the specifics of the routine.
|
|
|
|
Gideon Zhi
Guest
|
|
« Reply #34 on: March 05, 2007, 01:27:53 pm » |
|
My issue with compression is that I'm not a programmer. Oh sure, I can do 65816 no problem, but when it comes to C++ or anything similarly high-level I'm a bit of a fish out of water. I've done some simple utilities and I managed to write one program to decompress data (basically, just a rerendering of the 65816 code in C++) but I never managed to get it recompressing properly beyond the first 128 bytes or so. Decompression is generally fairly simple - if you can figure out what the routine's doing, you can rewrite it in another language. Recompression is quite a bit more complicated; even if you know what the thing's doing on a conceptual level, it's still pretty tough to code a proper recompressor. That's why I've always gone with the "let's break it" mentality :p
|
|
|
|
Nightcrawler
Guest
|
|
« Reply #35 on: March 05, 2007, 01:37:00 pm » |
|
I think my long rant established compression was just ONE example of a case in point. This is bigger issue than just a compression. It's adding things to this document that aren't in others. An example like RedComet gave is one thing. It's a simple explanation of how to tie together various bits of knowledge into actually accomplishing something. You CAN point the finger in many cases where people get stuck. This doesn't have to be any huge document. Just some blurbs that make that bridge so people can understand how to do something with what they know.
I just answered several questions this week on the forums that exemplify this situation that are NOT about compression. I distinctly remember the SNES palette guy today. He had the skills, he just didn't know what to do with them. He need the mechanics of putting things together to figure out where to start to accomplish a task.
Another guy was the MTE guy. He knows how to read assembly. What he didn't know was how to put what he knows to use in figuring how how to accomplish his task. He really didn't know where to start there.
THAT's the kind of thing that we need around here. Get people over these hurdles that AREN'T in any documents to date really aside form straight code examples. Code examples are one thing, but EXPLANATIONS on a higher level are another. All the explanations I've made to people that I get thanked most for are for those that are slightly above a line by line code level, but put the missing links together for them on how they can utilize what they know to come up with a plan to accomplish a task.
I think if you're going to write new document today and you really want it to be helpful and useful, do something with it that hasn't been done to death already. It's evolution of our kind. Yeah, some people still don't know how to make a table even with 50 documents on it, maybe ROM hacking isn't for them. But there is a large group of up and coming people that know the basics, learn a bit of everything, but can't progress from there because something is missing.
Some of the old top dinosaur fossils are dropping out of this community now a days. Who's going to replace them? I see a lot of up and coming people at the intermediate level that could use a small kick or two to hurdle them into super stardom to be able to tackle even the most difficult of projects. But they need to get that kick of information.
If I only had the time to do this, I'd write such a thing, but all I can do is preach in the shades of MKendora in hopes that those who do have the time can take heed and evolve.
Ask yourself some questions. Think about taking on the most difficult project available for your system of liking. Some of you out there could handle any game for your platforms of interest, but I expect many of you could not. Ask yourself why that is? What would you need to know to be able to do so? Is it a lack of hardware knowledge? Is it a lack of assembly syntax? What questions have you asked since you've come to this board? Is it possible that your questions could have been answered in a document outlining how to apply some of your knowledge to certain situations? Or perhaps explaining some concepts to you to help things click or know what to look for in that 300K assembly file?
I'm guessing the answer to those last two might be 'yes'.
|
|
|
|
Cyberman
Guest
|
|
« Reply #36 on: March 05, 2007, 04:08:41 pm » |
|
My issue with compression is that I'm not a programmer. Oh sure, I can do 65816 no problem, but when it comes to C++ or anything similarly high-level I'm a bit of a fish out of water. I've done some simple utilities and I managed to write one program to decompress data (basically, just a rerendering of the 65816 code in C++) but I never managed to get it recompressing properly beyond the first 128 bytes or so. Decompression is generally fairly simple - if you can figure out what the routine's doing, you can rewrite it in another language. Recompression is quite a bit more complicated; even if you know what the thing's doing on a conceptual level, it's still pretty tough to code a proper recompressor. That's why I've always gone with the "let's break it" mentality :p Hey no one is beating on you LOL. Actually what I really like to see is pointers on how to get from A to B. Two heads are better than none. Experience creates the voice of reason. In any case, giving people the tools to recognize the problem and define it in there head, is what I'm talking about. I was told once "You write complicated code", by someone who didn't know how to program. In perspective (IE reversing the perspective) I tend to write simple code. If the first time you run into something you have no idea what might be, (happens all the time for me) it would be good to have some simple guidelines to recognize things and to help one come up with a plan of action. Thus far what you have is fairly good. I suppose the best way to look at it. "So I've read this bible now what?" In other words "More advanced topics and If you get stuck what to do." These aren't EASY questions to answer, since you are just one person. However you can get information from contributers if that's the case. I'm sure people are willing to help you. I already volunteered some assistance. I am thankful I had someone like Gemini to point things out I was baffled at, and some of the people at forums.qhimm.com who discused delta compression. It takes a lot more thinking to do things once you get beyond finding tiles and palettes, especially when you hit 3d. Maybe you should make a section of "Making a Plan of Action you can live with in your project" More than half of all projects get bogged down when someone hits a brick wall. I know I had to debug some LCD driver code recently that puts a bit map to a display. The bug was in my rasterization code turned out for the bitmap. Not the LCD code (ugh). So a little mental guidance goes a long ways. Still have a bug with Variable width fonts though Cyb
|
|
|
|
Spikeman
Guest
|
|
« Reply #37 on: March 05, 2007, 05:47:20 pm » |
|
On a slightly unrelated, but still related note, one thing I think would be REALLY helpful would be some "case studies". As in a detailed step-by-step process of translating an entire game. Of course you could do one that you've done before, and use your work to shorten the time writing the document. Honestly, this kind of thing would REALLY help me, even though my system of choice is different than yours (actually, I'm not sure if you've done any GBA hacks.) And Nightcrawler, since you seem to understand the conceptual parts of the hacking process so well, might I suggest that you write a document on that?
|
|
|
|
Gideon Zhi
Guest
|
|
« Reply #38 on: March 05, 2007, 05:57:29 pm » |
|
On a slightly unrelated, but still related note, one thing I think would be REALLY helpful would be some "case studies". As in a detailed step-by-step process of translating an entire game. Of course you could do one that you've done before, and use your work to shorten the time writing the document.
I was planning on that, actually. If you'd read some of the first posts, you'd know that I was going to use a Monstania rehack as a guinea pig
|
|
|
|
creaothceann
Guest
|
|
« Reply #39 on: March 05, 2007, 06:03:20 pm » |
|
//... And Nightcrawler, since you seem to understand the conceptual parts of the hacking process so well, might I suggest that you write a document on that? If I only had the time to do this, I'd write such a thing, but all I can do is preach in the shades of MKendora in hopes that those who do have the time can take heed and evolve.
|
|
|
|
Aeris130
Guest
|
|
« Reply #40 on: March 06, 2007, 12:08:47 pm » |
|
Speaking of case-studies, it's something I've missed in many docs. Although there's no need to write a fully fledged walkthrough complete with data offsets for every chapter in the book, it'd be nice with a short disclaimer at the end of every chapter (save for the most basic stuff), suggesting some suitable roms/isos that are known to store whatever you're trying to hack in the same way the guide describes.
|
|
|
|
Nightcrawler
Guest
|
|
« Reply #41 on: March 06, 2007, 12:47:50 pm » |
|
Another useful thing would be to explain how each concept can easily be applied to any similar platform.
That's another heavy hitter. "I've read read the documents, but it was for system xxxx. It won't help for system yyy". Few realize that IT DOESN'T MATTER. I guess it's a lack of understanding of the concept, but they don't realize you do almost the same thing and the same process for all similar platforms. Even the non similar platforms have only a handful of differences you need to overcome.
|
|
|
|
Numonohi_Boi
Guest
|
|
« Reply #42 on: March 06, 2007, 06:56:28 pm » |
|
Instead of the "new romhacker's bible", you should call this the "Romhacking Gideon's Guide", I guess less people would catch the reference, but it still means the same thing.
|
|
« Last Edit: March 06, 2007, 07:57:36 pm by Panzer88 »
|
|
|
|
hippiejake
Guest
|
|
« Reply #43 on: March 06, 2007, 07:29:41 pm » |
|
I can see both sides of this.
Sometimes, people just need a bit of conceptual information to figure out how to do things. Like that bit that RedComet said, that explained so much to me, and it could be used on not only fonts, but sprites, etc.
Then there's the other side. They should be smart enough to figure it out once they reach that level. Also, the big mack daddies of hacking didn't have any explanation hand-fed to them. They went and did it themselves, and that mentality should be inspiring to those of us who rather suck, but want to get better.
I don't have an opinion. I'm just saying that this issue definitely could go both ways.
|
|
|
|
Suzaku
Guest
|
|
« Reply #44 on: March 07, 2007, 09:42:38 am » |
|
Hey, it could always just be the Romhacking Book of Gideon.
|
|
|
|
|