tag:blogger.com,1999:blog-60077947158072651252024-03-13T14:43:14.541+11:00Dreammaster's disassembly blogA blog about my interests in diassembling old adventure games, and what I'm up toDreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.comBlogger57125tag:blogger.com,1999:blog-6007794715807265125.post-38681957584729976392024-01-15T04:54:00.000+11:002024-01-15T04:54:39.755+11:00Christmas 2023 holiday roundup<p> Seems like it's been a full year since my last blog posting. I think that at this point, I'll just have to accept that I'm not very good at making regular postings. At least you all can take solace that my output of work on implementing support for new games in ScummVM remains unaffected. :)</p><p>So, what's been happening in the last year? Well, first of all, I finished implementing support for Might and Magic 1. The game itself was pretty primitive, but it was an excellent candidate for something I'd wanted to do for a while - not just reimplementing a game, but adding an enhanced mode that would breathe new life into it. In this case, I was able to use my experience with working on the Xeen games some years previously to add it's user interface into M&M1 as an option. I think it turned out pretty nicely:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEifVkOG6b8wyhzWkXBZ4BVWWdKdgCCyTF2KecQgMqnM4aLOK2dmsEbUr0Fjg8xNGP7WThho_t8j7Ma4rXVO6efVnll9L_AuuMtxtnIqvoFGF_8P2iUdCRe0Jtvrfg0sX5cwWA8tLA2rec1kBt3e9cxgtvCdd-Ax-N_cgAJSvVPeVVQ6f76a8Xhso166Pco" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="602" data-original-width="781" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEifVkOG6b8wyhzWkXBZ4BVWWdKdgCCyTF2KecQgMqnM4aLOK2dmsEbUr0Fjg8xNGP7WThho_t8j7Ma4rXVO6efVnll9L_AuuMtxtnIqvoFGF_8P2iUdCRe0Jtvrfg0sX5cwWA8tLA2rec1kBt3e9cxgtvCdd-Ax-N_cgAJSvVPeVVQ6f76a8Xhso166Pco" width="311" /></a></div><br /><br /><p></p><p>After that, I decided to revisit another game that had been bit-rotting, in this case for nearly twenty years - the M4 engine, which supports the games Orion Burger and Ripley's Believe It Or Not - The Riddle of Master Lu. This engine had special meaning to me, as it was the first engine I worked on collaboratively with others, shortly after I first joined the group. Back then though, things got mixed up with trying to have the one engine support both those games as well as the earlier MADS games like Rex Nebular in a single engine, and work stalled on it. Since that time, I'd re-implemented the MADS engine separately (though currently not all MADS games are supported), so decided it was finally time to tackle the M4 engine. And, just in time for Christmas, I finally finished implementing it with Orion Burger fully playable.</p><p>I've just announced an official testing period for the game. All the logic of the game was hardcoded, and had to be manually implemented. As such, my implementation of the various game rooms' logic suffered from many minor errors that had to be identified and fixed. Hopefully this testing period will have others discover any other remaining issues I'd overlooked, and the game will finally be officially supported.</p><p>Other than that, on a final note, as with previous years I choose to muck around with something different over the Christmas holidays. In this case, rather than fall back on working on the early Legend games yet again, I decided on the Wasteland engine, in particular disassembling Fountain of Dreams. I'd done some previous examination, and it looked like the codebase was fairly straightforward and simple (at least for the small parts of the executables I investigated). My time was productive, and I was not only able to reverse engineer and reimplement the entirety of the character creation screens, but also get a basic map display going. I give you, Fountain of Dreams startup in ScummVM:</p><p><a href="https://youtu.be/1EtVHEv5awE" target="_blank">ScummVM Fountain of Dreams</a><br /></p><p><br /></p><p>As you'll see from the attached YouTube video, I even added basic mouse control support for the character creation screens, just for the heck of it. At the moment, what's shown is pretty much all there is, but it's a firm starting point that further work can be done on.</p><p>Will further work be done? At this point I'm not entirely sure. I haven't firmly committed to working on Fountain of Dreams and/or Wasteland. It may be that I'll just putter around with them between working on other games. After all, I still have the drudgery of implementing all the game scenes' logic for Riddle of Master Lu to look forward to :P.. maybe I can use working on this as a way to unwind as I work my way through it all.</p><p><br /></p><p><br /></p>Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-7148132549734333212023-01-18T16:44:00.001+11:002023-01-18T16:44:49.969+11:00Christmas holidays roundup<p> Welcome to 2023, everyone. It's now mid-month in January, and I'm back from my long overdue Christmas holidays. Unfortunately :). Whilst I did end up having to spend some of my time taking care of work, I still got significant time to rest, read some novels and, of course, muck around with ScummVM stuff :). As with several previous years, I mostly ended up fiddling around with my disassembly of the early Legend games, and trying to re-implement them in ScummVM.</p><p>Thankfully, I was finally able to make some proper progress this year. Working further on my disassembly of the first Gateway game, I started a new iteration of the Legend engine in ScummVM using the latest meta engine/detection code, and gradually pulled in the work I'd previously done in the prior version of the engine. Then using it as a basis, I started implementing the code for the main text window. This included all the logic for caching text, line wrapping, and the -MORE- indicator. When it was done, I finally had the basis for displaying game messages.</p><p>Once that was done, for the first time, I started looking at the actual engine implementation. And I made progress, stumbling on a set of methods that implement what seemed like a basic C++ class hierarchy in C, with a table of 734 "objects" that were each one of eight different categories/types. Each type having a given amount of currently unknown data values. The only field of these structures that is currently known is a "method index" into a table of functions. Whilst the bulk of objects have a unique function, there are a few that don't have one, or share the same method. This is also how I know the code was a manual implementation of classes, since the function for getting the method index is basically a big switch statement that reads the appropriate offset, which is different for each category. If it was proper C++, they'd have a common base class and the field would be the same for all of them.</p><p>And these methods being pointed to.. what are they? It seems to be nothing less than all the per-game custom logic. With a bit of experimentation, I found that the first category was for rooms. Changing rooms in game is basically a matter of changing the room object being pointed to, and executing some code in the room for entering it. There's a global "room object num" that has the starting room by default, but by changing the value during startup in the DosBox debugger, I was able to change the room I started in.</p><p>Funnily enough, I tried changing the "room" to one of the other category types, and realized that it's type was for items. It seems the core engine has some resiliency, since by setting it to the item, the game told me that I was now "in" the item, as if it were a room :). Presumably, the other categories may represent other types of game entries, like non-inventory scenery or room exits, and their internal functions will have the logic for looking at items, moving to new rooms, etc.</p><p>This is all a big deal, since it means that when I have time free to work on it, I now have a solid starting point for figuring out game logic, and the methods they call in the engine to do the various things. Once I spend a bit more time figuring out the sentence parser, I'll be able to rapidly progress in finally supporting the Gateway game and, likely in the future, other early Legend games as well. Though it's going to be a pain having to manually implement and test each and every object function in each game. It's a good thing I have the patience to do that kind of thing. Though come to it, if I figure out all the engine methods the game calls, maybe Ghidra can assist in quickly producing readable C code.</p><p>As a final treat, a screenshot of the current engine. The listbox, font drawing, and other elements were pulled in from work done in prior years, and they still need work. Mostly, I focused on the text area, and better understanding how the game works:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgGhHosqdSskzuC3W_uYSBnakEiDHiu_BlmPJpeKxUO0wwbCcqgZV5X2FkrBrcrQztJnvvS8XbrhnjK0n-OwcWDM8Dkykmu3bMV7CHCdUlhpdhyb8uBSDx3Siz9byyt5rSIXlBm11PYFkoR5UBNftc0qoquHpIhFclAQiiYST4_kXZqMY_AllemlFkc" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="617" data-original-width="781" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEgGhHosqdSskzuC3W_uYSBnakEiDHiu_BlmPJpeKxUO0wwbCcqgZV5X2FkrBrcrQztJnvvS8XbrhnjK0n-OwcWDM8Dkykmu3bMV7CHCdUlhpdhyb8uBSDx3Siz9byyt5rSIXlBm11PYFkoR5UBNftc0qoquHpIhFclAQiiYST4_kXZqMY_AllemlFkc" width="304" /></a></div><br /><br /><p></p>Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-47200978082730445852022-12-19T05:20:00.001+11:002022-12-19T05:20:42.847+11:00Seasons Greetings<p> With the end of the year upon us, I thought it was time to do a blog post. And then I double-checked, and realized that I hadn't actually done a blog post since the start of last year. Awkward! So, yes, it's definitely well past time that I did a new post :) Things have obviously moved on since the last time I posted. In that time, the AGS engine has been merged into master, and is now officially part of ScummVM. I also spent some finishing the game Chewy Esc From F5, and it now too is finally part of the official releases.</p><p>My passion has always been reverse engineering, and due to certain real world considerations, I wanted something simple to work on for the last six months. I ended up choosing Might and Magic 1 for several reasons. I had already done some minor initial work on it, and found the disassembly areas I'd already looked at fairly straightforward. Plus, we already supported the Xeen games. I figured that if M&M1 were supported, M&M2, which I think is closely based on 1, would also be easy to support in the future. Plus, Xeen was closely based on M&M3, so it would be easier to support that as well. So we could eventually end up supporting all the Xeen games 1 through 5. And since I know there's projects implementing M&M6 and later games, maybe one day they could be merged in, and ScummVM could support the entire series. And then maybe onto Wizardry and Final Fantasy :)</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3IW4oIhS965oVy5PJcxck0EBVYxaKGD_DnOs996Yr7JHRCpKNsq48Jc2Zk4xe1V0MSzhium40HOdnAGVsvAeHCNNwHhh4Of7e-mp_afRU5c6fms4jGwQ92HIXPzeWvNIx7N49dEJ2Bvpw__R8v4I8bDmV020RvZ4zX1YXYTQljqymZGgdb5Ox47v0/s781/scummvm-mm1-00000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="617" data-original-width="781" height="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3IW4oIhS965oVy5PJcxck0EBVYxaKGD_DnOs996Yr7JHRCpKNsq48Jc2Zk4xe1V0MSzhium40HOdnAGVsvAeHCNNwHhh4Of7e-mp_afRU5c6fms4jGwQ92HIXPzeWvNIx7N49dEJ2Bvpw__R8v4I8bDmV020RvZ4zX1YXYTQljqymZGgdb5Ox47v0/s320/scummvm-mm1-00000.png" width="320" /></a></div><br />.Plus, given my familiarity with the Xeen engine, I was very interested in the possibility of introducing an "enhanced" mode to M&M1 that brought the user interface of the Xeen games to the primitive user interface of M&M1. With cleaner visuals for the various dialogs and niceties like the minimap, it could encourage people who never played the original to finally try out the game that started the entire series.<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_9An1n-3LlfO9U2QxuRAUUabmc5udOjOiZSqyDnMijq9r4hDuI7MYQ0l-MuR36imTi2B5g3_EtPW9WaGGCPL5eyPZjet623QuEuwqLCGLDUt_Wsv8mNLbjTEpRNCiu0ZeuxFTL46sLt59EjulnHsTbtyRHT7vpR4pZsTQKDyONkyw5btNmKRa1ABR/s781/scummvm-mm1_enh-00000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="617" data-original-width="781" height="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_9An1n-3LlfO9U2QxuRAUUabmc5udOjOiZSqyDnMijq9r4hDuI7MYQ0l-MuR36imTi2B5g3_EtPW9WaGGCPL5eyPZjet623QuEuwqLCGLDUt_Wsv8mNLbjTEpRNCiu0ZeuxFTL46sLt59EjulnHsTbtyRHT7vpR4pZsTQKDyONkyw5btNmKRa1ABR/s320/scummvm-mm1_enh-00000.png" width="320" /></a></div><br /><p>So, how is my work on the game progressing? As of today, I'm finally closing in on finishing re-implementing the original. All the user interface and game logic is done and tested, and all that's left is debugging the combat subsystem. Just last night I finished fixing the code relating to party members attacking the monsters, and now I need to start working on the code for when monsters attack the party. Apart from that there'll just be general testing.</p><p>Though there are a few other limitations remaining. I wasn't able to figure out the PC speaker code it uses for sound, so the game will be soundless. At least for the original un-enhanced version. Since the original only had sound in a few places, like when bumping into a wall, or a few notes when a party member is killed in combat, I don't consider that as a major loss. I was also never able to properly figure out entirely how the original did image decoding and handled the EGA palette. The engine has it's own decoding that I figured out from trial and error, but it does mean that the monsters appear with the wrong colours. Unless I can figure out exactly how the original did the image rendering, I may simply have to use one of the online wikis to see the colors of each monster, and set them up in ScummVM to be correct..</p><p>Anyway, I'm hoping to at least get the remaining bulk of combat debugged before my Christmas holidays, and then worry about playing around with doing an enhanced version in the new year. As has been my habit in previous years, I'll likely find something else to do during my holidays. I'm kind of thinking to return to work on the Legend engine again. That seems to be my default fallback project. And who knows, one of these days I may even actually finish re-implementing one of the games :)</p><p>So seasons greetings to everyone, and I hope both myself and you all will have a good 2023 to look forward to.</p><p><br /></p>Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com2tag:blogger.com,1999:blog-6007794715807265125.post-42148989553500234972021-01-19T04:17:00.002+11:002021-01-19T04:17:37.464+11:00A Great Surprise<p> I only realized with the turning of the new year that I have characteristically, once again, fallen behind on posting interesting news in my blog. My bad. As many of you will already know, last year I got sucked into working on the Comprehend series of games as a sub-engine under Glk. They were an interesting early series of combination graphics and text entry games including Transylvania, Crimson Crown, and OO-Topos.</p><p>As has become a bit of a custom over the last few years, over the Christmas holidays I scouted around for an existing source code project to work on integrating into ScummVM, as somewhat of a break from the various disassembly-based work I frequently do throughout the year. And behold the result of my work:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnr3n6KmWxMxdNBabSRAjyWyz5MPW2pWwbZum1jw3xHs-WvyCCb_IESfAjpeOjmTpIL65nGv3m3YI8iaygnKNDxbtifSa3fNaB7itBX6mobt9HLBxxgUpCnX5E9VQdQ3HPd2Vhplevo7M/s960/scummvm-bcremake-00001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnr3n6KmWxMxdNBabSRAjyWyz5MPW2pWwbZum1jw3xHs-WvyCCb_IESfAjpeOjmTpIL65nGv3m3YI8iaygnKNDxbtifSa3fNaB7itBX6mobt9HLBxxgUpCnX5E9VQdQ3HPd2Vhplevo7M/s320/scummvm-bcremake-00001.png" width="320" /></a></div><br /><p><a href="https://www.youtube.com/watch?v=ULMOvrZSMnY" target="_blank">See here for a Youtube link</a> of a screen capture I did for the game starting up. The Black Cauldron Remake is a point and click remake of the early Sierra game of the same name which I did over a dozen years ago, before I started working in earnest on the ScummVM project. Having it supported in ScummVM will be satisfying.. harkening back to my earliest days of open source work, and bringing it all full circle. And the important part of the remake is that it was done in AGS.</p><p>That's right. There's now an AGS engine for ScummVM that's at least capable of playing one AGS game through in it's entirety. Late last year I broached the subject of ScummVM integration on the AGS Forums, and the response was positive. AGS is currently in transition between 3.5 and a major redesign for 4.0, so there wasn't a rapidly moving codebase to try and keep up with. This engine, like the standalone interpreter it's based on, should eventually support all the AGS games from 2.5 to 3.*. Luckily, this covers the majority of the popular AGS games The ScummVM implementation will be just another implementation of AGS, so it'll be an alternative to the standalone interpreter, rather than a replacement.</p><p>The engine isn't yet available on Buildbot, so don't yet go rushing to grab a daily build just yet. I plan to do a pull request shortly, so it'll likely be merged into master in a few weeks once it's approved. There's still a lot of cleanup work to be done, bugfixing for other games, etc. so work on the engine will be ongoing. I added detection already for a few other games, but they crash on startup due to other unimplemented methods.. Rather than importing Allegro and other libraries used by the AGS interpreter in their entirety, I created a mockup layer that either remaps Allegro calls to ScummVM equivalents, or implements a bare-bones equivalent for the project from scratch. But so far I've only focused on what Black Cauldron needed, so other parts will be implemented as I gradually test other games.</p><p>On a side note, for all the AGS games to work on non-Windows/Linux/Mac systems, all the various plugins games used will also need to be re-implemented in C++.. I can't just have ScummVM loading in DLLs like the original interpreter does. Thankfully, I've already been able to obtain the original source for the AgsCreditz plugin from it's author, Hopefully other plugin authors will be likewise generous when I get to them. When done, this'll mean AGS games will be playable on more systems that AGS wasn't previously available for.</p><p>One day in the near future, expect to play some of the classic AGS games like the AGDI Sierra remakes in ScummVM :)</p><p><br /></p>Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com4tag:blogger.com,1999:blog-6007794715807265125.post-15793206830781794542020-05-29T06:06:00.000+10:002020-05-29T06:06:09.926+10:00A retrospective and a look forward<div dir="ltr" style="text-align: left;" trbidi="on">
Hey everyone. I hope everyone has done okay so far this year, considering the circumstances. On a more positive note, you will have seen we've recently announced the testing period for three of the games in the Ultima series of RPGs.. Ultima IV: Quest of the Avatar, Ultima VI: The False Prophet, and Ultima VIII: Pagan. Thanks go to the original projects and all the developers that previously put work into them, as well as in particular to mduggan, who has put a lot of work into Ultima VIII since it was merged in.<br />
<br />
So now that we've got three of the Ultima series merged in, are more to follow? That's actually a bit of a tricky question. Of the remaining games<br />
* Ultima IX: This is more in scope for ResidualVM than ScummVM. I remember reading previously that the "Ultima Source Code Offline Archive Project" had obtained the code for Ultima IX. So there may be a future possibility of it being reimplemented in ResidualVM.<br />
* Ultima VII/SE: Exult is currently an independent project with active development and users. So there's no immediate plans for integration until and unless the developers decide they don't plan to do any further development, or decide themselves the time is right to merge it into ScummVM<br />
* Underworlds: I'm vaguely aware that there are projects for the two Underworld games, but I haven't researched how complete they are. Despite their 3d look and feel, these games were done in the DOS days where all 3D was done in software. Because of this, unless the reimplemented code specifically uses 3D APIs, then it would actually be in scope for ScummVM rather than ResidualVM.<br />
* Ultima I: I have previously been working on a reimplementation of Ultima 1. It's currently on hiatus, but I eventually plan to return to work on it. As with Ultima IV, I hope to allow for an enhanced version of the game that uses better tilesets<br />
* Escape from Mt Drash: There is source for a PC reimplementation <a href="https://github.com/gondur/ultima_mtdrash">available here</a>. So support for it could be added. Even if the only real relevance to the Ultima series was a marketing ploy, it is still technically an Ultima game, so support for it should be added I guess for completeness if nothing else.<br />
* The rest. This is where it gets difficult. Apart from Ultima V, the earlier Ultima games were, honestly, less sophisticated, and I'd be less likely to enjoy playing them. So without pre-existing projects for them, it becomes harder for me to justify spending all the time reverse engineering them from scratch for support. It's even more frustrating in that it looks like there's been previous people that reverse engineered the earlier games, such as <a href="http://ultima2.voyd.net/">ultima2.voyd.net</a>. But since they never released source code for their work, it really doesn't help us that much. If any of those authors happen to read this, or anyone knows how to get in touch with them, we'd really appreciate getting our hands on their work. Otherwise, it's unlikely the earlier Ultima games will be supported any time soon. Or if the previously mentioned Offline Archive Project has source for any of the earlier games, we'd welcome getting access to them.<br />
<br />
So in any case, what are my plans now that those three games are working their way towards being officially supported? Well, first of all, I kind of fell into working on integrating the <a href="https://github.com/RyanMallon/recomprehend">Recomprehend engine</a> into ScummVM. Given the simplicity of the engine, it seems like a fun project to unwind with after all the work on the Ultima games. I don't anticipate it'll take all that long. There are apparently some minor issues in the screen rendering that may require reverse engineering bits of the original games for comparison, but again due to the simplicity of the games that shouldn't be too hard.<br />
<br />
Next after that, I kind of want to get Glk into a more officially completed state. Technically all the sub-engines in it so far could be considered complete on their own, but the exception is the Frotz interpreter.. the current one didn't properly support all the version 6 graphical games. I would really like to support them properly. So I've tentatively started working on the newest version of Frotz to convert it to run on Glk from scratch. At this point I'm not sure if this new work will be able to be cleanly set up to handle both v6 and non-v6 games.. it may end up being easier to keep the existing version for non-v6, and just focus on v6 games for this. I'll have to wait and see.<br />
<br />
After from that, Glk should be suitable for an official release. Though there are other sub-engines I could work on in the future, such as TADS support. The TADS 2 version in the codebase doesn't currently work properly, and when I previously tried starting on converting TADS 3 to compile under ScummVM, it led me down a rabbit hole of compilation issues. Support for both will likely be on hiatus for work again in a future release.<br />
<br />
Apart from them, what else this year? Good question.. I had had some tentative plans to look into early AGS games support.. years ago I did a remake of Black Cauldron in AGS, and it'd be very satisfying to finally support it in ScummVM. However, just the other day Strangerke made an intriguing discovery. A non-interactive demo for Shannara, one of the later Legend Entertainment games, has what looks like debug information at the end of the executable (though it's not in a known format IDA can automatically handle). As long time readers will know, supporting the Legend Entertainment games has been one of my long term goals for years. Whilst the demo itself is effectively just a slide show, the debug information makes it look like they implemented the demo using large chunks of the full game's engine for rendering the demo scenes. In fact, there may even be a bunch of code from the main game the demo doesn't even use that wasn't optimized out. As such, it may prove a valuable insight for understanding the full Shannara game, and from that, the earlier games of Companions of Xanth and Death Gate.<br />
<br />
So it may well be that I'll end up switching my focus back to the Legend games for a while. We'll have to wait and see. Considering that I ended up working on Comprehend by chance, there's always a possibility that something else interesting may pop up :)<br />
<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-37220479268502910472020-02-10T15:40:00.003+11:002020-02-10T15:40:38.811+11:00It's ScummVM ULTIMAte<div dir="ltr" style="text-align: left;" trbidi="on">
A little over a week ago, it finally happened. I had combined my various work on the Ultima games into a single branch, and it was merged into master. This means Ultima VI: The False Prophet (from Nuvie), and Ultima VIII: Pagan (from Pentagram) are now playable in daily builds. \o/. Ultima 1, as before, is startable, but still a work in progress, and work on that is on hiatus right now. The Ultima8 sub-engine already has a splash-screen, provided by Dominus Dragon, added to give credit to the Pentagram team that wrote the original code:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAVlIHosumhCIkjk1TbjBirftu02gwHiUAY3FMWUn4ZB3ENW3kbGwwJ1ezDW77N4EfhieI-Hg_ptpA2WMTivhZjikPROBhHkz7FS9ojzzJd5CIjCJQuvDaLaYjlLLCo-ytCuJjaCcLjXY/s1600/pentagram.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAVlIHosumhCIkjk1TbjBirftu02gwHiUAY3FMWUn4ZB3ENW3kbGwwJ1ezDW77N4EfhieI-Hg_ptpA2WMTivhZjikPROBhHkz7FS9ojzzJd5CIjCJQuvDaLaYjlLLCo-ytCuJjaCcLjXY/s320/pentagram.png" width="320" /></a></div>
<br />
Nuvie will similarly get it's own splash image eventually to recognise it as well.<br />
<br />
However, I'll hasten to say, that any Ultima fans should not necessary come running to play either game just yet. I'm still not quite ready to announce an official testing period for either game just yet. I plan to be doing my own testing of the games first, and it's always possible that there might be minor issues or changes to the savegame formats. So any experimenters be warned to play at your own risk.<br />
<br />
I'd originally intended to move straight onto testing of Ultima VIII post-merge, but I got slightly sidetracked adding new functionality to the base Engine class that all game engines derive from. Functionality that future game engines will benefit from, including the Ultima engine.<br />
<br />
1) The first of these is extended savegames. There was a recently added enhancement to savegames to take care of the drudgework of storing the savegames' names, thumbnails, total play time, etc. However, it still required engine writers to manually call various methods manually in their savegame code. With the addition of new methods in the Engine class that builds on this, this is all now taken care of automatically now, as well as opening and closing savegame files in general. All that needs to be done in future engines is overriding of two methods: loadGameStream and saveGameStream, which get passed a handle to a savefile for reading/writing it's contents, and the Engine takes care of everything else, including the savegame description and thumbnail.<br />
<br />
2) The second is improved handling for autosaves, still under code review before inclusion. It adds autosave support to almost all the game engines. ScummVM has a global option for an Autosave interval, but it previously required each engine to implement their own code for autosaves. Because of this, only about 1/5th of the engines actually implemented autosave. When these changes are integrated autosaves, when turned on, are provided by practically all game engines. Only a few engines, like DreamWeb, that don't allow saving the from Global Main Menu won't support autosaves.<br />
<br />
3) The third one, also under review, is an improvement to engine debuggers, the console window that pops up in many engines when you do Ctrl-D. Engines now just have to create their own debugger and pass it to the Engine to take care of, and it handles all the work for opening it when necessary, and updating it. One of the benefits of this, apart from standardization & centralization, is that when errors occur now, the debugger window is more guaranteed to open up to show it. Even if the engine doesn't have it's own debugger, the base Engine class will create one on the fly. This means it will be easier for users to see fatal errors, rather than ScummVM simply terminating.<br />
<br />
With these three done, I think my "base Engine binge" is fully tapped out. Barring any further issues or suggestions coming up in the pull requests for points 2 and 3, I'll probably take a few days to relax. Maybe do some further messing around with cleaning up Starship Titanic code, before I move back to my original plan of doing a test play-through of Ultima VIII.<br />
.</div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-23024442298054089772020-01-07T16:49:00.000+11:002020-01-07T16:49:29.840+11:00Post New Years report<div dir="ltr" style="text-align: left;" trbidi="on">
Once again, it seems I fell out of habit of doing regular blog updates. Though hopefully you'll at least be ameliorated by the fact that I have at least been busy over the last few months. With the start of 2020 finally upon us, I thought it would be an opportune time to summarize what I've been up to.<br />
<br />
First of all, the ScummVM Glk engine has been going well. Before I put it on hiatus late last year, I'd already converted over 2/3rds of the existing engines to run under ScummVM, with cleanups and hooking into the ScummVM save system, etc. At this point, there's only a few sub-engines from the original Gargoyle suite left, chief among them TADS which, due to it's complexity, will likely take some time to clean up so that it compiles properly as part of ScummVM. I had previously done a preliminary conversion of TADS 2, but there were problems with immediate crashes that I couldn't figure out. I'm hopeful that doing a fresh conversion of both TADS 2 and 3 at the same time may result in a better outcome this time.<br />
<br />
Next comes my work on Ultima. Back at the start of December, I'd had plans to keep working on Glk right up to my Christmas holidays. However, after a non-serious prodding from Dominus Dragon about Pentagram (the engine for playing Ultima 8) still not yet being supported, I started looking at what it would take to convert over the engine. And ended up getting hooked on it, to the exclusion of all else for the next several weeks. So below, I present a screenshot of Ultima 8 running in ScummVM:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV8k78FPgdIcc_WIQdIXE8PrLhDeFg_8keNApID770mUwi2zGXYyt4_gtXD6Fw28jBQS3Mw9fxzCrFM8WmsZI0C8hhMVwbPQ9h6kov5m2EJrvHnlyCpOguJxczlQ8Xjuf9DDvMGgj6iWI/s1600/scummvm00000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV8k78FPgdIcc_WIQdIXE8PrLhDeFg_8keNApID770mUwi2zGXYyt4_gtXD6Fw28jBQS3Mw9fxzCrFM8WmsZI0C8hhMVwbPQ9h6kov5m2EJrvHnlyCpOguJxczlQ8Xjuf9DDvMGgj6iWI/s320/scummvm00000.png" width="320" /></a></div>
<br />
<br />
As of Christmas, the only area I had left to update/correct was that of the music player.. I still need to figure out how to hook it into ScummVM. Otherwise, everything else seems to be working fine.. I was able to play the introduction, get through the docks cutscene, and then enter the city. I haven't yet really put the game through it's paces beyond that. It's better to wait until I'm happy that everything is properly implemented.<br />
<br />
And I got distracted yet again, although this time intentionally. At this point, my Christmas holidays had been reached, and I really wanted something a bit more fun to work on than just bugfixing and playtesting of Ultima 8. Trying to work my way through all those caves and jump platforms wasn't my first choice for my R&R activities.. I much preferred working on the engine itself than actually playing the game.<br />
<br />
Back when I started getting hooked on working on Ultima 8, and realized how fun it was, I again set my sights on Nuvie, the engine for Ultima 6 and spinoffs. I'd previously asked the lead developer, Eric Fry, nearly two years back about integrating Nuvie into ScummVM, but at that time he still wanted to work further on Martian Dreams on it with it still being the familiar stand alone project. Trying again in December, I got no response, even after several weeks, and asking a co-developer, again Dominus Dragon, to also try contacting him. Since it had also been several months since any commits had been done, I decided that I'd done my due diligence, and it was finally fair game.<br />
<br />
So over the holidays I put Ultima 8 aside, and worked on integrating Nuvie into ScummVM as well, creating a shared "ultima" engine that contains both Ultima 6 and Ultima 8. And, eventually, hopefully the other remaining Ultima games as well. Including my in-progress reimplementation of Ultima 1, which I originally put on hiatus way back when I first got agreement to merge Gargoyle into ScummVM.<br />
<br />
So over the holidays I had further fun of converting the Nuvie engine. To make things easy for users, the new ultima6 subengine has two different detection entries, with "original":<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqcXlWRmriE1FHcXiTWak_zDY9pLMzkittJtJfiEde9DSR2PFtc5CbUR1MYEABtN0UvnXGC_6bvzWs5Btsy8WD4RKP8n4YOqzi4Zbuigd3wSyHulyB1lSTCpr_yzLixV2SXadHtgr2qkM/s1600/scummvm00001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqcXlWRmriE1FHcXiTWak_zDY9pLMzkittJtJfiEde9DSR2PFtc5CbUR1MYEABtN0UvnXGC_6bvzWs5Btsy8WD4RKP8n4YOqzi4Zbuigd3wSyHulyB1lSTCpr_yzLixV2SXadHtgr2qkM/s320/scummvm00001.png" width="320" /></a></div>
<br />
<br />
and "enhanced" versions:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmUjGxZ5agM5sCp2muhedA9Yu6S9e9_luqm_u8XfOJXmjTS3331ruxcdwA5QHDQymezdvxEQDwWfLIfLoNDraDSenyAmqlStl0DZ1yXpk2FnggoBD6KYMGDNlct6SlduLkGIRPE7QHoHQ/s1600/scummvm00002.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="1280" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmUjGxZ5agM5sCp2muhedA9Yu6S9e9_luqm_u8XfOJXmjTS3331ruxcdwA5QHDQymezdvxEQDwWfLIfLoNDraDSenyAmqlStl0DZ1yXpk2FnggoBD6KYMGDNlct6SlduLkGIRPE7QHoHQ/s320/scummvm00002.png" width="320" /></a></div>
<br />
<br />
As you can see, original gives the look and feel of the original game, whereas the Enhanced version automatically sets all the Nuvie options for enhanced graphics, full screen map, and Ultima 7 style gumps. So users will easily be able to play in an enhanced mode without worrying about fiddling around with configuration options.<br />
<br />
As of now, the game starts up and runs. It still needs some work testing the various areas, and making sure it all works properly. In it's case, the sound code was a lot easier to convert than for Pentagram. It actually already used copies of the ScummVM mixer and stream classes. So it was a pretty straightforward process to simply remove them, and remap the code to use the original ScummVM classes instead. Such a relief :).<br />
<br />
A downside, though, is parts of the engine, such as the cutscenes, were written in Lua rather than C code. This has disadvantages for properly supporting the game in ScummVM. First of all, I've had to duplicate the entire Lua runtime from Nuvie just to run the scripts, as the version of Lua we have in common/lua/ can't execute the scripts, giving an error when I tried. Plus, when the intro sequence is running, the system isn't properly updated, which means I can't drag the window or, more seriously, exit ScummVM by closing the Window. Long term, I'm likely going to have to rewrite the Lua scripts as C++ code and classes, so Lua can eventually be entirely removed. That will give me greater flexibility to properly handle events for the cutscenes. Also, for enhanced mode, I set the graphic mode to double size of the original, so rewriting the cutscenes in C++ would also make it easier to either stretch the cinematics, or temporarily switch back to 320x200 mode for them.<br />
<br />
<br />
So where am I going from here? Well, the larger portion of my time is going to return to working on Glk again. Given it's advanced state, I'd be happier getting it finally done and properly supported. At least for the engines that were already part of the Gargoyle suite. Converting other non-Glk-compliant interactive fiction engines can wait until some other time in the future. I'll also spend some smaller portion of my time still working on Ultima 6. For now I'm not worrying about converting the Lua scripts.. I'll first be focusing on ensuring that the rest of the engine is running solidly under ScummVM. After that I can worry about the scripts. We'll see how I go :)<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com2tag:blogger.com,1999:blog-6007794715807265125.post-26584391836011263392019-01-24T13:52:00.000+11:002019-01-24T13:52:01.163+11:00Frotz work in ScummGlk is ongoing<div dir="ltr" style="text-align: left;" trbidi="on">
Work on the ScummGlk engine is ongoing. Since Christmas holidays I've been primarily focusing on adding support for the version 6 graphical Infocom games to the Frotz sub-engine, with particular focus on Zork Zero to begin with. Whilst I don't count the version 6 games among the best of the original Infocom line-up, ScummVM has always been about providing as good a coverage of old games as possible. So from my point of view it's worth spending the extra time adding support for the remaining Infocom games now rather than later, and only moving onto the next sub-engine once that's done.<br />
<br />
Below is a screenshot of Zork Zero as of when I came back on the 7th January:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSitFWkkh2DmH9BfGRnmIBmT2Yuf2k8ZQfcMjJ4R4k3vFvLbZXaK48VWIF2tbb0dZO8H2rqC72BSBuH67O8fbT5wGUfmf7HmLY_9Zdhkw1QdxCEMHffb0_WxPvmH6eywZHskVM8NErjlQ/s1600/scummvm00004.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSitFWkkh2DmH9BfGRnmIBmT2Yuf2k8ZQfcMjJ4R4k3vFvLbZXaK48VWIF2tbb0dZO8H2rqC72BSBuH67O8fbT5wGUfmf7HmLY_9Zdhkw1QdxCEMHffb0_WxPvmH6eywZHskVM8NErjlQ/s320/scummvm00004.png" width="320" /></a></div>
<br />
<br />
As you can see from the screenshot, the background is starting to be properly rendered, and the 'A' and location glyphs are properly rendered and interleaved with the text. The status bar still needs some work, though.. although it is being rendered, it's then being completely wiped out by the status bar window, which isn't even displaying any text yet.<br />
<br />
Right now, I'm focusing on adding the 8x8 fixed size font used in the original games. If you look closely at the screenshot above, the current resizable TrueType font doesn't really look crisp at that low a resolution, and is somewhat blurry. I'm currently making a tool to use the same font data that Frotz has, and produce bitmaps to add to the fonts Zip file for the different 8x8 font styles (normal, bold, emphasized, and bold & emphasized). It has had it's share of issues:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEielCOogO5xOP1lq28y7U8AVO3SvSbcM_sG-S852AF2Wum9N8BUzYg48X7bcrp1bL4P9UdVkbqjgWv2VA3wv95MrDcQYHFa5dy_JeqN8e9FFnhI-tJn1fnRBdJ2D1tEUls3I_g7EkrGC-w/s1600/scummvm00005.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEielCOogO5xOP1lq28y7U8AVO3SvSbcM_sG-S852AF2Wum9N8BUzYg48X7bcrp1bL4P9UdVkbqjgWv2VA3wv95MrDcQYHFa5dy_JeqN8e9FFnhI-tJn1fnRBdJ2D1tEUls3I_g7EkrGC-w/s320/scummvm00005.png" width="320" /></a></div>
Somehow, I don't think I've got it quite right yet.. :)<br />
<br />
Anyway, progress is gradually being made, and I hope to have Zork Zero fully supported soon. The other v6 games currently error out immediately during startup, but I'm hoping further adding support for them will be a simpler matter.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-54563942938632936962018-12-23T12:12:00.000+11:002018-12-23T12:12:01.232+11:00Seasons greeting and off on holiday<div dir="ltr" style="text-align: left;" trbidi="on">
Seasons greetings to all, and I hope everyone has a happy time as we finish 2018 and look forward to the beginning of 2019. As I sit in the Los Angeles airport writing this, I'm reminded yet again of how much fun I have doing reverse engineering, since even the smallest thread of work can lead you in unexpected directions as you trace the flow of what methods call a given method, and what methods it calls.<br />
<br />
For something to do, I opened up my Companions of Xanth in progress disassembly to do a bit of pottering around with it. Just by chance, scrolling through the disassembly, I noticed a call to a method I was pretty sure took in a string, but in this case was passing in a value. I then quickly realised it was an offset n the data segment that hadn't been properly resolved, so I did that. The message in question turned out to be a generic "insufficient memory". So from it's usage, I was able to identify the entire method it was in.. it's called during startup, and ensures that there's enough free memory to play the game. Ironically, it looks like the code to actually display the message if there isn't enough was broken, as evidenced when I ran the game in DosBox debugger.<br />
<br />
From there, identified a method just before the message call that took in a number and passed out a string.. it wasn't too hard to identify it as a number formatting routine. I confirmed it in the DosBox debugger by putting a breakpoint in the machine code, and seeing the result from the method. I then noticed that there was one other method calling it, so I was intrigued to find out what it was. It turned out it too was a block of code in a method that printed out the free memory, in a method that looked like a big switch based on a single value. It looked like a keypress. Playing the game, I was able to confirm that pressing 'M' on the keyboard did call the code, which printed out the amount of free memory in the status area.<br />
<br />
This was a good woo-hoo method. I had identified the method for handling keypresses. Likely, some of the other keypresses do fairly identifiable things, like saving and loading, so knowing where each key's code is, I'll be able to trace into what they do, and what methods they call. In fact, the free memory printing for the 'M' key already prove dividends. From it, I was able to identify the method that cleared the status area, and a version of 'sprintf' for printing text in the status area. In the future I'll be able to look further into other places that call the method to add lines to the status area, and start identifying other parts of the game code. For example, looking at an item produces a description in the status area, so by setting a breakpoint in the status text writing code, I'll be able to identify what method calls it, and start identifying the structure of items that likely have a "description" as one of it's fields.<br />
<br />
All this progress today came from a minor diversion to pass the time sitting in the airport. The notice of an incorrect parameter to a method call has unraveled more of the game's secrets, and giving me multiple further areas I can investigate. A nice way to start the holidays.<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-53588691861547865372018-12-15T07:18:00.000+11:002018-12-15T07:18:00.180+11:00It's time for a very verbose change of pace<div dir="ltr" style="text-align: left;" trbidi="on">
One of the things I like about my work on ScummVM is that it's a hobby rather than a profession, which gives me the freedom to switch what I'm working on when I desire. I've previously had encouragement to resurrect my Gargoyle engine work from about a decade ago. It was an early attempt to add support for Frotz into ScummVM, and at the time it was rejected due to multiple reasons. It was specific only for Infocom/zcode games, lacked clipboard, truetype fonts, and Unicode support. Most of these things have since been solved by the ScummVM framework itself, which now supports all of them. So I thought, why the heck not, and decided to work on it again.<br />
<div>
<br /></div>
<div>
However, my original Gargoyle engine being limited to just Zcode was a problem. Not only wasn't it extensible to other IF systems, but it would also take work to properly reimplement it to support Unicode. In the end, it was just simpler to scrap it all, and start from scratch. Luckily, there's the very convenient Glk specification, that was specifically created for easing porting the various Interactive Fiction interpreters to other systems. I knew that by supporting it, it would make it easier to create a framework that could support multiple different interpreters as sub-engines.</div>
<div>
<br /></div>
<div>
So I created a new Gargoyle engine, which I've since renamed it to ScummGlk to avoid ambiguity with the stand-alone Gargoyle project which also implements the Glk specification. After over a month of work, I've created an engine that implements the bulk of the API, and implements two sub-engines under it so far.. Scott for Scott Adams games, and Frotz for playing Z-Code/Infocom games. Scott was useful as a test case, due to it's simplicity. With it as the first sub-engine, I was able to focus on getting the ScummGlk core working. Then Frotz was a fairly obvious second choice, given it's popularity and large number of available fan games.<br />
<br />
As of last weekend, the review period was completed for the engine, and it was merged into master. Daily builds now include support for both sub-engines. I've spend the last week fleshing out detection entries for all the ZCode games on the if-archive, so all the games will be automatically detected by the launcher. I'm going to wait on announcing an official testing period until I finish support in Frotz for V6 games. But if you can't wait, you're certainly welcome to jump the gun and try playing some games. Maybe one of the early Zork from GOG, or the immortal classic Freefall :)<br />
<br /></div>
<div>
Having the engine included into master will be a real nostalgia trip for me. Even prior to my original Gargoyle module, and before I even started working on implementing adventures in ScummVM, I spent several years as part of the Interactive Fiction community. Indeed, the Hitchhiker's Guide to the Galaxy was one of the first computer games I ever had. I spent way too many hours getting frustrated with the game's puzzles.</div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-4643330233488293162018-05-21T11:42:00.000+10:002018-05-21T11:42:30.278+10:00It's an UltimaTE unwinding<div dir="ltr" style="text-align: left;" trbidi="on">
After all this time, support for the Might & Magic Xeen games is finally close to being finished. It's currently the official testing period for all the Xeen games, and I've already had a whole bunch of useful bug reports, which I've all taken care of. At this point, it's almost certain that the Xeen engine, and games, will be supported in the next official ScummVM release. It's a nice feeling. More than any other game engine I've worked on, the Xeen engine was the one that got put on hiatus the most number of times. It's very satisfying to finally complete it.<br />
<br />
What's in store for me now? Well, I do want to spend some time properly unwinding and actually playing some games. They've only just come out with the sequel to Pillars of Eternity, which I'm now getting into, and plus I have a backlog of other RPG and adventure games that I want to have a chance to properly play. So that will probably keep me occupied for a few months at least. I've also got a move from the US East Coast to West Coast coming up next month. So various preparations are consuming my time, and it's all to the good that work on Xeen has already finished.<br />
<br />
Not that I'm totally stopping work on ScummVM stuff. As with the early days of work on Starship Titanic, I started doing some casual reverse engineering during my lunch breaks. By happenstance, I experimented with the Ultima 1 executables, and become enamored by how simple they were to reverse. Both due to the simplicity of the game (considering how old it was), and the way the code was implemented. It was just the thing for doing casual reversing work in chunks of an hour at a time.<br />
<br />
As a result of this, I've already implemented a great deal of foundation code in ScummVM for supporting the game, with a long term eye for having a clean engine structure that could support all the early Ultima games. Here's some screenshots of the player within the overworld, a city, and within the dungeons:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9PtqBfOEke7V0xBURL9I54mLLm-lHIClvex3SOUVb2GtePAdLVZ-KSdNePDL31BKSYTjEy7PIJQxYa_VgSIsqPUqiY8zPmbKeQIbIuY1GOlEsVoZwAVzdG2WkigW0gjWaK0faRXHYEaw/s1600/scummvm00001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9PtqBfOEke7V0xBURL9I54mLLm-lHIClvex3SOUVb2GtePAdLVZ-KSdNePDL31BKSYTjEy7PIJQxYa_VgSIsqPUqiY8zPmbKeQIbIuY1GOlEsVoZwAVzdG2WkigW0gjWaK0faRXHYEaw/s320/scummvm00001.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLf3M8h2kGjQIOZbkfkC8ueJKifTP4PKUDrHm_jDhFBe2hAjvrDAt2UMTqSFu9GJg08k5dgP1CjHa7Qle6KWuoqSZRoa1iXMlFuywIPjq3mWMsQVC7Wv6Qwj-rZnPE_9t8S1JtVM1SoN8/s1600/scummvm00003.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLf3M8h2kGjQIOZbkfkC8ueJKifTP4PKUDrHm_jDhFBe2hAjvrDAt2UMTqSFu9GJg08k5dgP1CjHa7Qle6KWuoqSZRoa1iXMlFuywIPjq3mWMsQVC7Wv6Qwj-rZnPE_9t8S1JtVM1SoN8/s320/scummvm00003.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV4Uc_G9aqZEPG6Ad6_2QEBC6_h3c6vW_6dEpbza3mvs_gLmaZIaoHMaTvmB4l5cE0iJlx9kXrrNgJPpQAj3HvWT1QeB4mT5dpRiRCDhV3LjPkR6u6sSbZF5jaOy6aDCzkF6p9Jv9TS-o/s1600/scummvm00000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="960" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV4Uc_G9aqZEPG6Ad6_2QEBC6_h3c6vW_6dEpbza3mvs_gLmaZIaoHMaTvmB4l5cE0iJlx9kXrrNgJPpQAj3HvWT1QeB4mT5dpRiRCDhV3LjPkR6u6sSbZF5jaOy6aDCzkF6p9Jv9TS-o/s320/scummvm00000.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
At this point, I've got basic movement functionality working. The player can roam around the map, entering cities, castles, and dungeons. Dungeon movement also works, as does going up and down ladders, so the dungeon can be further ventured down into, or left entirely. That's it at the moment, though; it doesn't yet have any combat, magic, inventory, character interaction, etc. See this <a href="https://youtu.be/_YuxIeniEcM">Youtube video</a> for an example walk-about that covers venturing into a dungeon, and then heading over to the city of Britian.<br />
<br />
Don't expect too much more work on this in the near term, though. As I said, this is more a side thing, and I definitely do want to spend some time unwinding and playing some games. When I get my fill, later in the year, then we'll see. At the moment, it's looking strongly like I'll keep working further on this. I've some crazy ideas about not only adding support for the game, but also reusing the Ultima VI tileset to create an "enhanced" Ultima 1 experience that's more visually pleasing, and rework the controls and user interface to be likewise more player friendly. I know that oh, pretty much everyone who ever played their way through the space simulation section would much appreciate mouse control for aiming at enemy vessels. :)<br />
<br />
I kind of consider the RPG series of Ultima, Might & Magic, and Wizardry the triumvirate of major RPGs from the last century and, now that ScummVM is embracing RPGs in addition to adventures, I'd love to see them all under the ScummVM banner. Now that at least some of the earlier Might & Magic games are supported, it may well be time to start in on Ultima support :) Of course, I'm not entirely abandoning working on adventure games. I'd previously made some promising initial progress in reversing some of the Legend Entertainment games.. Companions of Xanth and Gateway. Irrespective of how much time I end up devoting to working on Ultima I, I may end up also devoting some of my time to those games as well.<br />
<br />
Anyway, that's probably enough for now. I still want to get in some Pillars of Eternity 2 before it's time for bed. :)<br />
<br />
<br />
<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-38340773850853269212018-04-14T00:56:00.001+10:002018-04-14T00:56:26.556+10:00Working my way through Dark Side<div dir="ltr" style="text-align: left;" trbidi="on">
I've spent the last week with an in progress working my way through Dark Side of Xeen. As of last night, I've mapped out the bulk of the southern side of the map up to the mountains, released the first stage of Queen Kallindra's castle, finished the Temple of Bark, and am just winding up clearing Sandcaster and it's sewers of enemies. I've identified and fixed multiple problems as I went, and not all of them were specific to Dark Side.<br />
<br />
For example, one bug I'd been meaning to fix for a while was finally resolved - when monsters were triggered to hate & attack the entire party at the same time, they were originally only ever attacking the first party member. This made the Dragon Caves on the Clouds side *so* much easier.. all the dragons would wail away on my first character, even after he was dead, leaving the rest of my party to kill them at their leisure. By the time I left the dragon caves, my first party member was below -5000HP :) Another, ironically, coincidental script problem meant the dragon containers didn't tax me when I walked past them, so I didn't even have to worry about losing gold and gems.<br />
<br />
I also fixed some bugs in the sound system that could cause crashes when multiple sound effects were played in rapid succession, and a bug in the combat system that could case a crash when recalculating the order that characters and monsters fight in based on their speed. The later I'm particularly hoping will help resolve the outstanding rare crashes I was previously getting with extended combat. It may be naive.. I'll have to wait and see whether any further crashes occur.<br />
<br />
So yeh, things are progressing, and the engine is getting more and more stable; I'll keep working my way through the game. I've got a long weekend coming up tomorrow, but honestly, I'm not sure how much time over the weekend I'll end up playing the game. I think I'll keep the bulk of it for some R&R and doing other things. We'll see how it goes.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-77197287989854789722018-04-10T03:47:00.003+10:002018-04-10T03:47:47.779+10:00Clouds of Xeen is now completable<div dir="ltr" style="text-align: left;" trbidi="on">
Good news on the progress of the Xeen engine in ScummVM. As of the weekend, I finally finished my first playthrough of Clouds of Xeen! I finally got to give smackdown on Lord Xeen, and watch the ending cutscene. Which I've admittedly previously seen multiple times when I was first implementing the code, but there was the emotional satisfaction of seeing it properly, rather than just using the mirror cheats. Though I will confess that I got so overeager to finally finish the game, that I lacked the patience to do it properly (with well/fountain stat raises), and instead used my debugger cheats instead. But it doesn't take away from the fact that the first game is confirmed as completable. \o/<br />
<br />
Now, it's onto Dark Side. There were actually more bugs due to changes done for Dark Side (code I'd implemented but not yet tested) than I anticipated. I already fixed multiple different bugs on the weekend that were causing hangs and crashes. Currently I'm looking into a somewhat complicated problem with how wall decorations are handled.. it causes a crash that makes Ellinger's Tower impossible to complete. So Dark Side definitely isn't completable, and needs further testing. On the plus side, though, it shouldn't take as long to test and finish Dark Side than my Clouds testing took, since Dark Side benefits from all the fixes I've already done for Clouds.<br />
<br />
At this rate, my initial play through & tests of both Dark Side and Swords should be done within the next month, and the game will likely be stable enough for a public testing period. Not long now :)<br />
<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-76231571288283788592018-04-03T01:12:00.000+10:002018-04-03T01:12:24.253+10:00Work on Xeen is up in the air<div dir="ltr" style="text-align: left;" trbidi="on">
As tempting as it would be to write this post yesterday on April Fools, I've left it to today to make a status update on my playthrough through Xeen. Work has been proceeding well. I've fixed lots of bugs that were identified by both myself and Dark-Star, who's been assisting me with preliminary testing. As of yesterday, I've:<br />
- Finished the Dwarf Mines<br />
- Mostly finished Rivercity (except for the Knights behind the training area)<br />
- Completed the Witch Tower. Hence the post title.. the levitate spell works, and I've been walking around on the clouds :)<br />
<br />
Many of the bugs were crashes and hangs, so it highlights that the game still isn't ready for general public testing. The Witch Tower, for example, required two fixes to get it to work.. one where I wasn't setting whether the party had the key (so it was never letting you in), and one where I incorrectly implemented the "return from subroutine" script opcode, so trying to go back into the Witch Tower from the stairs in the Witch Tower clouds hung the game. Ironic that both ways into the tower were glitched. :)<br />
<br />
I think combat is also more settled now. Previously, there were various crashes, and none of the combat spells worked. Now, though, I'm able to properly cast them, so the monsters around Rivercity are much more manageable to defeat. Though I did still experience a problem once where the party wasn't immediately registered as dead after all being killed, so obviously still needs a bit of polishing to fix such remaining issues. Speaking of spells, I decided it was acceptable, given the circumstances, to cheat to give myself extra funds to purchases all the spells for each character. There was little point in delaying matters, trying to accumulate wealth. It also meant I was able to buy pathfinding and mountaineering skills for my party so I wouldn't be constrained where I could travel. It already proved useful, as I was able to identify and fix several bugs with the close-up rendering of forests, where they weren't properly being scaled, and were drawing outside of the scene area.<br />
<br />
From here, I'll keep on working my way through the game, fixing bugs as I go. Given I'm already significantly into the game (at least for Clouds), I don't anticipate there'll be any major impediments to my progress. Apart from tracking down remaining issues with combat, hopefully the rest of the game will just be fixing minor problems, like that which prevented me from entering the Witch Tower.<br />
<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-13675888844471026772018-03-24T00:51:00.003+11:002018-03-24T00:51:16.366+11:00Xeen playthrough in progress<div dir="ltr" style="text-align: left;" trbidi="on">
The Xeen playthrough is finally in progress, having been restarted last night. And by the time I went to work this morning, the first town of Clouds, Vertigo, was completable. Huzzah. \o/. I did identify and fix some further problems, such as armor not equipping, reading the note in the Vertigo warehouse, and getting the experience from the mayor. Next stop will be the Dwarf Mines, which I don't anticipate having much, if any, problems. Particularly since I've done some previous testing in them, though I've not yet been beyond the first level.<br />
<br />
This is a good omen for completing the rest of the game. Particularly given the amount of care I've already given to fixing bugs I'd introduced in my reimplementation of the combat system. And the fact that all the various towns share the same code base for implementing the different tavern, guild, etc. locations. I'm expecting/hoping that the rest of the game will be straightforward. Indeed, most of the fixes I did last night were fairly straightforward to fix once I'd identified the bugs in question.<br />
<br />
Of course, it's been years since I played the game, and honestly I only really remember some of the highlights of the Dark Side. So this playthrough will be a chance to re-experience the games once again. And my unfamiliarity will ensure I go poking into every nook and cranny. Though the downside is that I may not notice minor things not working, such as searching specific locations not giving the expected items, monsters, or loot. Once I've finished my first playthrough, hopefully some more experienced Xeen players can be enticed to play through the game as well.<br />
<br />
Well, better get back to work. It may be Friday, but I still have the entirety of the work day to get through before I can get back to my playtesting :)<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-2848978381343302552018-03-22T00:29:00.002+11:002018-03-22T00:29:48.252+11:00Xeen there, done that<div dir="ltr" style="text-align: left;" trbidi="on">
After all the time spent both in development, and in hiatus, work on the Xeen engine nears it's completion. Last weekend, I finally considered the engine finally stable enough to start my first playthrough of the game. Though, unfortunately, that idea rapidly crashed and burned.. I immediately discovered a variety of bugs in my exploration of Vertigo that my previous casual testing hadn't revealed, particularly with the Guild display and purchase of spells.<br />
<br />
Thankfully, I fixed all those, and over the course of the week so far I've also since discovered further minor issues that I've also been fixing as I identify them. As of today, I'm once again at a point where I'm not aware of any other bugs, so I'm once again going to restart my playthrough and start playing my way through the game. As last weekend showed, it's better for me to finish my own playthrough first before I announce official testing, since it will be a chance to fix all of the more obvious errors that crop up, and ensure anyone else that tests the game will have a smoother, more enjoyable experience. And speaking of testing, some good news.. rather than concentrating on just getting World of Xeen working, I've also implemented the necessary extra functionality and main menus for Clouds and Dark Side individually, as well as for Swords of Xeen. That means three separate games as well as the combined World will be playable using this new engine.<br />
<br />
Recent discussions on the GOG forums have also got me to thinking about what the future could hold for this engine. Whilst I'm definitely going to move onto working on other games after this, and likely take a break to play some games, there are some fertile areas someone with a knowledge of C++ could work on and submit patches to the ScummVM group for. First of all, <a href="https://www.gog.com/forum/might_and_magic_series/help_with_ludmeisters_world_of_xeen_mod">this GOG thread</a> talks about "Ludmeister's mod". It seems to have been abandoned by the author without ever having an official release. But it did have a description of all the changes it introduced. Since ScummVM also allows for game-specific options in the launcher dialog, someone could add checkboxes for enabling/disabling the functionality and add relevant code to the engine to implement them. Some of the mod's changes sounded nice, particularly regarding the passage of time.<br />
<br />
Another thing I'd love to see is for this to revive some interest in creating new areas and/or content. Not to spoil things too much, but I couldn't resist adding some new content of my own. I'll leave it up to players to see if they can discover it. If we're lucky, maybe it will help spur others to create further new content for the game. After all, Swords of Xeen was original a mod done by fans. With the engine source code as a guide, maybe someone can cannibalize the scene rendering code and build a visual world editing tool around it.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com3tag:blogger.com,1999:blog-6007794715807265125.post-86315524438664588162018-01-11T13:30:00.003+11:002018-01-11T13:30:47.130+11:00Christmas holiday round-up<div dir="ltr" style="text-align: left;" trbidi="on">
Over the course of the Christmas holidays, I took the opportunity to take a break from working on completing the Might & Magic Xeen engine to do some further work reverse engineering and implementing code for my Legend Entertainment games engine in ScummVM. The result can be seen in a <a href="https://youtu.be/96HSfOudFDY">demonstration video on Youtube</a>.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZtGf_XKF4H8wTPH2pCa6EJfsEpKWQHWE7iQH1cPC1MeJRTtx6hlHHuPKvJLBuNhfWmGjcT6z8mPP1OULeQEYu2qhr8JW2AxRTCAQt2G6qWv_9OxYGbjOdsG6VGOysJCEcjOE7Ozv06_w/s1600/gateway.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="518" data-original-width="656" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZtGf_XKF4H8wTPH2pCa6EJfsEpKWQHWE7iQH1cPC1MeJRTtx6hlHHuPKvJLBuNhfWmGjcT6z8mPP1OULeQEYu2qhr8JW2AxRTCAQt2G6qWv_9OxYGbjOdsG6VGOysJCEcjOE7Ozv06_w/s320/gateway.png" width="320" /></a></div>
<br />
<br />
For now, I've been focusing on disassembling two of the Legend Entertainment games, Frederick Pohl's Gateway, and Companions of Xanth. The former is one of the earlier games with a combined picture and text parser, whilst the latter is the more standard point-and-click type of adventure game. Both share a lot of code in common; though as I've discussed previously in details, the two have some key differences.. Xanth is a full 8-bit point-and click game, but it uses a very annoying overlay manager that makes dissasembling and debugging the code really trouble. Gateway, being an earlier EGA game, is much easier to do.<br />
<br />
Over the week I spent on the games, and my engine, there were two main advances. Firstly, I increased my understanding of how the original did the core event handling and renderings. Renderings in particular were somewhat messy, with different parts of UI elements being drawn in different methods; some when the game window is first opened, and others as event changes occur. Using the listbox in particular as an example, I tracked down where the different parts were being drawn and cleanly re-implemented them together into their own draw method of a Listbox class.<br />
<br />
Likewise, the Commset also proved useful in being a separate custom screen which nevertheless used many of the core engine's methods for creating windows, defining interactable regions of the screen, drawing, and event processing. I was almost to the point of implementing the initial text display within the Commset when the week ended.<br />
<br />
The second major area was fleshing out my engine's event handling. As can be seen from the example video, game views can now be rendered, and the listbox interacted with. This represents a lot of modular code in my engine taking care of managing what the current view is, as well as gathering mouse events and dispatching them into a hierarchy of parent/child game views that can represent the game screen, and then individual elements within it like the Listbox. A lot of this draws inspiration from the Titanic engine, which heavily uses a hierarchy of game views, as well as message handling for passing messages between elements.<br />
<br />
The result is that I'll be able to avoid, for example, a lot of hardcoding the original has for handling the two different listboxes. Instead, I can set up the listbox to generate a message when an item is clicked, and then have processing code in a parent "User Interface" class for the overall game screen that will handle replacing the listbox contents as necessary, as well as passing on selected words and/or phrases to the text entry widget where the player inputs their commands.<br />
<br />
All this is to keep hardcoding to a minimum, particularly once I've got a better feel for the engine, and start working on Companions of Xanth in greater detail. The idea is to eventually make it easy to support other games, like Death Gate\, so the less hard-coding there is for specific games the better. This current design allows for a modular build up of game content, with an overall "project item" that contains the different views like the game interface, conversations, and so on. And visual elements like the game interface can be further build up of individual UI components like the game scene, inventory, command buttons, and so on. Much of which will be able to be cleanly re-used between the games.<br />
<br />
Anyway, now that the holidays are over, I've returned to work on the Might & Magic Worlds of Xeen engine. I'm wary of calling it "nearly completed", but it certainly has a lot of functionality in it already. So it deserves to be finished, fun as continuing to work on the Legend games might be.<br />
<br />
Once that's done, though, who knows when I'll next return to work on the Legend games. I may well do it directly after. Although, the recent discussions of AGS have also piqued my interest as well. Depending on whether or not anyone else has taken up the reins of working on it to any degree, I may refocus my effort onto it. And of course, there are the later MADS games like Return of the Phantom & Dragonsphere that could be worked on. Not to mention the Access engine, with Martian Memorandum. Plus various different other RPGs. It's always nice to have options :)<br />
<br />
DreamMaster.<br />
<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-26327623634298591192017-12-18T10:23:00.004+11:002017-12-18T10:23:35.439+11:00It has to be Xeen to be believed<div dir="ltr" style="text-align: left;" trbidi="on">
Hi everyone,<br />
<br />
Been a while since my last blog posting, despite regular protestations that I'd be posting more frequently. My bad. At least my development work has been proceeding with it's usual speed in the interim. The just released ScummVM 2.0 spurred me to write a blog posting.<br />
<br />
So what's been happening? First of all, as most will know, I finished work on Starship Titanic. So with the new release, both the English and German versions of the game are now officially supported. I've been informed that there's also a French version, but it isn't available commercially (<a href="http://gog.com/">GOG</a> only has the English and German versions). Given that, for now I'm unwilling to spend the multiple weeks worth of effort to add support for it. Maybe if GOG adds it in sometime in the future, support for it can be revisited.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggvSsLO_yEp41DsS9W50VZt4AUXsQb500vOKMFZoAbbl5yKUk2lXoDFqSqcPukQxZudh08iYxx6SJACd3_Ie3wXFr5h465mcUO3js3Z5616tv8MYTX6KytZOCkr745W7kzjKbatPihIWc/s1600/starship_titanic.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="330" data-original-width="428" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggvSsLO_yEp41DsS9W50VZt4AUXsQb500vOKMFZoAbbl5yKUk2lXoDFqSqcPukQxZudh08iYxx6SJACd3_Ie3wXFr5h465mcUO3js3Z5616tv8MYTX6KytZOCkr745W7kzjKbatPihIWc/s320/starship_titanic.png" width="320" /></a></div>
<br />
Next, let's move onto what I'm currently working on, now that Titanic is done.. World of Xeen. Support for the game has already improved enormously over the last few weeks. You can walk around the starting town, fight monsters, then leave the town and wander around the outdoors. Even go to the Dwarf Mine and listen to the dwarf's cheesy intro. Not all maps are working yet, but at least it's getting there.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCVj3q-vpb1UaMKYU6WEsWQbGAJB2xJ_Hklh60IeG4wSCRFd2iV2o8oJMAK9wqKSo7AuLRlyPqrT68ZjJqHI382AI_lIWDrsIwOphHFjglww4Tcb_h1LZ8mq_fW2QKeNjlv9WG2rkQWUc/s1600/xeen1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="320" data-original-width="488" height="209" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCVj3q-vpb1UaMKYU6WEsWQbGAJB2xJ_Hklh60IeG4wSCRFd2iV2o8oJMAK9wqKSo7AuLRlyPqrT68ZjJqHI382AI_lIWDrsIwOphHFjglww4Tcb_h1LZ8mq_fW2QKeNjlv9WG2rkQWUc/s320/xeen1.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaPRgGKya6IDkcuaKhz9T0PEGudShhwim_89uEQzEXQsF3LDNmnVQmNOysQhx405fJAy_M4PKtE2t92mD5e9hgOWIMFxEPPfNGb0vHpRwYKeiwk94JNF90gLHErPlAanYdnhWLTm9dVEk/s1600/xeen2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="320" data-original-width="488" height="209" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaPRgGKya6IDkcuaKhz9T0PEGudShhwim_89uEQzEXQsF3LDNmnVQmNOysQhx405fJAy_M4PKtE2t92mD5e9hgOWIMFxEPPfNGb0vHpRwYKeiwk94JNF90gLHErPlAanYdnhWLTm9dVEk/s320/xeen2.png" width="320" /></a></div>
<br />
Still lots of other UI bugs to be looked into. I spent much of the past week cramming to implement all the various in-game cutscenes without testing them yet, so there's bound to be problems. The intro/endgame cutscenes for the games are also still only partly implemented. I'll also likely need to do further work on the savegame code, so the engine definitely isn't ready for any serious playing yet.<br />
<br />
Anyway, with Christmas holidays coming up, I'll likely take a brief break for a few weeks from my resumed work on Xeen. Much of the work for reversing the game had already been previously done by the excellent work of WizardStan, and then further by myself. As such, a lot of the time I've spent simply implementing remaining missing code in ScummVM and lots of debugging. Since I've already spent so much time in debugging Starship Titanic code to get it working, I'm looking for a more relaxed, cerebral exercise of reverse engineering something fresh over the holidays.<br />
<br />
I'll probably return to my preliminary work on the Legend entertainment games, Companions of Xanth and Gateway. Both share a lot of common engine though Xanth, the more recent, has a horrible overlay manager that makes it a pain to debug. As such, I've found working on the earlier game makes things easier to understand in many places. Though since the game uses EGA graphics, understanding how things are being written to the screen does get problematic in places. So a balanced effort reversing both at the same time is called for. An excellent mental challenge.<br />
<br />
Everyone have a Happy Holidays/Year end/Christmas or whatever, as 2017 comes to a close. :)<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com3tag:blogger.com,1999:blog-6007794715807265125.post-38194572869546717262017-06-12T08:26:00.002+10:002017-06-12T08:26:50.227+10:00The player is finally home.. the long way round<div dir="ltr" style="text-align: left;" trbidi="on">
It's taken so many nights spent slowly debugging the original executable versus my ScummVM implementation, but the final starfield puzzle of Starship Titanic is finally working.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9E-aCHjYshrLgYkQy-mrMz78LY3zUf5oDlVvWhh4egg4HRLKnlCXj8TcaBQar4VzOoPtpgmLGKhAKNY2fdjM0wsWRSpH6URBRdciyFe8eBL-8bXQPIpfbd8P0CQSEWW7nHf3Y_ZrXyRQ/s1600/titanic1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="962" data-original-width="1279" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9E-aCHjYshrLgYkQy-mrMz78LY3zUf5oDlVvWhh4egg4HRLKnlCXj8TcaBQar4VzOoPtpgmLGKhAKNY2fdjM0wsWRSpH6URBRdciyFe8eBL-8bXQPIpfbd8P0CQSEWW7nHf3Y_ZrXyRQ/s320/titanic1.png" width="320" /></a></div>
<br />
I'm not 100% happy with how the starfield rotates to selected markers when you've locked them in, but frankly, given that I've spent multiple months on disassembling, implementing, and fixing just this one puzzle, I'm just happy at this point that it works at all.<br />
<br />
So now with the starfield puzzle finally completable, I was able to initiate the endgame, and see the ending video and credits:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkEiZwi-36AbK2Ocs5BlKoCo04YEHUUqfSxgEQSNWRaDWHJo62GJAfQ-GkSu-MImJz9lo20G8P9F8cN03pW1keL5RPNKNnbAfpyQRbK2gXsPnwZu2vG9HXF4e1doaapDTPDW8QRVrqp9k/s1600/titanic2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="999" data-original-width="1296" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkEiZwi-36AbK2Ocs5BlKoCo04YEHUUqfSxgEQSNWRaDWHJo62GJAfQ-GkSu-MImJz9lo20G8P9F8cN03pW1keL5RPNKNnbAfpyQRbK2gXsPnwZu2vG9HXF4e1doaapDTPDW8QRVrqp9k/s320/titanic2.png" width="320" /></a></div>
I have to say, there were times when I grew weary of implementing and testing all the matrix code that the puzzle required. But I guess I'm just too stubborn not to see it all the way through.<br />
<br />
So what's happening next? Firstly, there are various minor bugs that I was aware of, but hadn't previously gotten around to fixing. I'm currently working into fixing them now For example, I fixed some jerking of text in the end credits, and some black boxes that briefly appeared over the flames in the canal. I've got some outstanding issues with NPC idle animations to look into. The Bellbot also won't currently bugger off if you tell him goodbye :) Once that's done, I'll give it another playthrough just to make sure before it's announced for public testing. So expect it to be soon. For those of you that don't have the game already, the current GOG sale has it discounted. So it's a very opportune time to pick it up.<br />
<br />
On a final note, there are couple of things associated with the game that I don't have any immediate plans to spend time on:<br />
* The QSound library the game uses for simulating sounds in a 3D space using standard stereo output. Many know my distaste of working sound code. So I'll leave it as a future exercise for someone else to work on. I've implemented the low level sound calls using mostly the same interface as QSound exposes, so it should prove convenient for anyone who chooses to do so<br />
* The Indeo 4 decoder still doesn't handle cases where transparency information is embedded directly into the video frames, rather than as a separate video track. Since codecs are installed directly into Windows, I'm not even sure which DLL implements the decoder. I'll try and spend a bit of time trying to figure it out, but worst case, it may be something the game just has to live with. The game currently has "best guess" code that estimates what the transparencies should be. It's not perfect, but it's reasonably servicable for now.<br />
* I don't have any near-term plans to do any further work on the German version. I'm simply too burned out over the game, and want to move on from it. I may return to it one day; I'd also welcome anyone else who wants to look into it themselves.<br />
<br />
DreamMaster.<br />
<br />
<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com2tag:blogger.com,1999:blog-6007794715807265125.post-31787870512528537662016-11-05T04:11:00.000+11:002016-11-05T04:11:25.676+11:00There's been a Titanic amount of progress<div dir="ltr" style="text-align: left;" trbidi="on">
Since my prior posting earlier in the year, there's been a great deal of progress in Starship Titanic. I decided to put aside the problem of reverse engineering all the Star Map classes until I had the rest of the game working better. In that respect, I've made great strides since, as of last weekend, I was able to complete the entire "prologue" of the game That included using the computer, experience the crash, talking to the Doorbot, entering the ship, and viewing the Credits. Huzzah. \o/<br />
<br />
I was going to prepare a video showing the intro, but with the most recent changes, there seems to be some instability showing up. It seems like something that was already present, just coincidental that the newer changes result in more frequent crashes. It's kind of hard to narrow down the cause, as there's also a problem with the implementation of the Indeo video decoder we're using for NPC videos like for the Doorbot, where it's reading past the end of the frame data. So it's difficult to track down the memory corruption, as warnings about the decoder are completely overwhelming everything else.<br />
<br />
So for now, I'll present a screenshot of the amazing multi-color Doorbot :)<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBnC5wBOAYGrFB-URWYC4CVRrmkGk1xIHw6lDmW6JpSixkIQBzrydP6C_jXa9amae1b2J-nhJ9HZT4qWQwY5Nblar_jzUw6IYFKicSWf2XDKaqancKkc90cDHK9kBOTre0A-5d03uJlRE/s1600/doorbot.png" imageanchor="1"><img border="0" height="255" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBnC5wBOAYGrFB-URWYC4CVRrmkGk1xIHw6lDmW6JpSixkIQBzrydP6C_jXa9amae1b2J-nhJ9HZT4qWQwY5Nblar_jzUw6IYFKicSWf2XDKaqancKkc90cDHK9kBOTre0A-5d03uJlRE/s320/doorbot.png" width="320" /></a><br />
<br />
<br />
After thinking over matters, I've decided to keep progressing into the game, and come back to look at the problem later on. Part of the trouble I'd been having with the code was the sheer length of the intro as I got further and further into it.. requiring me to wait through several minutes of cutscene & conversation every time I made any changes or bugfixes. Even if the intro has suddenly become unstable, I still have savegames I made from beyond it, so I'm using them as a starting point to make further progress testing into the game.<br />
<br />
Speaking of testing, I've had a major boon to my efforts to track down bugs in the code. I was previously stymied trying to test the original Windows executable in the IDA debugger, since it kept crashing on me. Plus running in compatibility mode full-screen didn't help either. And without the ability to see "valid" values in the original executable, I anticipated it would be difficult to track down errors in my code, since I wouldn't know whether values/state at any point in time were already wrong on not.<br />
<br />
Luckily, though, I stumbled on a solution. Using the Visual Studio "Attach to Process" allowed me to attach to the game executable without it crashing, unlike IDA. At least, for the majority of the time. Though switching from the game to the debugger and back again caused severe corruption of the full-screen display. Luckily, though, there had been some previous discussions about running the game in a window - I was able to use a utility called DXWnd that intercepted the game's DirectX calls and forced it to run a window. The result wasn't perfect, in my opinion, for anyone wanting to play through the game, but it's worked well enough for my purposes, in conjunction with Visual Studio.<br />
<br />
As a result, I'm now making much better progress than I had anticipated, and hunting down bugs is in general much easier than I'd anticipated. Let's hope that stays the case.. my next major gameplay milestone is to complete more extensive conversation with the Deskbot to get myself a room. The basic yes/no detection for the Doorbot worked pretty smoothly first time I tried it. The Deskbot, though, is using more of the conversation parser - I've already located and fixed some problems with it. Let's hope that there won't end up being too many.<br />
<br />
On a final note, the one downside of my surging progress with Titanic is that I'm currently spending less time working on finishing my Xeen engine. I'd originally anticipated the frequent roadblocks trying to hunt down bugs in Titanic would have me growling in frustration, and switching to Xeen for awhile to unwind a bit. Now with the ability to debug the original executable, that hasn't really happened so far, and hopefully won't happen. I'll probably end up spending more time right now focused solely on Titanic, and see if I can't get the bulk of the game with the exception of the final starmap working by the end of the year. Then I'll be in a better position to alternate between working on Xeen and trying to disassemble the remainder of the Starmap classes.<br />
<br />
DreamMaster.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-84835415583434306882016-07-17T04:31:00.000+10:002016-07-17T04:31:37.630+10:00PET is PrETty much done<div dir="ltr" style="text-align: left;" trbidi="on">
Hi all. Been a while since my last posting; and a hectic last couple of weeks. After flying down to Florida for a 4th July long weekend getaway, I was barely back before I had to get ready for a work trip to the west coast. Then I get back to two days of company photo-shoots, which kind of disrupted getting back to normal work; but no sooner than they were done, I come down with a cold and had to stay home for several days. And was too sick to even really use my computer until now. :(<br />
<br />
Nevertheless, despite all of that, work has been progressing on Starship Titanic since my last posting, and I've made a lot of progress. Firstly, there's the PET User Interface. This was one of the major areas that still had to be investigated last time. Since then, I've spent a lot of time figuring out how everything works, and re-implementing it all in my ScummVM engine. Behold the fruits of my labor:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHOizYhGF5Zfx_96etu71jU-iZzUAbFAGF2n13-994_GsUdH1te0E0sLiFNuf_JmkYm9-qVxWhnR7UZOX-AgOgfgpD14XIPcujb14kdY1nfSG4CxUSk4_xX9GTeDTydbvLA_kRRhzZk5I/s1600/titanic.png" imageanchor="1"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHOizYhGF5Zfx_96etu71jU-iZzUAbFAGF2n13-994_GsUdH1te0E0sLiFNuf_JmkYm9-qVxWhnR7UZOX-AgOgfgpD14XIPcujb14kdY1nfSG4CxUSk4_xX9GTeDTydbvLA_kRRhzZk5I/s320/titanic.png" width="320" /></a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkHw3xIFS-0AQNfPxm_G2ZSU9uXP_UPLiEEwJEpP0X_GxnhZiZPkhjWDyJuGcUGh217qeQhZeiq1GubXZDcXmkXu8FONjdPoObz-LDZx7AlhiRkqh5o6SSWDdW9BUgUj4kicKcJYCfIzQ/s1600/scummvm00001.png" imageanchor="1"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkHw3xIFS-0AQNfPxm_G2ZSU9uXP_UPLiEEwJEpP0X_GxnhZiZPkhjWDyJuGcUGh217qeQhZeiq1GubXZDcXmkXu8FONjdPoObz-LDZx7AlhiRkqh5o6SSWDdW9BUgUj4kicKcJYCfIzQ/s320/scummvm00001.png" width="320" /></a><br />
<br />
Code for all the various areas has been pretty much all implemented. You can enter text in the conversation tab, and click stuff in all the others. There's just a few minor visual and functionality glitches here and there, such as the dials in the above image. Some of the tabs, like the Remote and Rooms tabs, are mostly untested at the moment.. it will be easier to test when I have more of the game logic implemented, and can properly test them with live data. Saving and Loading is also waiting on me to finish the saving and loading code.<br />
<br />
As before, I was aided somewhat in implementing the PET by how clean the class hierarchy was designed, and the fact there were various internal error messages scattered about the code to give an indication of the classes' original names and purposes. For example, the code has a single common class CPetGlyph for the square glyphs used in all the different PET tabs, and CPetGlyphs for the collection of glyphs, with all the logic for scrolling and rendering. Individual PET areas simply derived from that, and instantiated whatever glyphs were needed, be they action icons like for Save, Load, and Quit, or inventory items that can be selected.<br />
<br />
Once the PET was pretty much done, I started looking into the conversation handling and text parsing. To begin with, I worked one of the easier, self contained areas of vocabulary loading. All the vocab data for Starship Titanic is held in a resource inside the game executable called STVOCAB.TXT. It holds the data for an impressive 4700 words! A lot of these come down to synonyms for common words, and in fact even includes foreign language words, like "ein" as a synonym for "a". Each word entry in the file also contains data for what kind of word it is (such as action, adjective, etc.), as well extra information that is used by the parser to uniquely identify the words. The vocabulary loader builds up the data for each word in turn, building a master list of overall words support by the game, and a linked list of synonyms for each word.<br />
<br />
Once I did that, I started looking into the input processing code that gets called when a line of text is entered in the PET. First of all, the game calls some pre-parse methods. These take care of a bunch of standard processing of the line, including:<br />
<br />
<ul style="text-align: left;">
<li>lowercasing everything and removing extra spaces between words</li>
<li>Conversion of numbers from textual formats into decimal. That was surprising.. the game has special code in place so you can enter numbers like '42', 'forty two', or even 'LXXIII'. That's right.. the parser is so sophisticated, it even handles roman numerals. :)</li>
<li>Expanding common contractions like "it's" to "it is".</li>
</ul>
<br />
Once preprocessing is done, it then breaks down the entered sentence into a series of words. Each word is checked against the word synonym lists to identify the known word for each entered word. Here again, the parser has extra complexities that surprised me. It can not only find word matches based on exact word matches, it has two fairly complicated routines that can identify various forms of pluralization, as well as common word prefixes like 'super', 'anti', 'counter', etc. and find matches on the word without the prefix. At the end of this process, the entire input has been broken done into a linked chain of recognized words, and the real handling of figuring out the player's input can be done using custom logic for each NPC script.<br />
<br />
At this point, whilst I had made some inroads on NPC script handling, I somewhat started drifting on what area I worked on. Strangerke had been looking at some of the core game scripts to add me in writing game logic, so to make things simpler in the long-run I went back to the main CGameObject base class, and spent some time figuring out all the remaining unknown methods. Quite a few of the methods were basically stubs calling PET Control methods, so my prior work on the PET was a great help.<br />
<br />
Next, there was a bunch of CGameObject methods calling movie code, so I decided it was finally time to figure out what it was doing. I was able to properly implement the game's OSMovie class which handles movies, and it's AVISurface which handles the low level AVI files. It was actually interesting in that some videos have a second video track - I haven't 100% disassembled the movie drawing code, but my impression is that pixels in one are transparent, and in the other are not. This required me to add an extension to our AVIDecoder to allow a callback method for selecting which streams the AVIDecoder would use. That way, I could set up two decoders in the AVISurface class, one with the audio and first video track, and the second with the second video track, if present. This neatly avoids the limitations of one video track per decoder. The replacement movie code is all implemented now, but apart from cursor loading working (the cursors are stored frame by frame as a video), movie playback is still crashing, and will have to be debugged further. But at least I know what the bulk of the movie methods are now.<br />
<br />
Finally, I moved onto the Star Control class. This is what implements the game's starmap control.. with PET mostly implemented, NPC scripts partially implemented, and movie handling more or less done, only it and sound are left unlooked at. And as I've mentioned previously, sound handling has a lot of spatial processing for sounds that I may simply not implement - the AVIDecoder already handles video sound, and I'm hoping to hook up the other sound methods to simply use standard ScummVM sound decoders. So that makes the star control the only remaining large area to look into.<br />
<br />
Oh boy, and what a large area it is. Even basic identification of classes was already bumping up to nearly 30 classes all specific to just the star control. I felt greater and greater dismay the further I looked into it. Honestly, it was like Hopkins FBI all over again, where they included a full Wolfeinstein 3D style minigame in the PC release. In that case we were lucky, and could simply ignore it. Not so much in this case - all the classes will have to be implemented.<br />
<br />
As I've looked into it further, and worked on classes that don't depend on others (so are self contained, and in theory easier to figure out), some of them have started to make sense. There's matrix and vector classes.. somewhat surprisingly, two versions of each, one for floating point numbers and one for doubles. I can only assume back in the day there was space vs speed considerations for having both types; I've not yet reversed all the methods, so I don't know if they can be merged in the future. I've also identified two "star points" classes that use large lists of spatial data that was hard-coded in the executable. Hopefully, the other remaining classes won't be too hard to figure out likewise.<br />
<br />
So, all in all, things are going good. I'm starting to feel better, and I'm making inroads on the last major area of the game. Once I get that out of the way, it'll be a matter of finishing off the NPC scripts, implementing miscellaneous methods here and there such as sound handling, and then implement the game logic. Which, with all the core engine implemented, will hopefully be a straightforward, if slow process.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-17440855984835312132016-04-12T11:55:00.001+10:002016-04-12T12:37:43.459+10:00Implementing Starship Titanic in ScummVM<div dir="ltr" style="text-align: left;" trbidi="on">
What began as a casual lunch-time activity (spread across many, many lunches), my work on Douglas Adam's Starship Titanic is finally starting to see significant results. See below for an example of the current progress from the start of the game: <a href="https://www.youtube.com/watch?v=8ypLR4fS6vE">https://www.youtube.com/watch?v=8ypLR4fS6vE</a><br />
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/8ypLR4fS6vE/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/8ypLR4fS6vE?feature=player_embedded" width="320"></iframe></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
You can't get as far as the ship crashing into your house yet, but at least you can fiddle around with the computer and, more importantly, use the television, and see Douglas Adm's visage scolding you to get on with the game. :)</div>
<div>
<br /></div>
<div>
I was originally attracted to working on the game more for technical reasons then the actual gameplay for several reasons. Firstly, because it was a Windows game.. my previous disassembly work has all been on DOS games, so I thought it would make a nice change of pace. Secondly, the original executable relies on compatibility tweaks to run on modern Windows systems and, according to the GOG forums, multiple people have had trouble getting it to run at all. And thirdly, the ease of disassembly.</div>
<div>
<br /></div>
<div>
The game has, in my opinion, the cleanest and most well thought class structures of any game I've worked on. Part of this clean hierarchy involved the bulk of classes in the game deriving from a common "CSaveableObject" base class that defines, amongst other things, the name of the class. The entire game state is laid out as a tree sturcture, with CRoomItem objects for rooms, CNodeItem objects for positions within a room, CViewItem for the different directional views within a node, and game objects under each view. Saving and loading games, including loading the initial game state for new games, is then a simple matter of dumping the hierarchy to and from disk.</div>
<div>
<br /></div>
<div>
Because of this, it's made it somewhat easier to reverse engineer how the classes are implemented. Since the game loading code needs to be able to locate and create classes by their textual name, it meant that I was able to properly identify and name all the classes the same way they were in the original source code (which I don't have access to). Since the original uses C++ classes and inheritance, it's meant that when I've identified the meaning of a virtual method, I could apply the same name to the same method in all the other objects. So, for exmaple, when I identified the methods for saving and loading an item's fields, I was able to focus on those same methods in all the other classes to handle loading the entire game state.</div>
<div>
<br /></div>
<div>
So as you can see from the above video, progress on the engine is going well. I have all the core logic in place for loading the initial game state, displaying views, and moving between them. I've also been able to graft some of the original's movie class, that handles playing AVI videos for all the game's animations, to use the existing ScummVM AviDecoder. Although there's still parts of CMovie that I don't understand. And there also isn't any background sound yet. Apart from that, there's two main areas left to be handled: firstly, all the hardcoded game logic. If there's one thing that I find a pity, it's that rather than using some kind of scripting system, they chose to implement all the game logic in code. So all the various interactable objects in the game have their own code that will need to be slowly implemented.</div>
</div>
<div>
<br />
The other main area remaining to be figured out is the PET control that provides the game's user interface. Particularly the conversation system, where you can converse with the characters within the game. I've already made some minor in-roads into implementing display of the PET, and various error messages in the original executable have given me some ideas of various classes in the conversation system. But it will still take a while to fully implement the game. Plus of course, dispute recently working on it quite a bit in my off hours, work on this game has primarily been a lunch-time diversion, so it's somewhat limited by my work's totally unreasonable policy that lunch should be limited to only a single hour each day :)</div>
<div>
<br /></div>
</div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-48260442618682278242015-11-05T05:32:00.000+11:002015-11-05T05:32:07.672+11:00The intricacies of disassembling RTLink/Plus<div dir="ltr" style="text-align: left;" trbidi="on">
Finally, after many years of half-hearted attempts, I've finally rewritten my RTLink decode tool practically from scratch to handle disassembling the RTLink/Plus overlay management used by the later Legend entertainment games. See <a href="http://github.com/roguevm/roguevm-tools/">rtlink_decode</a> for the result. The original version of the tool was pretty dodgy, and pretty much hardcoded to only handle the MADS games (Rex Nebular, Return of the Phantom, and Dragonsphere). In this posting, I'll go into more detail of what RTLink/Plus was for those who may be interested.<br />
<div>
<br /></div>
<div>
In the latter days of DOS gaming, game developers started running into a problem. Namely, that their executables were starting to hit the limits of main memory. Not every game could, or would, take advantage of scripting game content, so having all the game logic in the main executable caused the size to bloat. So what to do if your executable was now too big to fit in memory? This was the problem RTLink/Plus was designed to solve.</div>
<div>
<br /></div>
<div>
RTLink is essentially an overlay manager. It splits a program's compiled code into multiple different segments, and allows them to be loaded in as needed, and then replaced when code in other segments needs to execute. RTLink can handle recursive segments, with individual segments split up into their own set of swappable sub-segments. It also allows for multiple different "loading areas" in memory that can independently have their own set of segments. See <a href="https://books.google.com/books?id=2jT7iztEdKcC&pg=PT54&lpg=PT54&dq=rtlink/plus&source=bl&ots=-VRp9aFcNC&sig=2sjYCvg182nwvzdOtzjlviKsiUQ&hl=en&sa=X&ved=0CB0Q6AEwAGoVChMIk5r79qnqyAIVBVc-Ch0fQwkS#v=onepage&q=rtlink%2Fplus&f=false">this article</a> for more general information about RTLink.</div>
<div>
<br /></div>
<div>
Before I go into more details of the problem this posed for disassembly, let's go over how RTLink implements the overlay manager in code. So far, I've encountered three different variations on RTLink being used in executables. What I'll call variation 1 & 2 seem to be the most common form of RTLink in games I've examined. When a program is compiled with either of these versions of RTLink/Plus, one of the segments in the code will contain the RTLink logic, as well as two main areas: the dynamic segments and the function thunks.</div>
<div>
<br /></div>
<div>
The segment list is a list of the dynamic segments within the application. It contains the following information:</div>
<div>
<ul style="text-align: left;">
<li>The segment in memory where the dynamic segment should be loaded</li>
<li>Whether the segment is stored in the application or a secondary overlay file (Variation 1 only, version 2 only ever uses the executable).</li>
<li>The file offset and size of the segment</li>
<li>The number of relocation entries the segment has; Variation 1 only. Variation 2 has it as part of the starting header for the segment pointed to.</li>
</ul>
<div>
When a segment is needed, the above details are used to read the segment's data from file, and load it into the correct place in memory. The data for a segment consists of two parts: an initial header area, and the date/code for the segment. For variation 1, the header area simply consists of a list of relocation entries. Whereas for variation 2, the details of segment size and number of relocations are provided in a header at the start of the segment, before the relocations list.<br />
<br />
A segment's relocation entries are used for the same purpose as relocation entries in a standard application - executables can be loaded at different locations within memory, so all segment references need to be relative to the starting point of where the program is loaded. By keeping the relocation entries for each dynamic segment together with the segment data itself, it's easier for RTLink to apply any needed segment adjustments each time a dynamic segment is loaded.</div>
</div>
<div>
<br /></div>
<div>
This is fine to handle shifting the segments in and out of memory, and allow them to have valid memory references, but what causes them to be loaded? The answer is the method "thunks" area of the RTLink segment. When dealing with dynamic segments, you can't just do a far call to some offset in the area of memory segments are loaded in.. you couldn't be sure that the segment you want is actually loaded, or still in memory and not unloaded by some other segment. For this purpose, the thunk list is present.</div>
<div>
<br /></div>
<div>
For every method in a dynamic segment that is referenced by any other segment, a thunk/stub method is created. These consist essentially of the following: a call to the RTLink manager to load the correct segment for the method, a far jump to the method in the correct memory location in the loaded segment, and a following 16-bit value specifying which segment the thunk is for. This way, the thunk method acts as a wrapper, ensuring the correct segment is loaded and passing control to the method to execute.<br />
<br />
For variations 1 and 2, the thunk methods have some minor differences, such as version 2 using far calls to the RTLink segment loading code, and having an optional word after the segment index. The segment selector in the far jump call is also already loaded with the memory segment in variation 1, whereas in version 2 it's normally 0 initially, and then set to the correct segment when the thunk method is called. This allows variation 2 to dynamically load the segment in different places in memory, whereas variation 1 is limited to a single specific loading point.</div>
<div>
<br /></div>
<div>
The RTLink segment loader method also mucks around with the stack to push a new intermediate return address on the stack for when the method that's jumped to finishes. This return address points to a code fragment that also handles the case where a method in a dynamic segment calls a method in another one.. in that case, it handles reloading the original segment, so that the original caller's code can be safely returned to.</div>
<div>
<br /></div>
<div>
Put altogether, this scheme allows programs of practically any size needed. As the program grows, the code simply needs to be split into more and more dynamic segments which will get loaded only when needed, and remain on disk when not. Great for having big programs, but not so great for those of us interested in reverse engineering the game by disassembling the executable.</div>
<div>
<br />
There were several problems to be solved for disassembling such games, which I'll go into now.</div>
<div>
<br /></div>
<div>
<b>A standard IDA disassembly doesn't have all the code</b></div>
<div>
<br /></div>
<div>
Well, it wouldn't. If you try to disassemble an RTLink/Plus compiled game, IDA will give you an error about unused data at the end of the executable. This will be for one or more RTLink segments. Additionally, as previously mentioned, some of the code for the program can also be stored in a separate OVL (Overlay) file.</div>
<div>
<br /></div>
<div>
<b>Well, I could just load the raw data for them into the disassembly, right?</b></div>
<div>
<br /></div>
<div>
Well, no. That wouldn't help much, because of all the thunk methods. They all have their references to the same area of memory where segments are expected to be loaded. If you were doing things manually, you'd need to get the details of each segment from RTLink, manually load the code and/or data into new IDA segments, and then manually adjust the thunk methods to point to those methods.</div>
<div>
<br /></div>
<div>
You'd also need to worry about the dynamic segment relocation entries. If you manually loaded the code for a segment, you'd have to read the list of relocation entries for the dynamic segment and manually adjust each relocation entry within the segment. Segment selectors may point to code within the segment (or another sub-segment within the loaded overall RTLink segment), to a low memory area of the executable that remains static in memory, or to the data segment (at a higher memory segment). All in all, you'd have to be extraordinarily patient to all that by hand.</div>
<div>
<br /></div>
<div>
<b>So that's why you wrote rtlink_decode, right? That's what it does?</b></div>
<div>
</div>
<div>
Yes and no. A bit part of what it does is indeed doing the above to create a new executable suitable for disassembly. This includes laying out all the dynamic segments sequentially (without their segment headers and relocation lists), handling relocation fixups, and the thunk methods adjusted to point to their methods in the decoded executable. However, another problem crops up in the handling of the data segment.</div>
<div>
<br /></div>
<div>
In my experience with RTLink, I've come across across two types of data segments:</div>
<div>
<ol style="text-align: left;">
<li>In the case of the later Legend Entertainment games, the executable has a single RTLink segment, with the remainder of the segments coming from an OVL file. The single executable segment is for the main data segment as well as a few other miscellaneous segments.</li>
<li>In the case of the MADS games, the data segment isn't an RTLink segment, but all the RTLink segments follow it in the executable.</li>
</ol>
<div>
In both cases, we have a problem doing a proper disassembly. Executables are normally expected to have the data segment at the end of the program, because the data segment may be longer than the end of the executable. For example, a game's data segment may only have 1Kb of pre-set values which are stored in the executable, but it still requires 40Kb of unallocated/uninitialized space. That's why you'll frequently see, when you do a disassembly of a program, areas at the end of a data segment with '?' mark values, indicating the memory isn't part of the executable, so doesn't have any specific value when the program starts.<br />
<br />
So if we did just lay the segments end to end, the data segment, coming before other dynamic segments, would end up being shorter than it should be, and a lot of the references to data within it would end up wrapping onto the following dynamic segments in the reworked executable. To avoid this, the rtlink_decode tool ensures that the data segment falls at the end of the generated executable, after all the other segments. This, however, causes it's own share of problems. All the existing references to the data segment refer to where the data segment was expected to be loaded in memory, not to where the data segment actually is in the new executable. Because of this, all the references to the data segment in the executable have to be adjusted accordingly.</div>
</div>
<div>
<br /></div>
<div>
<b>Ouch! Sounds fiddly.</b></div>
<div>
<br /></div>
<div>
It is. And took a lot of messing around to get right. Even then, that's not the entirety of the picture. For Companions of Xanth, the Legend game I used for testing when rebuilding the tool, the data segment has some extra gotcha's.. It contains segment references into the middle of the memory area RTLink segments are loaded into. Presumably these are used in some special controlled circumstances when a specific segment (or segments) are loaded to access particular data. But it's impossible to know without understanding the game a lot better. </div>
<div>
<br /></div>
<div>
Worse, the presence of the references were screwing up some of the loaded dynamic segments in the disassembly, causing them to be split in half. To handle this, the tool explicitly looks for such "bad" references in the data segment, and removes the relocation entries for them. This way, the value in the data segment will remain as a static word, and the segments don't get incorrectly split up. The user can always then later manually set up a pointer to an appropriate segment if they wish. This handles the bulk of such errors, but Xanth at least, there are still references in the low part of the executable (that remains static in memory) to locations within the RTLink segments. Since I can't know which particular RTLink segment is meant to be loaded when the code they're in is called, these few remaining references will have to be later manually adjusted as well.</div>
<div>
<br /></div>
<div>
<b>So that's it?</b></div>
<div>
<br /></div>
<div>
Yep. After all these years, I'm finally able to generate a (mostly valid) "decoded" executable, and produce a clean disassembly of Companions of Xanth. I also, initially, had two separate versions of the the tool, one the old hacky version for MADS games, and the legend variation for Legend-style RTLink usage. I've since updated my tool to properly handle MADS games, so now there's only the single rtlink_decode tool, and it can handle both variations 1 and 2.<br />
<br /></div>
<div>
<b>Oh, wait.. what about the 3rd variation you mentioned?</b><br />
<br />
Ah, yes, I didn't really get into that, did I. This version seems to be somewhat different than the other two variations. In this case, the RTLink code is stored in a separate rtlinkst.com file, and then loaded into memory. It then shifts part of the program downwards in memory, and uses it's own relocation table to manually process relocation entries on the shifted code. This variation is proving tricky to disassemble, so whilst I have located the segment list, I still need to:<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
<div style="margin: 0px;">
</div>
<ul style="text-align: left;">
<li>Figure out how relocation data is encoded. I think I've located the correct data in the executable, but the code RTLink uses to update relocation entries is pretty nasty and overcomplicated.</li>
<li>How much of the start of the executable to remove so that the produced executable doesn't have any of the old code at the start of the executable that gets overwritten</li>
<li>Find the thunk methods, and see whether the existing code will handle them.</li>
</ul>
<br />
<div style="margin: 0px;">
Hopefully I can quickly figure out the remaining details for the third variation soon. The goal is to have a tool that both myself and others can use in the future to help them disassemble any game that used RTLink/Plus. Then no-one else will have to go through all the frustrations that I did trying to deal with this %#@! thing.</div>
<div style="margin: 0px;">
<br /></div>
</div>
</div>
</div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com1tag:blogger.com,1999:blog-6007794715807265125.post-37029234650469601602015-10-31T15:01:00.001+11:002015-10-31T15:01:43.969+11:00Sherlock Testing Resounding Success <div dir="ltr" style="text-align: left;" trbidi="on">
Hi everyone,<br />
<br />
It seems like the testing of the Sherlock games has been a success. There were lots of bug reported, and all the ones reported so far have been fixed. The foreign language versions haven't all been fully tested yet, but hopefully now the immediate crashes with conversations and inventory in both Serrated Scalpel and Rose Tattoo foreign language versions have been fixed, and the rest of the games can be tested. I'd like to thank everyone who's tested so far for your efforts, and feel free to post any more bugs you come across. Though hopefully there won't be too many more to find :). And if one else has copies of either game, particularly different foreign versions, any other testers would be appreciated.<br />
<br />
Right at the moment I'm at the sweet point where all the outstanding bugs for Sherlock have been resolved, so unless new ones come in, I can turn my attention to other things.<br />
<br />
So, whats coming next for me? Several things:<br />
<br />
<b>Serrated Scalpel 3DO</b><br />
<br />
Serrated Scalpel 3DO still isn't completable. There are some areas, such as the darts game to fix up, and there are still some differences in sprite positioning that would need to be accounted for. It's somewhat constrained by the fact that we don't have any original source for it, and I don't have any experience with reverse engineering 3DO games. It may simply be a case of do as best we can with hardcoded fixes as necessary.<br />
<br />
The 3DO version also has a few missing things compared to the PC version; notably it lacks the journal the PC version has. A "nice to have" for the future would be to reintroduce it, so that the 3DO version could be the definitive version of the game, containing all the PC version functionality as well as the video for all conversations. We'd likely create a tool that extracts necessary graphics for UI buttons and the journal background from the PC version, and produce a Dat file that ScummVM can use automatically when playing the 3DO version.<br />
<br />
<b>RTLink Overlay manager</b><br />
<br />
Next, there's the RTLink/Plus overly handling in Companions of Xanth. I'd previously had some luck writing a tool to process Rex Nebular and create a flat executable with all the segments suitable for disassembling, but doing the same for Companions of Xanth proved elusive. Over the years I made several attempts, but none bore fruit. Until now. As of yesterday, I was finally able to write a new version of the tool that successfully generated a flat executable that could be disassembled. So one day, after a great deal of disassembly work, ScummVM may support the game.<br />
<br />
My only disappointment is that RTLink/Plus seems to have had quite a number of variations. Rex and Xanth's RTLink mechanism were fundamentally similar, with all the RTLink code, segment list, and method thunks/stubs in the executable and/or overlay file. For several other games with RTLink that I tried, however, they seem to use a bizarre alternate method where a file called 'rtlinkst.com' is loaded, then an '.RTL' file for the game, and finally only then the game executable. Which means that my tool doesn't work with them, and I'd need to figure out this new mechanism from scratch if I want the tool to be general-purpose enough to handle any RTLink game anyone might want to use it on in the future.<br />
<br />
Guess that can be another long-term project to muck around with. Hopefully it won't take as long as it did to finally get Xanth properly disassembled. :). I'll probably make another posting in a day or so about RTLink in more detail, for those that are interested.<br />
<br />
<b>Might & Magic, World of Xeen</b><br />
<br />
Next there's my work on re-implementing Might & Magic - World of Xeen (and Clouds of Xeen, Dark Side of Xeen, and Swords of Xeen). With some free time last weekend, I finally returned to working on them, and likewise was finally able to properly disassemble the algorithm the original used for scaling. So as of now, sprites are now correctly scaled, as before I was only using a rough guess scaling code I'd nicked from another game engine. I've also fixed some other drawing bugs for drawing outdoor areas (outside town). So as of now the game scenes now display properly! :)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXanvanF3bKx4ucvgoBM9YY4U6bZO9-Z7kdTGweWFu6InOCtdpm8mfxrQ_aG33L4QK5FkvrEP_ViejzulUgQURdZUV4EjUpS8A3wPvrE3N_qGz7oQlGp04T6vGJ6TE0s7Z7LDGCQp0Xzo/s1600/scummvm00001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXanvanF3bKx4ucvgoBM9YY4U6bZO9-Z7kdTGweWFu6InOCtdpm8mfxrQ_aG33L4QK5FkvrEP_ViejzulUgQURdZUV4EjUpS8A3wPvrE3N_qGz7oQlGp04T6vGJ6TE0s7Z7LDGCQp0Xzo/s320/scummvm00001.png" width="320" /></a></div>
<br />
<br />
That's right, you can now walk around, fight monsters, go visit the various buildings in town, and even leave the town! In fact, most of the functionality for the games are already implemented. Apart from lots of testing and minor bugfixes that will be needed, only the following major areas remain to be implemented:<br />
<br />
<ul style="text-align: left;">
<li>Introduction/ending sequences for the games</li>
<li>character management, and title screens.</li>
<li>Sound. I'm hoping I can simply slot in one of ScummVM's audio decoders without much further work.</li>
<li>Savegames</li>
</ul>
<br />
At the moment I'm concentrating on getting World of Xeen (which combines Might & Magic 4 and 5 together) working. But then afterwards I'll implement separate support for 4 and 5, as well as for Swords of Xeen. It might even be feasible to handle Might and Magic 3 - Isles of Terra as well, since I'm given to understand the engines are nearly the same.<br />
<br />
Those interested in following the progress can see it at the brand-spanking new <a href="http://github.com/roguevm/roguevm">RogueVM</a> Github account. That's right; after all the years of idle talk, I've finally set up a place to properly store RPG related game engines. No website yet, but at least it's a start. :) I'll likely spend the near-term focusing mostly on finishing support for the game before I move onto any other adventure games, considering how far along the engine is already.<br />
<br />
<b>Return of the Phantom</b><br />
<br />
Strangerke has put a lot of work recently into implementing scene logic for Return of Phantom, the next MADS game that was published after Rex Nebular. There are quite a few stubs for missing engine functionality that was added in though. So when I do return to working on adventures, it will likely be to assist him in completing the game.<br />
<br />
DreamMaster.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0tag:blogger.com,1999:blog-6007794715807265125.post-19262654514048189892015-07-06T14:09:00.000+10:002015-07-06T14:09:17.157+10:00Rose Tattoo is in progress<div dir="ltr" style="text-align: left;" trbidi="on">
Hi all,<br />
<br />
Work is progressing well on Rose Tattoo, and as of tonight I hit a major milestone.. the entire game intro sequence is now completable. See the screenshots below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH9wLwLQIgGD5sdZ8OU6aggIMyhsKqbGg2iwalpUgjAB7yDeHBO-BweIqoleZbErof2nXmFCKeZzH00rzUOKkbRfXozFNiPh1f48OjITZugzf8dDI6oXqKjlSNqbWxSSPqOeMk_h9BCR4/s1600/sherlock1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH9wLwLQIgGD5sdZ8OU6aggIMyhsKqbGg2iwalpUgjAB7yDeHBO-BweIqoleZbErof2nXmFCKeZzH00rzUOKkbRfXozFNiPh1f48OjITZugzf8dDI6oXqKjlSNqbWxSSPqOeMk_h9BCR4/s320/sherlock1.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPnDpsmFDwB41pBWjYdNK9LlgjEebP__YhXA8W8TIcyZhZWSo4zv3eGeL4NzLoGqhz5-dldKJXDRxXRZkuY3PTslIQWpEu7b-VLYNZ-JDPNIpAxBbi9Y1AOduLo9TH9eYCIG2s5N_KDpo/s1600/sherlock2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPnDpsmFDwB41pBWjYdNK9LlgjEebP__YhXA8W8TIcyZhZWSo4zv3eGeL4NzLoGqhz5-dldKJXDRxXRZkuY3PTslIQWpEu7b-VLYNZ-JDPNIpAxBbi9Y1AOduLo9TH9eYCIG2s5N_KDpo/s320/sherlock2.png" width="320" /></a></div>
<br />
There are still some minor graphic glitches at various points, and one of the scenes which is a double-side scene isn't properly scrolled horizontally yet, but even so, it's a great step forward in supporting the game. We're actually lucky in that Rose Tattoo implemented all of the introduction using standard game scenes. So it saved a lot of effort implementing manual introduction code like had to be done for Scalpel.<br />
<br />
Now that the introduction is working, more or less, I'll be devoting more time to implementing the game-play. I'd already been spending some of my working on it, so some interaction is possible.. you can look at objects, open up the inventory, and conversations with characters are partially working. The game map is also already implemented, so you can get to other game locations as well.<br />
<br />
On another subject, I had good luck implementing the original EA logo at the start of Serrated Scalpel. I was able to complete support for it in the TsAGE engine, and then used that as a basis for copying necessary code into the Sherlock engine. With some most welcome assistance from others, the EA logo at the start of the game now displays when you start the game, just like the original does.<br />
<br />
Finally, on yet another tack, there have been some promising first steps towards supporting the 3DO version of Serrated Scalpel by m-kiewitz, with some assistance from clone2727. The 3DO was a superior version of game, and included 16-bit color, video portraits, and full speech for every conversation in the game. Supporting this would be great. It would be nice if, one day, the 3DO version could be re-released with ScummVM, so a wider audience could properly enjoy the game.<br />
<br /></div>
Dreammasterhttp://www.blogger.com/profile/09618034545788778027noreply@blogger.com0