Christmas holidays roundup

 Welcome to 2023, everyone. It's now mid-month in January, and I'm back from my long overdue Christmas holidays. Unfortunately :). Whilst I did end up having to spend some of my time taking care of work, I still got significant time to rest, read some novels and, of course, muck around with ScummVM stuff :). As with several previous years, I mostly ended up fiddling around with my disassembly of the early Legend games, and trying to re-implement them in ScummVM.

Thankfully, I was finally able to make some proper progress this year. Working further on my disassembly of the first Gateway game, I started a new iteration of the Legend engine in ScummVM using the latest meta engine/detection code, and gradually pulled in the work I'd previously done in the prior version of the engine. Then using it as a basis, I started implementing the code for the main text window. This included all the logic for caching text, line wrapping, and the -MORE- indicator. When it was done, I finally had the basis for displaying game messages.

Once that was done, for the first time, I started looking at the actual engine implementation. And I made progress, stumbling on a set of methods that implement what seemed like a basic C++ class hierarchy in C, with a table of 734 "objects" that were each one of eight different categories/types. Each type having a given amount of currently unknown data values. The only field of these structures that is currently known is a "method index" into a table of functions. Whilst the bulk of objects have a unique function, there are a few that don't have one, or share the same method. This is also how I know the code was a manual implementation of classes, since the function for getting the method index is basically a big switch statement that reads the appropriate offset, which is different for each category. If it was proper C++, they'd have a common base class and the field would be the same for all of them.

And these methods being pointed to.. what are they? It seems to be nothing less than all the per-game custom logic. With a bit of experimentation, I found that the first category was for rooms. Changing rooms in game is basically a matter of changing the room object being pointed to, and executing some code in the room for entering it. There's a global "room object num" that has the starting room by default, but by changing the value during startup in the DosBox debugger, I was able to change the room I started in.

Funnily enough, I tried changing the "room" to one of the other category types, and realized that it's type was for items. It seems the core engine has some resiliency, since by setting it to the item, the game told me that I was now "in" the item, as if it were a room :). Presumably, the other categories may represent other types of game entries, like non-inventory scenery or room exits, and their internal functions will have the logic for looking at items, moving to new rooms, etc.

This is all a big deal, since it means that when I have time free to work on it, I now have a solid starting point for figuring out game logic, and the methods they call in the engine to do the various things. Once I spend a bit more time figuring out the sentence parser, I'll be able to rapidly progress in finally supporting the Gateway game and, likely in the future, other early Legend games as well. Though it's going to be a pain having to manually implement and test each and every object function in each game. It's a good thing I have the patience to do that kind of thing. Though come to it, if I figure out all the engine methods the game calls, maybe Ghidra can assist in quickly producing readable C code.

As a final treat, a screenshot of the current engine. The listbox, font drawing, and other elements were pulled in from work done in prior years, and they still need work. Mostly, I focused on the text area, and better understanding how the game works:

A retrospective and a look forward

Hey everyone. I hope everyone has done okay so far this year, considering the circumstances. On a more positive note, you will have seen we've recently announced the testing period for three of the games in the Ultima series of RPGs.. Ultima IV: Quest of the Avatar, Ultima VI: The False Prophet, and Ultima VIII: Pagan. Thanks go to the original projects and all the developers that previously put work into them, as well as in particular to mduggan, who has put a lot of work into Ultima VIII since it was merged in.

So now that we've got three of the Ultima series merged in, are more to follow? That's actually a bit of a tricky question. Of the remaining games
* Ultima IX: This is more in scope for ResidualVM than ScummVM. I remember reading previously that the "Ultima Source Code Offline Archive Project" had obtained the code for Ultima IX. So there may be a future possibility of it being reimplemented in ResidualVM.
* Ultima VII/SE: Exult is currently an independent project with active development and users. So there's no immediate plans for integration until and unless the developers decide they don't plan to do any further development, or decide themselves the time is right to merge it into ScummVM
* Underworlds: I'm vaguely aware that there are projects for the two Underworld games, but I haven't researched how complete they are. Despite their 3d look and feel, these games were done in the DOS days where all 3D was done in software. Because of this, unless the reimplemented code specifically uses 3D APIs, then it would actually be in scope for ScummVM rather than ResidualVM.
* Ultima I: I have previously been working on a reimplementation of Ultima 1. It's currently on hiatus, but I eventually plan to return to work on it. As with Ultima IV, I hope to allow for an enhanced version of the game that uses better tilesets
* Escape from Mt Drash: There is source for a PC reimplementation available here. So support for it could be added. Even if the only real relevance to the Ultima series was a marketing ploy, it is still technically an Ultima game, so support for it should be added I guess for completeness if nothing else.
* The rest. This is where it gets difficult. Apart from Ultima V, the earlier Ultima games were, honestly, less sophisticated, and I'd be less likely to enjoy playing them. So without pre-existing projects for them, it becomes harder for me to justify spending all the time reverse engineering them from scratch for support. It's even more frustrating in that it looks like there's been previous people that reverse engineered the earlier games, such as ultima2.voyd.net. But since they never released source code for their work, it really doesn't help us that much. If any of those authors happen to read this, or anyone knows how to get in touch with them, we'd really appreciate getting our hands on their work. Otherwise, it's unlikely the earlier Ultima games will be supported any time soon. Or if the previously mentioned Offline Archive Project has source for any of the earlier games, we'd welcome getting access to them.

So in any case, what are my plans now that those three games are working their way towards being officially supported? Well, first of all, I kind of fell into working on integrating the Recomprehend engine into ScummVM. Given the simplicity of the engine, it seems like a fun project to unwind with after all the work on the Ultima games. I don't anticipate it'll take all that long. There are apparently some minor issues in the screen rendering that may require reverse engineering bits of the original games for comparison, but again due to the simplicity of the games that shouldn't be too hard.

Next after that, I kind of want to get Glk into a more officially completed state. Technically all the sub-engines in it so far could be considered complete on their own, but the exception is the Frotz interpreter.. the current one didn't properly support all the version 6 graphical games. I would really like to support them properly. So I've tentatively started working on the newest version of Frotz to convert it to run on Glk from scratch. At this point I'm not sure if this new work will be able to be cleanly set up to handle both v6 and non-v6 games.. it may end up being easier to keep the existing version for non-v6, and just focus on v6 games for this. I'll have to wait and see.

After from that, Glk should be suitable for an official release. Though there are other sub-engines I could work on in the future, such as TADS support. The TADS 2 version in the codebase doesn't currently work properly, and when I previously tried starting on converting TADS 3 to compile under ScummVM, it led me down a rabbit hole of compilation issues. Support for both will likely be on hiatus for work again in a future release.

Apart from them, what else this year? Good question.. I had had some tentative plans to look into early AGS games support.. years ago I did a remake of Black Cauldron in AGS, and it'd be very satisfying to finally support it in ScummVM. However, just the other day Strangerke made an intriguing discovery. A non-interactive demo for Shannara, one of the later Legend Entertainment games, has what looks like debug information at the end of the executable (though it's not in a known format IDA can automatically handle). As long time readers will know, supporting the Legend Entertainment games has been one of my long term goals for years. Whilst the demo itself is effectively just a slide show, the debug information makes it look like they implemented the demo using large chunks of the full game's engine for rendering the demo scenes. In fact, there may even be a bunch of code from the main game the demo doesn't even use that wasn't optimized out. As such, it may prove a valuable insight for understanding the full Shannara game, and from that, the earlier games of Companions of Xanth and Death Gate.

So it may well be that I'll end up switching my focus back to the Legend games for a while. We'll have to wait and see. Considering that I ended up working on Comprehend by chance, there's always a possibility that something else interesting may pop up :)

Seasons greeting and off on holiday

Seasons greetings to all, and I hope everyone has a happy time as we finish 2018 and look forward to the beginning of 2019. As I sit in the Los Angeles airport writing this, I'm reminded yet again of how much fun I have doing reverse engineering, since even the smallest thread of work can lead you in unexpected directions as you trace the flow of what methods call a given method, and what methods it calls.

For something to do, I opened up my Companions of Xanth in progress disassembly to do a bit of pottering around with it. Just by chance, scrolling through the disassembly, I noticed a call to a method I was pretty sure took in a string, but in this case was passing in a value. I then quickly realised it was an offset n the data segment that hadn't been properly resolved, so I did that. The message in question turned out to be a generic "insufficient memory". So from it's usage, I was able to identify the entire method it was in.. it's called during startup, and ensures that there's enough free memory to play the game. Ironically, it looks like the code to actually display the message if there isn't enough was broken, as evidenced when I ran the game in DosBox debugger.

From there, identified a method just before the message call that took in a number and passed out a string.. it wasn't too hard to identify it as a number formatting routine. I confirmed it in the DosBox debugger by putting a breakpoint in the machine code, and seeing the result from the method. I then noticed that there was one other method calling it, so I was intrigued to find out what it was. It turned out it too was a block of code in a method that printed out the free memory, in a method that looked like a big switch based on a single value. It looked like a keypress. Playing the game, I was able to confirm that pressing 'M' on the keyboard did call the code, which printed out the amount of free memory in the status area.

This was a good woo-hoo method. I had identified the method for handling keypresses. Likely, some of the other keypresses do fairly identifiable things, like saving and loading, so knowing where each key's code is, I'll be able to trace into what they do, and what methods they call. In fact, the free memory printing for the 'M' key already prove dividends. From it, I was able to identify the method that cleared the status area, and a version of 'sprintf' for printing text in the status area. In the future I'll be able to look further into other places that call the method to add lines to the status area, and start identifying other parts of the game code. For example, looking at an item produces a description in the status area, so by setting a breakpoint in the status text writing code, I'll be able to identify what method calls it, and start identifying the structure of items that likely have a "description" as one of it's fields.

All this progress today came from a minor diversion to pass the time sitting in the airport. The notice of an incorrect parameter to a method call has unraveled more of the game's secrets, and giving me multiple further areas I can investigate. A nice way to start the holidays.

It's an UltimaTE unwinding

After all this time, support for the Might & Magic Xeen games is finally close to being finished. It's currently the official testing period for all the Xeen games, and I've already had a whole bunch of useful bug reports, which I've all taken care of. At this point, it's almost certain that the Xeen engine, and games, will be supported in the next official ScummVM release. It's a nice feeling. More than any other game engine I've worked on, the Xeen engine was the one that got put on hiatus the most number of times. It's very satisfying to finally complete it.

What's in store for me now? Well, I do want to spend some time properly unwinding and actually playing some games. They've only just come out with the sequel to Pillars of Eternity, which I'm now getting into, and plus I have a backlog of other RPG and adventure games that I want to have a chance to properly play. So that will probably keep me occupied for a few months at least. I've also got a move from the US East Coast to West Coast coming up next month. So various preparations are consuming my time, and it's all to the good that work on Xeen has already finished.

Not that I'm totally stopping work on ScummVM stuff. As with the early days of work on Starship Titanic, I started doing some casual reverse engineering during my lunch breaks. By happenstance, I experimented with the Ultima 1 executables, and become enamored by how simple they were to reverse. Both due to the simplicity of the game (considering how old it was), and the way the code was implemented. It was just the thing for doing casual reversing work in chunks of an hour at a time.

As a result of this, I've already implemented a great deal of foundation code in ScummVM for supporting the game, with a long term eye for having a clean engine structure that could support all the early Ultima games.  Here's some screenshots of the player within the overworld, a city, and within the dungeons:

At this point, I've got basic movement functionality working. The player can roam around the map, entering cities, castles, and dungeons. Dungeon movement also works, as does going up and down ladders, so the dungeon can be further ventured down into, or left entirely. That's it at the moment, though; it doesn't yet have any combat, magic, inventory, character interaction, etc. See this Youtube video for an example walk-about that covers venturing into a dungeon, and then heading over to the city of Britian.

Don't expect too much more work on this in the near term, though. As I said, this is more a side thing, and I definitely do want to spend some time unwinding and playing some games. When I get my fill, later in the year, then we'll see. At the moment, it's looking strongly like I'll keep working further on this. I've some crazy ideas about not only adding support for the game, but also reusing the Ultima VI tileset to create an "enhanced" Ultima 1 experience that's more visually pleasing, and rework the controls and user interface to be likewise more player friendly. I know that oh, pretty much everyone who ever played their way through the space simulation section would much appreciate mouse control for aiming at enemy vessels. :)

I kind of consider the RPG series of Ultima, Might & Magic, and Wizardry the triumvirate of major RPGs from the last century and, now that ScummVM is embracing RPGs in addition to adventures, I'd love to see them all under the ScummVM banner. Now that at least some of the earlier Might & Magic games are supported, it may well be time to start in on Ultima support :) Of course, I'm not entirely abandoning working on adventure games. I'd previously made some promising initial progress in reversing some of the Legend Entertainment games.. Companions of Xanth and Gateway. Irrespective of how much time I end up devoting to working on Ultima I, I may end up also devoting some of my time to those games as well.

Anyway, that's probably enough for now. I still want to get in some Pillars of Eternity 2 before it's time for bed. :)

Christmas holiday round-up

Over the course of the Christmas holidays, I took the opportunity to take a break from working on completing the Might & Magic Xeen engine to do some further work reverse engineering and implementing code for my Legend Entertainment games engine in ScummVM. The result can be seen in a demonstration video on Youtube.

For now, I've been focusing on disassembling two of the Legend Entertainment games, Frederick Pohl's Gateway, and Companions of Xanth. The former is one of the earlier games with a combined picture and text parser, whilst the latter is the more standard point-and-click type of adventure game. Both share a lot of code in common; though as I've discussed previously in details, the two have some key differences.. Xanth is a full 8-bit point-and click game, but it uses a very annoying overlay manager that makes dissasembling and debugging the code really trouble. Gateway, being an earlier EGA game, is much easier to do.

Over the week I spent on the games, and my engine, there were two main advances. Firstly, I increased my understanding of how the original did the core event handling and renderings. Renderings in particular were somewhat messy, with different parts of UI elements being drawn in different methods; some when the game window is first opened, and others as event changes occur. Using the listbox in particular as an example, I tracked down where the different parts were being drawn and cleanly re-implemented them together into their own draw method of a Listbox class.

Likewise, the Commset also proved useful in being a separate custom screen which nevertheless used many of the core engine's methods for creating windows, defining interactable regions of the screen, drawing, and event processing. I was almost to the point of implementing the initial text display within the Commset when the week ended.

The second major area was fleshing out my engine's event handling. As can be seen from the example video, game views can now be rendered, and the listbox interacted with. This represents a lot of modular code in my engine taking care of managing what the current view is, as well as gathering mouse events and dispatching them into a hierarchy of parent/child game views that can represent the game screen, and then individual elements within it like the Listbox. A lot of this draws inspiration from the Titanic engine, which heavily uses a hierarchy of game views, as well as message handling for passing messages between elements.

The result is that I'll be able to avoid, for example, a lot of hardcoding the original has for handling the two different listboxes. Instead, I can set up the listbox to generate a message when an item is clicked, and then have processing code in a parent "User Interface" class for the overall game screen that will handle replacing the listbox contents as necessary, as well as passing on selected words and/or phrases to the text entry widget where the player inputs their commands.

All this is to keep hardcoding to a minimum, particularly once I've got a better feel for the engine, and start working on Companions of Xanth in greater detail. The idea is to eventually make it easy to support other games, like Death Gate\, so the less hard-coding there is for specific games the better. This current design allows for a modular build up of game content, with an overall "project item" that contains the different views like the game interface, conversations, and so on. And visual elements like the game interface can be further build up of individual UI components like the game scene, inventory, command buttons, and so on. Much of which will be able to be cleanly re-used between the games.

Anyway, now that the holidays are over, I've returned to work on the Might & Magic Worlds of Xeen engine. I'm wary of calling it "nearly completed", but it certainly has a lot of functionality in it already. So it deserves to be finished, fun as continuing to work on the Legend games might be.

Once that's done, though, who knows when I'll next return to work on the Legend games. I may well do it directly after. Although, the recent discussions of AGS have also piqued my interest as well. Depending on whether or not anyone else has taken up the reins of working on it to any degree, I may refocus my effort onto it. And of course, there are the later MADS games like Return of the Phantom & Dragonsphere that could be worked on. Not to mention the Access engine, with Martian Memorandum. Plus various different other RPGs. It's always nice to have options :)

DreamMaster.

It has to be Xeen to be believed

Hi everyone,

Been a while since my last blog posting, despite regular protestations that I'd be posting more frequently. My bad. At least my development work has been proceeding with it's usual speed in the interim. The just released ScummVM 2.0 spurred me to write a blog posting.

So what's been happening? First of all, as most will know, I finished work on Starship Titanic. So with the new release, both the English and German versions of the game are now officially supported. I've been informed that there's also a French version, but it isn't available commercially (GOG only has the English and German versions). Given that, for now I'm unwilling to spend the multiple weeks worth of effort to add support for it. Maybe if GOG adds it in sometime in the future, support for it can be revisited.

Next, let's move onto what I'm currently working on, now that Titanic is done.. World of Xeen. Support for the game has already improved enormously over the last few weeks. You can walk around the starting town, fight monsters, then leave the town and wander around the outdoors. Even go to the Dwarf Mine and listen to the dwarf's cheesy intro. Not all maps are working yet, but at least it's getting there.

Still lots of other UI bugs to be looked into. I spent much of the past week cramming to implement all the various in-game cutscenes without testing them yet, so there's bound to be problems. The intro/endgame cutscenes for the games are also still only partly implemented. I'll also likely need to do further work on the savegame code, so the engine definitely isn't ready for any serious playing yet.

Anyway, with Christmas holidays coming up, I'll likely take a brief break for a few weeks from my resumed work on Xeen. Much of the work for reversing the game had already been previously done by the excellent work of WizardStan, and then further by myself. As such, a lot of the time I've spent simply implementing remaining missing code in ScummVM and lots of debugging. Since I've already spent so much time in debugging Starship Titanic code to get it working, I'm looking for a more relaxed, cerebral exercise of reverse engineering something fresh over the holidays.

I'll probably return to my preliminary work on the Legend entertainment games, Companions of Xanth and Gateway. Both share a lot of common engine though Xanth, the more recent, has a horrible overlay manager that makes it a pain to debug. As such, I've found working on the earlier game makes things easier to understand in many places. Though since the game uses EGA graphics, understanding how things are being written to the screen does get problematic in places. So a balanced effort reversing both at the same time is called for. An excellent mental challenge.

Everyone have a Happy Holidays/Year end/Christmas or whatever, as 2017 comes to a close. :)

The intricacies of disassembling RTLink/Plus

Finally, after many years of half-hearted attempts, I've finally rewritten my RTLink decode tool practically from scratch to handle disassembling the RTLink/Plus overlay management used by the later Legend entertainment games. See rtlink_decode for the result. The original version of the tool was pretty dodgy, and pretty much hardcoded to only handle the MADS games (Rex Nebular, Return of the Phantom, and Dragonsphere). In this posting, I'll go into more detail of what RTLink/Plus was for those who may be interested.

In the latter days of DOS gaming, game developers started running into a problem. Namely, that their executables were starting to hit the limits of main memory. Not every game could, or would, take advantage of scripting game content, so having all the game logic in the main executable caused the size to bloat. So what to do if your executable was now too big to fit in memory? This was the problem RTLink/Plus was designed to solve.

RTLink is essentially an overlay manager. It splits a program's compiled code into multiple different segments, and allows them to be loaded in as needed, and then replaced when code in other segments needs to execute. RTLink can handle recursive segments, with individual segments split up into their own set of swappable sub-segments. It also allows for multiple different "loading areas" in memory that can independently have their own set of segments. See this article for more general information about RTLink.

Before I go into more details of the problem this posed for disassembly, let's go over how RTLink implements the overlay manager in code. So far, I've encountered three different variations on RTLink being used in executables. What I'll call variation 1 & 2 seem to be the most common form of RTLink in games I've examined. When a program is compiled with either of these versions of RTLink/Plus, one of the segments in the code will contain the RTLink logic, as well as two main areas: the dynamic segments and the function thunks.

The segment list is a list of the dynamic segments within the application. It contains the following information:

• The segment in memory where the dynamic segment should be loaded
• Whether the segment is stored in the application or a secondary overlay file (Variation 1 only, version 2 only ever uses the executable).
• The file offset and size of the segment
• The number of relocation entries the segment has; Variation 1 only. Variation 2 has it as part of the starting header for the segment pointed to.

When a segment is needed, the above details are used to read the segment's data from file, and load it into the correct place in memory. The data for a segment consists of two parts: an initial header area, and the date/code for the segment. For variation 1, the header area simply consists of a list of relocation entries. Whereas for variation 2, the details of segment size and number of relocations are provided in a header at the start of the segment, before the relocations list.

A segment's relocation entries are used for the same purpose as relocation entries in a standard application - executables can be loaded at different locations within memory, so all segment references need to be relative to the starting point of where the program is loaded. By keeping the relocation entries for each dynamic segment together with the segment data itself, it's easier for RTLink to apply any needed segment adjustments each time a dynamic segment is loaded.

This is fine to handle shifting the segments in and out of memory, and allow them to have valid memory references, but what causes them to be loaded? The answer is the method "thunks" area of the RTLink segment. When dealing with dynamic segments, you can't just do a far call to some offset in the area of memory segments are loaded in.. you couldn't be sure that the segment you want is actually loaded, or still in memory and not unloaded by some other segment. For this purpose, the thunk list is present.

For every method in a dynamic segment that is referenced by any other segment, a thunk/stub method is created. These consist essentially of the following: a call to the RTLink manager to load the correct segment for the method, a far jump to the method in the correct memory location in the loaded segment, and a following 16-bit value specifying which segment the thunk is for. This way, the thunk method acts as a wrapper, ensuring the correct segment is loaded and passing control to the method to execute.

For variations 1 and 2, the thunk methods have some minor differences, such as version 2 using far calls to the RTLink segment loading code, and having an optional word after the segment index. The segment selector in the far jump call is also already loaded with the memory segment in variation 1, whereas in version 2 it's normally 0 initially, and then set to the correct segment when the thunk method is called. This allows variation 2 to dynamically load the segment in different places in memory, whereas variation 1 is limited to a single specific loading point.

The RTLink segment loader method also mucks around with the stack to push a new intermediate return address on the stack for when the method that's jumped to finishes. This return address points to a code fragment that also handles the case where a method in a dynamic segment calls a method in another one.. in that case, it handles reloading the original segment, so that the original caller's code can be safely returned to.

Put altogether, this scheme allows programs of practically any size needed. As the program grows, the code simply needs to be split into more and more dynamic segments which will get loaded only when needed, and remain on disk when not. Great for having big programs, but not so great for those of us interested in reverse engineering the game by disassembling the executable.

There were several problems to be solved for disassembling such games, which I'll go into now.

A standard IDA disassembly doesn't have all the code

Well, it wouldn't. If you try to disassemble an RTLink/Plus compiled game, IDA will give you an error about unused data at the end of the executable. This will be for one or more RTLink segments. Additionally, as previously mentioned, some of the code for the program can also be stored in a separate OVL (Overlay) file.

Well, I could just load the raw data for them into the disassembly, right?

Well, no. That wouldn't help much, because of all the thunk methods. They all have their references to the same area of memory where segments are expected to be loaded. If you were doing things manually, you'd need to get the details of each segment from RTLink, manually load the code and/or data into new IDA segments, and then manually adjust the thunk methods to point to those methods.

You'd also need to worry about the dynamic segment relocation entries. If you manually loaded the code for a segment, you'd have to read the list of relocation entries for the dynamic segment and manually adjust each relocation entry within the segment. Segment selectors may point to code within the segment (or another sub-segment within the loaded overall RTLink segment), to a low memory area of the executable that remains static in memory, or to the data segment (at a higher memory segment). All in all, you'd have to be extraordinarily patient to all that by hand.

So that's why you wrote rtlink_decode, right? That's what it does?

Yes and no. A bit part of what it does is indeed doing the above to create a new executable suitable for disassembly. This includes laying out all the dynamic segments sequentially (without their segment headers and relocation lists), handling relocation fixups, and the thunk methods adjusted to point to their methods in the decoded executable. However, another problem crops up in the handling of the data segment.

In my experience with RTLink, I've come across across two types of data segments:

1. In the case of the later Legend Entertainment games, the executable has a single RTLink segment, with the remainder of the segments coming from an OVL file. The single executable segment is for the main data segment as well as a few other miscellaneous segments.
2. In the case of the MADS games, the data segment isn't an RTLink segment, but all the RTLink segments follow it in the executable.

In both cases, we have a problem doing a proper disassembly. Executables are normally expected to have the data segment at the end of the program, because the data segment may be longer than the end of the executable. For example, a game's data segment may only have 1Kb of pre-set values which are stored in the executable, but it still requires 40Kb of unallocated/uninitialized space. That's why you'll frequently see, when you do a disassembly of a program, areas at the end of a data segment with '?' mark values, indicating the memory isn't part of the executable, so doesn't have any specific value when the program starts.

So if we did just lay the segments end to end, the data segment, coming before other dynamic segments, would end up being shorter than it should be, and a lot of the references to data within it would end up wrapping onto the following dynamic segments in the reworked executable. To avoid this,  the rtlink_decode tool ensures that the data segment falls at the end of the generated executable, after all the other segments. This, however, causes it's own share of problems. All the existing references to the data segment refer to where the data segment was expected to be loaded in memory, not to where the data segment actually is in the new executable. Because of this, all the references to the data segment in the executable have to be adjusted accordingly.

Ouch! Sounds fiddly.

It is. And took a lot of messing around to get right. Even then, that's not the entirety of the picture. For Companions of Xanth, the Legend game I used for testing when rebuilding the tool, the data segment has some extra gotcha's.. It contains segment references into the middle of the memory area RTLink segments are loaded into. Presumably these are used in some special controlled circumstances when a specific segment (or segments) are loaded to access particular data. But it's impossible to know without understanding the game a lot better. 

Worse, the presence of the references were screwing up some of the loaded dynamic segments in the disassembly, causing them to be split in half. To handle this, the tool explicitly looks for such "bad" references in the data segment, and removes the relocation entries for them. This way, the value in the data segment will remain as a static word, and the segments don't get incorrectly split up. The user can always then later manually set up a pointer to an appropriate segment if they wish. This handles the bulk of such errors, but Xanth at least, there are still references in the low part of the executable (that remains static in memory) to locations within the RTLink segments. Since I can't know which particular RTLink segment is meant to be loaded when the code they're in is called, these few remaining references will have to be later manually adjusted as well.

So that's it?

Yep. After all these years, I'm finally able to generate a (mostly valid) "decoded" executable, and produce a clean disassembly of Companions of Xanth. I also, initially, had two separate versions of the the tool, one the old hacky version for MADS games, and the legend variation for Legend-style RTLink usage. I've since updated my tool to properly handle MADS games, so now there's only the single rtlink_decode tool, and it can handle both variations 1 and 2.

Oh, wait.. what about the 3rd variation you mentioned?

Ah, yes, I didn't really get into that, did I. This version seems to be somewhat different than the other two variations. In this case, the RTLink code is stored in a separate rtlinkst.com file, and then loaded into memory. It then shifts part of the program downwards in memory, and uses it's own relocation table to manually process relocation entries on the shifted code. This variation is proving tricky to disassemble, so whilst I have located the segment list, I still need to:

• Figure out how relocation data is encoded. I think I've located the correct data in the executable, but the code RTLink uses to update relocation entries is pretty nasty and overcomplicated.
• How much of the start of the executable to remove so that the produced executable doesn't have any of the old code at the start of the executable that gets overwritten
• Find the thunk methods, and see whether the existing code will handle them.

Hopefully I can quickly figure out the remaining details for the third variation soon. The goal is to have a tool that both myself and others can use in the future to help them disassemble any game that used RTLink/Plus. Then no-one else will have to go through all the frustrations that I did trying to deal with this %#@! thing.

Sherlock Testing Resounding Success

Hi everyone,

It seems like the testing of the Sherlock games has been a success. There were lots of bug reported, and all the ones reported so far have been fixed. The foreign language versions haven't all been fully tested yet, but hopefully now the immediate crashes with conversations and inventory in both Serrated Scalpel and Rose Tattoo foreign language versions have been fixed, and the rest of the games can be tested. I'd like to thank everyone who's tested so far for your efforts, and feel free to post any more bugs you come across. Though hopefully there won't be too many more to find :). And if one else has copies of either game, particularly different foreign versions, any other testers would be appreciated.

Right at the moment I'm at the sweet point where all the outstanding bugs for Sherlock have been resolved, so unless new ones come in, I can turn my attention to other things.

So, whats coming next for me? Several things:

Serrated Scalpel 3DO

Serrated Scalpel 3DO still isn't completable. There are some areas, such as the darts game to fix up, and there are still some differences in sprite positioning that would need to be accounted for. It's somewhat constrained by the fact that we don't have any original source for it, and I don't have any experience with reverse engineering 3DO games. It may simply be a case of do as best we can with hardcoded fixes as necessary.

The 3DO version also has a few missing things compared to the PC version; notably it lacks the journal the PC version has. A "nice to have" for the future would be to reintroduce it, so that the 3DO version could be the definitive version of the game, containing all the PC version functionality as well as the video for all conversations. We'd likely create a tool that extracts necessary graphics for UI buttons and the journal background from the PC version, and produce a Dat file that ScummVM can use automatically when playing the 3DO version.

RTLink Overlay manager

Next, there's the RTLink/Plus overly handling in Companions of Xanth. I'd previously had some luck writing a tool to process Rex Nebular and create a flat executable with all the segments suitable for disassembling, but doing the same for Companions of Xanth proved elusive. Over the years I made several attempts, but none bore fruit. Until now. As of yesterday, I was finally able to write a new version of the tool that successfully generated a flat executable that could be disassembled. So one day, after a great deal of disassembly work, ScummVM may support the game.

My only disappointment is that RTLink/Plus seems to have had quite a number of variations. Rex and Xanth's RTLink mechanism were fundamentally similar, with all the RTLink code, segment list, and method thunks/stubs in the executable and/or overlay file. For several other games with RTLink that I tried, however, they seem to use a bizarre alternate method where a file called 'rtlinkst.com' is loaded, then an '.RTL' file for the game, and finally only then the game executable. Which means that my tool doesn't work with them, and I'd need to figure out this new mechanism from scratch if I want the tool to be general-purpose enough to handle any RTLink game anyone might want to use it on in the future.

Guess that can be another long-term project to muck around with. Hopefully it won't take as long as it did to finally get Xanth properly disassembled. :). I'll probably make another posting in a day or so about RTLink in more detail, for those that are interested.

Might & Magic, World of Xeen

Next there's my work on re-implementing Might & Magic - World of Xeen (and Clouds of Xeen, Dark Side of Xeen, and Swords of Xeen). With some free time last weekend, I finally returned to working on them, and likewise was finally able to properly disassemble the  algorithm the original used for scaling. So as of now, sprites are now correctly scaled, as before I was only using a rough guess scaling code I'd nicked from another game engine. I've also fixed some other drawing bugs for drawing outdoor areas (outside town). So as of now the game scenes now display properly! :)

That's right, you can now walk around, fight monsters, go visit the various buildings in town, and even leave the town! In fact, most of the functionality for the games are already implemented. Apart from lots of testing and minor bugfixes that will be needed, only the following major areas remain to be implemented:

• Introduction/ending sequences for the games
• character management, and title screens.
• Sound. I'm hoping I can simply slot in one of ScummVM's audio decoders without much further work.
• Savegames

At the moment I'm concentrating on getting World of Xeen (which combines Might & Magic 4 and 5 together) working. But then afterwards I'll implement separate support for 4 and 5, as well as for Swords of Xeen. It might even be feasible to handle Might and Magic 3 - Isles of Terra as well, since I'm given to understand the engines are nearly the same.

Those interested in following the progress can see it at the brand-spanking new RogueVM Github account. That's right; after all the years of idle talk, I've finally set up a place to properly store RPG related game engines. No website yet, but at least it's a start. :) I'll likely spend the near-term focusing mostly on finishing support for the game before I move onto any other adventure games, considering how far along the engine is already.

Return of the Phantom

Strangerke has put a lot of work recently into implementing scene logic for Return of Phantom, the next MADS game that was published after Rex Nebular. There are quite a few stubs for missing engine functionality that was added in though. So when I do return to working on adventures, it will likely be to assist him in completing the game.

DreamMaster.