Dreammaster's disassembly blog

Two milestones reached

2011-10-23T16:48:00.002+11:00

Yesterday I was chatting with Md5 about the recent progress in Blue Force, and I thought to myself that I should make a blog posting. Looking back, it seems it was June when I made the previously posting. I hadn't realised that it was that long.

As those of who frequent the IRC channel or forums would be aware, Ringworld was completed, and will be part of the upcoming official 1.4 release. We'd already had a successful testing period, with several bugs identified and fixed. It'll be nice to see another game I've worked on added to the trunk and become officially supported.

Now, onto more recent developments. As of yesterday, Strangerke and I have reached two more milestones in the progress of the TsAGE engine.

The first is that I completed the first complete play-through of Blue Force. This had been a gradual process over the last few weeks, as we identified game stopping bugs and fixed them. The save format also changed several times during this period, as we implemented the final few game scenes, and added missing game state variables. Yes, clone2727, the game is completable. So nyaaa. :)

The next step will be for us to start playing the game again from the beginning, comparing it against the original and looking for any minor graphics glitches or problems that we missed. We will also ensure that none of the fixes done for the later game scenes had any adverse effect on the earlier scenes. This includes a variety of minor issues that we identified during the first play through, but were deemed too minor to immediately worry about.

One of the more amusing glitches I came across was a scene in day 2 of Blue Force where a police inspection is done. This particular scene seems to use animation sprites different from anywhere else in both Blue Force and Ringworld. Likely some as yet un-handled flag or sprite offset - the result being that all the officers get drawn with their bodies on top of their head. Picture if you will headless officers walking around balancing on their own heads. :)

The second milestone is the commit of the beginnings of the ‘Return to Ringworld’ game - the third and final 2-D TsAGE adventure. I’d previously spent some time on the side starting to look at a disassembly of the game, and identifying all the common engine functions against my disassemblies of Ringworld and Blue Force, and got to the point where all the core functionality had been identified. I started on adding the game classes about a week ago, and have got to the point where I can at least display the first in-game scene.

I had been holding off on committing this to trunk until after the branch, since it affects a lot of the files, but now the 1.4 branch has been done. It also invalidates the current Blue Force savegame format (but not RIngworld) - since both of them share mostly the same UI code, it made sense to encapsulate it in a common base class. Given that Blue Force isn't official yet, it was easier to break the existing savegame format for it as part of this, then to try and maintain the savegames intact. So it was good that I’ve now completed the first play through of Blue Force, and can start from the beginning of the game again.

I have a tendency, for games I plan to work on disassembling, not to play them through using the original executables. That way I can have the fun of gradually playing through the game as I implement the code for it in ScummVM, and get the enjoyment from playing it for the first time in code that I’ve written. It also serves as an inducement to work hard, so I can see what happens in the game. I’m eager to see what happens to Quinn, Miranda, and Seeker in their second foray to the Ringworld.

Once we've cleaned up the Blue Force code-base a bit more, then work on Return to Ringworld will begin in earnest. Hopefully there won't be too many core engine changes, and work can proceed on it as rapidly as was done for Blue Force.

On a final note, for those of you wondering about Rex Nebular, I did start to return to looking at it, but I've been away from working on it in earnest for too long. There are still parts of the game not yet dis-assembled, and I'd need time to complete that and re-familiarise with the existing code. So it really needs some time when I'm not actively working on some other game. I'll have to see how I do when the TsAGE games are complete.

Let there be sound

2011-06-21T11:02:00.003+10:00

The TsAGE engine for Ringworld now has sound playing! I'd be hard to pressed to express just how happy I was when I ran ScummVM and finally had correct sound playing back. I was like, Oh My God! it's finally working. After all the time spent re-implementing all the sound code from the original, and then spending nearly as much time slowly tracing through the execution of the original versus ScummVM to try and figuring out the cause of differences, it was a relief to finally have it working.

As such, the Ringworld game is that much closer to being complete. There are still a few areas left to look into, and polish up, such as:
* Sound fading isn't yet working correctly
* I need to hook the sound manager into the savegame format, and ensure playing sounds are persisted
* There are certain suspect areas in the sound manager's main _sfRethinkVoiceTypes method to review; at least one variable is being assigned to at one point and not currently being used, so I need to double check if that's really the case.
* The sound manager could use an overall clean-up.

It won't be too much longer until it's ready for public testing. Soon Larry Niven fans will be able to explore the Ringworld, and search for a way to defeat the bloodthirsty Kzins.

So what's coming next? Well, I've promised myself some time to work on Rex Nebular again, which I've somewhat neglected in my zest to have Ringworld completed. Strangerke is also enthusiastic about working on the other games that used the TsAGE engine, so I'll probably be spending some time on the game Blue Ice to determine what changes have been made to the engine, and provide a basis for being able to re-implement the game scene logic similar to what was done for Ringworld. Finally, I may offer to help out somewhat on the other games the team has received original source for. Gotta keep busy after all. :)

TsAGE sound support progressing

2011-06-13T12:09:00.007+10:00

Work on the TsAGE sound has been progressing. I had originally hoped that the raw format used by the game would be a standard Adlib raw format that I could feed into one of the existing Adlib Sound FX classes. I tried to pass the data to the player class in the Cruise engine, which I had previously worked on, but unfortunately, whilst it didn't crash, it didn't play anything. And I'm not familiar enough with sound formats to have been able to diagnose why.

As such, I've spent some time powering through implementing all the relevant sound code directly from the demo executable. With the symbols identified, I at least had a heads up for the original engine names for a lot of the methods. Plus, we were able to garner some additional information from a demo version of the game Protostar. Whilst it's not an adventure game, the sound code has a lot in common with Ringworld; and not only did it have method names, it also had a lot of field information for the classes and structures. As with the Ringworld demo, it didn't have it for all the classes, but I was at least able to name fields in the sound classes and the sound manager.

From this basis I've now implemented what I think are all the relevant methods of the demo for both the Adlib sound driver and the sound manager classes - I'm currently using the OPL functionality with OPLWriteReg in place of direct port access. The resulting code I've completed doesn't actually play anything back yet, though. Making a log of the underlying writes to the Adlib port versus my ScummVM code, the results are different, so there's obviously some incorrectly implemented code.

My current major suspect is a method called _sfRethinkVoiceTypes - it's a enormously big method with over 500 lines, responsible for updating the voice channels based on queued sounds. It's one big mess, with a large pre-processing, post-processing, and main processing areas. I'll have to slowly trace through the execution both in ScummVM and the demo in the DOSBox Debugger to identify the problem. Hopefully once I've identified the differences, I'll finally hear some sound output.

Fingers crossed. :)

Most wonderful Ringworld demo

2011-04-24T20:19:00.009+10:00

Hopefully everyone had a happy Easter weekend; or for those of other religions, were able to enjoy a long weekend wherever you are. :)

It's been a little over a week since the TsAGE engine hit the trunk, and a big thanks go out to everyone who's helped out with commits - fixing compilation problems, and cleaning up the code. Special thanks again go to Strangerke, who has been busy working his way through the Ringworld game to fix bugs and make the game completable. I too have spent most of my time fixing some of the outstanding bugs, as well as newly identified bugs in the core engine.

On another topic, we've had a stroke of exceptional luck. I had been starting to get bogged down in all the various classes related to sound handling, and I wasn't looking forward to trying to figure out what it all did. But then Strangerke discovered that one of the two available Ringworld demos actually had embedded symbol information for most of the variables and methods of the demo!

Since the full game engine isn't anticipated to be much different from that used in the demo, it likewise means we've suddenly got a huge jump forward in understanding the remaining sound-related parts of the engine. Fuzzie was able to extract the information into a text file listing, and I wrote a quick and dirty script in IDA's IDC language to load the symbol information into a new IDB file for the demo executable.

This isn't quite as good as getting the full original source for the game, but it's still a major find. The biggest limitation of the symbol information is that such symbol tables don't include information for fields within structures. So, for example, the 'ASound' class has 282h bytes of information, but the instance of the class only has the symbol for the first byte, not all of the bytes. Therefore, I'll still need to spend a reasonable amount of time figuring out the contents of the various methods. But at least with the method names known, I can more easily figure out what each method actually does.

In the space of under two days, I've already implemented support for the two Ringworld demos. Luckily, the demos use a single script sequence rather than direct code, which made it very easy to add support for. All that still needs to be implemented for the demo are a few custom dialogs, and ensuring none of the full game functionality, like Saving and Restoring, is enabled for it.

Implementing support for the demo also identified a key difference between the two versions of the demo - the second demo was similar to the CD version, in that they changed the format of how the data for priority regions and walkable areas was stored. Having a disassembly of the demo with symbols made it easier to identify the relevant areas of code, and figure out the differences.

In addition to polishing up support for the demo, I'll likely making a branch to work on the sound classes separately. We currently have some hard-coded logic in the existing engine to properly signal actions that any played sound has immediately ended, so I don't want to interface with that whilst working on a proper implementation of the sound system.

So things are definitely rapidly moving forward. :)

TsAGE has now hit the trunk

2011-04-13T23:32:00.006+10:00

Hi all,

I'd been meaning to write a new blog post for a while, but progress was progressing so quickly, that I never quite got around to it. Now, though, work has progressed far enough that the tSage engine has now been merged into trunk. Huzzah! :)

Many thanks go to Strangerke, who joined me in implementing all the various game scenes of the Ringworld game. Together, we've implemented all the game logic.

The following is the state of the game:
* All the game scenes for the floppy version of the game are implemented, although some still need some bug-fixing, so the game isn't completable.
* Sound isn't supported yet, and still needs to be reverse engineered in the original game.

So we have the following goals for the near future:
* We'll be working on fixing the remaining scene bugs, and ensuring the game runs smoothly from start to finish.
* Implementing the sound code.
* I also want to implement support for the original game's cheating/debug functionality.
* Add support for the CD version of the game, which has some minor differences.
* And add support for the demo version, if possible.

All in all, it's a celebration for another engine being added to the ScummVM universe. :)

tSage engine proceeding quickly

2010-12-19T23:55:00.010+11:00

Hi all,

I've really been making some great strides forward with my implementation of the tSage engine. I offer the two screenshots below. One is from the first game scene when run in DOSBox, the other under ScummVM:

Hard to tell them apart, isn't it? :). The one on the left is from DOSBox, and the one on the right is ScummVM. All the initial hotspots in the first game room are now set up, are being drawn on the screen, and have correct look descriptions. The only exception is a minor graphic glitch with the doorway - the current drawing code doesn't yet support clipping objects based on priority calculations, so the top of the doorway isn't correctly clipped off.

One of the next major steps from here is to delve further into the movement classes and code. Whilst the player is now drawn on-screen, I don't yet have any code responsible for moving the player, such as when you manually walk somewhere, or when the computer moves the player to a set location, such as when you use the door beam.

From what little I've looked into the movement code so far, the game maintains a set of classes responsible for moving objects within the scene. For the player specifically, there's a lot of code that seems to make use of special 'walk regions', which indicate where on the screen the user can walk. The movement code is tied into this, and uses them to determine both where the player can move to, and for pathfinding to move the player to the given destination.

Given the pace I've been having recently in other areas of the engine, I don't anticipate that it will take too long. Hopefully, soon I'll be able to move the player about on the screen. Then I can start delving into implementing all the logic involved in the first cut-scene, where the Kzin comes to the door and speaks with the player. It's so much more fun working on an engine when you can almost day-by-day make visible progress in it's implementation.

PS: I've been making such rapid progress with tSage, I've been somewhat neglecting work on the MADS engine. I hope any fans of Rex Nebular will be patient with me.. I will get back to work on it eventually. :)

Update on the game engines

2010-11-21T17:48:00.003+11:00

Hi all,

It's been a while since my last blog posting, so I figure it was time to wax poetic on what I'm currently up to.

I've spent some considerable time working on the Broken Sword 2.5 engine with Sev. As has happened previously, what I originally intended as something else to work on alongside Rex Nebular, quickly overshadowed it, and I ended up spending most of my recent holiday just working on it. It's now available in the public SVN, and is running fairly well, so I'm going to leave any remaining work to any other interested parties, and have returned to work on my other engines.

-----

Rex Nebular is coming along nicely. As I've mentioned in the past, I've been creating a custom script engine, based on the Tinsel script interpreter, to handle interpretation of game scripts. I've then created a custom script program that iterates over specified functions in the Rex Nebular disassembly, and converts them to a script text file. I then have a simple script compiler I've developed that compiles the scripts, and creates a new 'mads.dat' file.

As of right now, I've only converted the scripts associated with the first game room. The scripts associated with scene entry and 'scene tick/step' are working properly, so Rex is now waking up and standing entirely based on script execution, rather than the hard coded behaviour I previously had. I consider this a big step, since it validates the entire process of converting original executable instructions to script code, and final execution works properly.

The next step is to implement some remaining missing pieces of the action dispatch code so that Rex can actually do actions, rather than just walking about. Right now, I'm aiming for right-click Look actions to work properly. Then it should only be minor further changes to have other actions work as well. Soon Rex will be able to do all the various things in the first room. And I'm looking forward to the day when Rex will finally be able to take his first steps into a bigger world and finally leave the cockpit.. :)

----

Another game engine I'm currently working on is the tSage engine, which is used by the Ringworld game. Previously, the fact that it was so Object Oriented annoyed me, because it made it harder to understand the game flow since some classes, such as the event manager, had four descendant classes, making it difficult to understand what each class in the hierarchy did. Now, though, I'm actually starting to see some benefit from it.

The classes related to handling the game state are very clean. For example, there is a generic 'Scene' class, from which each game scene dervices a specific implementation class. So, there is a 'Scene30' class for the first scene in the game, and it has fields for the various hotspots and objects within the scene. Speaking of hotspots, there too is a hierarchy, with a base 'SavedObject' (used for object persistence), which subclasses to a 'SceneHotspot' (used for basic hotspots), which subclasses to a SceneObject (which is an object in the scene that is drawn into it, rather than being part of the background). From there, a SceneObject is sub-classed to the NPC class - this is where the encapsulation really helps, since all logic related to drawing NPC's is already handled by the SceneObject class, so didn't need to be disassembled multiple times for different kinds of things in the scene. The NPC class only has to implement the additional logic for moving character around the scene.

In fact, the NPC class also further sub-classes to the 'Player' class, which has extra logic to handle allowing the player to interact within the game. A lot of the logic for handling the player is thus already handled by the NPC class, so the Player class only has to add on the extra logic for allowing direct player control.

Because of this OO model, all the game logic is implemented by deriving new custom classes from these base classes, and then adding them into a particular scene. So, for example, the first scene in the game has the player having to ring a doorbell by 'using' a motion sensing beam. This beam is implemented as a 'SceneObject' derived custom class, which overrides the 'action' method to either provide a description on-screen for the 'Look' action, and start the 'kzin answering the door' sequence in response to a 'Use'.

I'm still undecided about whether I'll re-implement the game by manually converting all the game logic to C++ code, or try to create a process, as with Rex Nebular, to convert the game executable routines into equivalent scripts that could be interpreted. The OO nature of the game engine would make it trickier to convert to scripts. I'm planning to at least, eventually, converting the logic for the first scene to full C++, and see how it looks.

Rex can now walk about on-screen

2010-07-27T13:34:00.004+10:00

Here I was bracing myself for the imagined complexities of movement logic, and it turned out to be a whole lot simpler than I imagined. Probably blame it on the other two engines I've had experience with path-finding for: Lure, and Tinsel, both of which have their share of complicated logic. For Rex Nebular, though, the implementation was a lot more elegant than I'd expected.

What the MADS engine does for walkable areas is to have two parts. The first is a priority/depth surface, which had already been previously implemented. This is a surface which both specifies the walkable areas, and is also used to determine which parts of an object, at a given 'depth', should be drawn on-screen. In the first room, for example, the area comprising the chair has some varying levels so that the player gets drawn behind it.

The other part to the movement functionality is a series of 'scene nodes'. These are a series of points distributed across the scene to represent appropriate intermediate points in movement. If a straight line between the start and end point isn't possible, because of a marked obstruction, then the engine calculates the distance between each node in the scene, including both the start and end points as two extra nodes. It then calculates which series of nodes will take the shortest amount of time to traverse, and uses that as the basis for movement - the actual walking then moves the player from node to node until the destination is reached.

This works well for the game since, unlike in Tinsel or Lure, there aren't ever any moving actors that could interrupt the walking path. The only NPC character movement that ever happens is in cut-scenes, where the player is either not moving, or is being controlled by the computer (such as when Rex is taken to the cells).

Now that I've got Rex moving around on the screen, the next immediate step is to get the hotspot and selection code working. Somewhat ironically, this is turning out to be more complicated than the walking code itself. I'm currently working my way through disassembling the various routines and implementing comparable code. Hopefully, I'll soon have enough implemented that I can properly implement some actual actions in the first room.

MADS/M4 engine now has cutscene support

2010-07-12T18:42:00.005+10:00

Well, it's been a very productive month for work on the MADS/M4 engine.

Since the last posting, I started working on the cutscene animation system. This is a separate module to the basic cycled animations that get loaded into a screen.. rather, the game uses an animation manager to handle so-called 'cutscene animations', such as that of Rex waking up in the first room. In this case, the animation manager handles the work of animating the sequence of frames of Rex waking up, looking around, and then getting up. Given that this was the fist major action in the game, it made sense to have that properly implemented first.

Above and beyond that, the animation manager is also used for handling all the full-screen animations done in both the introduction and ending sequences, so once I had Rex properly animated in the first scene, I moved onto implementing proper support for them. With some help from Md5 on the handling of digital voices during the intro, the introduction sequence now plays completely if you press 'F3' from the main game menu. There are only a minor few issues remaining with the introduction, in that:

The tempo of the intro isn't quite matched to the original yet; I'll need to properly calibrate the timing loops at same stage
I haven't implemented palette cycling, which is used by some scenes within the introduction sequence to simulate basic animation.

I consider the above minor things, and I'll get to them eventually. For now, I again got distracted from polishing up the remaining issues by making a breakthrough in my understanding of the code that controls the player (you) within the screen. Within the last week I've implemented a great deal of code associated with player control, and now Rex is corrected displayed when the wakeup animation is completed. Not only that, but the idle animations run for Rex, and you can even turn him around by using the Ctrl-T key.

The obvious next stage for me to look into is actually starting to give some movement control for Rex, and have him be able to walk around on-screen. This is where I'll need to worry about figuring out the pathfinding algorithm, and how actual movement is done. I've already identified some of the methods associated with handling Rex's movement but, as I expected, there's a lot of calls to unknown sub-methods that I'll need to figure out the purpose of. Since the MADS engine was designed to be generic for multiple games, there's a lot of complexity in both the movement and pathfinding code that I'll need to figure out.

I'm really looking forward to this. It'll be another major milestone reached when I can move Rex around the room. At that point the game will finally be properly interactive. I'll then be able to work on interacting with objects, which I already understand a lot about, and maybe even moving between rooms and/or teleporting.

MADS engine reaches a milestone

2010-05-23T17:47:00.014+10:00

Well, it's been a busy last month at work, but with things starting to settle down, I've finally had some free time to start working on the MADS engine again. And as of this weekend, I've just reached an important milestone in the development of the ScummVM engine implementaiton - the core graphics display framework is now working properly.

That's right.. the game logic for the first screen setup is still hardcoded, but the engine now properly implements all the necessary code to handle the calls to register a sprite sequence animation, and drawing elements on the screen; so animation sequences are now properly animated.

I consider that this reperesents a great step forward, since the entire game is based on the display of graphics - so I finally have a working core from which to build on further functionality with. This includes areas such as the logic for the player character itself.

The graphics core in the MADS engine has been more complicated then I've previously dealt with - the functionality is split between a main sequencing list for scheduling animations, graphics display logic, and various suppport classes, such as for dirty area handling. There's a whole mess of inter-relating code, which meant that I had to disassemble a lot of the code before I was able to implement something that works. There are still some fields that I've yet to figure out that handle certain special cases, such as the ability of dynamic hotspots to move their position, but that'll come in time.

If there's one thing that annoys me about the MADS engine it's the number of data variables that seem to get set everywhere. I'm presuming the original engine was designed to be generic for multiple games, but there seems to be a ridiculously large number of data values being set throughout the engine. It'll be interesting to find out whether the the majority of them are part of the 'core' MADS engine, or represent a whole lot of game-specific hacks for the Rex Nebular game.

I've included a screenshot from the first game room from Rex Nebular below. It may not be obvious from a still picture, but the various status displays now display flashing lights, and bubbles are escaping from the hull into the water.

MADS disassembly is back on track

2010-03-01T20:54:00.005+11:00

Since it's been a month since my last posting, I thought it was time to post a status update. The work on disassembly of the MADS (Rex Nebular) engine has been going well.. I've made a lot of progress on various areas of the game logic.

I haven't actually added much code to the ScummVM engine yet, since I decided it would be better to fully understand a given area before I implement it. At the moment, my reverse engineering has been some undirected - it's obviously more effective to work on whatever areas you can find that are easiest to disassemble, which in turn may make other areas easy to disassemble later on. However, it does mean that there are bits and pieces across the entire engine that are now understood better.

One area of the game engine that's going to prove interesting is all the hard-coded game logic - each scene has several routines associated with it for scene loading, scene animations, and handling interaction with elements. Obviously it would be preferable not to completely reimplement these in code, since it would take a lot of code space, and there are two other games whose code would also need to be reimplemented.

What will probably need to be done is to create some kind of basic script interpreter engine, and convert the various scene routines to an interpretable script. Luckily, most of the routines seem to use a fairly simple subset of x86 instructions, so it shouldn't prove an impossible task. In any case, that's something for the future, when I fully understand all the various routines the methods call. For now, I've set a short term goal of manually implementing the logic for the first game room, and will return to the idea of scripts at a later date.

On a final note, for those of you who like pictures, see below for an example of the scene walkable areas for the fist scene of Rex Nebular. It also has a slightly weird sprite frame from Rex inserted into the picture - Rex isn't movable yet, though.. it's just a test from some of the loaded sprites:

Roadblocks in the path of tSage

2010-01-29T13:00:00.017+11:00

Those of you who have been following the IRC channels will know that I made some real progress in the disassembly of the tSage engine, used by the Ringworld - Revenge of the Patriarch game, as well as several others. I hadn't really gotten into any of the game logic, but I'd made great strides in decoding the image formats and graphics display, so much so that my engine on Scummvm-misc could display test dialogs, as well as the right-click game dialog with proper highlighting and everything. Sample screenshots below:

However, given the perversity of life, I should have known better than to announce on the channel that I was making real progress. :(. Because as I was getting into the game execution logic, I began to realise just how complicated they made the engine. Whilst using C++ has it's advantages from the perspective of RE, since it does group related functions close together with the variables they use, in this case, they seemed to go overboard with it.

A preliminary look at the main event loop of the game shows they implemented a stack of "action objects" - multiple independent event handlers can be registered and control swapped form one to the other. I'm not sure, but I presume that this is to allow for things like cutscenes, where standard game operation is superseded.

For even more complexity, they went with a class hierarchy for the action objects which is at least four levels deep!.
The classes, as far as I've been able to determine, are as follows:

They have a core 'SavedObject' class, which has some internal logic for registering each object instance in a central list. I presume this is for automating saving and loading.
A child class which I'll call "SmartPointer". This class seems to encapsulate a pointer to a secondary object, and provides core logic for 'fixing' up the pointer reference. I can't be sure yet, but there's a lot of 'fixup' code in the engine, which I presume handles restoring pointer references between objects when a savegame is loaded, and all the objects are re-constructed. This class also implement a basic virtual table of processing and dispatch methods that automatically defer to the foreign object if one is set
Yet another child class which introduces basic event manager functionality amongst other things
A concrete fourth child class which seems to further override some of the event manager functionality, and is what gets added to the game's "action list".

As you can imagine, all this makes it somewhat different to understand what the blazes is going on. I've also examined the engine from some other points, such as scene loading, and even that seems to be overly complex, with method calling method - they really went in the direction of generalised/generic functionality, with the result that the code seems to be much more complex than it should otherwise need to be.

</whinge>

In any case, the tSage engine was only meant to be a part-time RE effort for when I needed a break from the MADS engine. I did somewhat get carried away with it for the last few weeks, but that was only because the areas I was looking at proved so easy to reverse engineer. Now that I'm running into more complex code, I've decided that the break's over, and I'm returning to work on the MADS engine again. For that, at least, the code seems somewhat simpler, even if it can be a pain to run in the DOSBOX debugger because the segments keep getting shifted around in memory. ;)

For those of you who are Ringworld fans, don't worry, I'll still be working on the engine. It's just there won't be as rapid a progress on it as I'd recently hoped.

Working on Rex Nebular

2009-12-07T19:41:00.003+11:00

The Scummvm-misc listing welcomes the new work in progress Dark Seed 2 engine by DrMcCoy. It looks very promising.

I've returned to spending a lot of my time working on the disassembly of Rex Nebular, and implementing it's functionality into the M4 engine. It's likely going to be an extended process, because the game engine is unbelievably complex. It likely comes from the MADS engine being designed as a generic game engine rather than just for single games specifically.

For example, I only just last night disassembled some of the code associated with displaying messages. Any given message is represented by a unique 'Message Id', which in turn loads a message resource that can contain an entire series of control elements for formatting dialogs. A sample message resource for looking at the curtains in the starting room is as follows:

[title25][sentence].Wincing at the "Drapes Of Wrath", you swear
to use them to light the fire when
you burn your ex-decorator at the stake

I'm dreading how complicated the actual script interpreter is going to be. :P

New SourceForge project - Scummvm-misc

2009-10-31T20:01:00.012+11:00

It's been a busy month. As those of you who follow the mailing list will be aware, I finished work on an engine module called Gargoyle over a month ago, that allows the playing of Infocom z-code text adventure games. Unfortunately, the consensus was that it really didn't fit in with the ScummVM theme of point and click adventure games. So I created a new project on SourceForge named Scummvm-misc to host the project. But I'd been so busy that I've only now had time to create a proper homepage for the project.

See Scummvm-misc for the project's new website. Rather than just create a project specifically for Gargoyle, I decided to keep the project name generic.. Gargoyle isn't the first non-point 'n click adventure engine, and there may be more in the future. ScummVM provides such a solid base for creating games that can run on multiple systems, I figured it would be good to keep track of other such engines.

The index page I've created also lists all the other publicly known ScummVM engines that are either in development or for which work has stalled. Anybody in the future who'd like to start work on any new engine, and wants the source to be publically available, is welcome to share SVN space on the project.

Finally, two screenshots from the later Infocom games, that used a combination of graphics and text:

Cruise for a Corpse Complete

2009-07-08T11:11:00.010+10:00

After many months of work, the Cruise for a Corpse engine plays the PC version of the game all the way through without any bugs.. I was also able to figure out and implement music and sound effects as well, although currently only the PC Adlib player is supported.

The Amiga and Atari ST versions of the game will probably play all the way through, although music and sound effects handling will need to be implemented for them.. I'll leave it up someone else more familiar with those systems to take care of that sometime in the future.

It's when you're diving into fixing and extending engine code like this, the old maxim 'The devil is in the details' comes to mind. Even the smallest mis-print or mistake in the code can cause errors that are hard to track down. It's actually kind of a relief that, as far as I know, it's all over and done with now.

Anyway, I plan to spend the rest of my vacation time (I'm on holiday until the 22nd) getting in some R&R, and some more generic mucking around on ScummVM - this will probably include some time spent working on the MADS disassembly, so people can look forward to further work on Rex Nebular in the near future. :)

Foray into Cruise for a Corpse

2009-05-22T14:10:00.003+10:00

Hello everyone. It's been a while since my last blog posting.. I really should get into a better habit of posting updates more often.

Having largely finished adding Discworld 2 support to the ScummVM Tinsel engine some months ago, rather than immediately returning to work on MADS, or some other project, I decided to take a detour and work on the existing 'Cruise' engine, which plays the game 'Cruise for a Corpse'. This was an existing incomplete engine in the ScummVM SVN, which had been languishing for a while without any significant progress being done.

Having been working on it since late February, it's now in a somewhat cleaner and more stable state than it was previously. A lot of bugs have been fixed, and I've also done a lot of cleanup of the code as well, such as giving structure variables more proper names, fixing up variable types, etc.

I'd originally hoped to have reached a milestone of having played the game all the way through to the end before I made this blog posting, but my debugging practices almost demand that I fix these bugs before carrying on with the game.. those remaining bugs tend to be annoying buggers.. I'm currently grappling with two really nasty ones:
* I've got an animation of a moving handle in the engine room that is drawn incorrectly for only two of it's frames.. it could be a problem with the transparency mask for the image, but all fixes I've tried so far screw up the decoding/display of every other image in the game.
* There's a bug in the path-finding code code that's particularly noticeable in the area below the top deck, in that you can walk through walls directly from one exit to another.

I was really on a roll up until I discovered those bugs.. I had reached the 1:30pm segment (the game is divided into time segments that only advance once you've done certain actions) and had fixed several crashing bugs, and was hoping the end of the game was in sight. Now it will have to wait until I've fixed these bugs.

I'm hoping to get Cruise to a nicely completable stage soon. The only other major remaining area in the game after that, apart from some minor ones like getting the game speed exactly the same as the original, will be the sound support. I was given a suggestion that I might be able to copy and paste the code from the previous version of the engine (cine), but whilst the formats seem similar, and I was able to get some sound playing in my experiments, the sounds played are incorrect.

As I've always been on record as saying, music handling is my least favourite area to work in. I may try some further experiments to get it working, and if i still have trouble, I may simply drop it completely, and leave it to anyone else who might be interested in taking it up in the future. At least they'll have a more solid game base to add onto.

On a final note, my holidays for this year were severely delayed, so I'm still looking forward to a month's holiday at some point. I anticipate some productive time spent happily disassembling and implementing games for ScummVM. :)

Discworld 1 Demo support added to SVN

2008-12-14T15:40:00.002+11:00

Whilst it's nowhere near as spectacular as adding support for Discworld 2, the latest SVN code now supports the original Discworld 1 Demo.

This demo is somewhat interesting in that it was built at an early stage during the development of the first game. As such, it used a more primitive version of the source code used in the final game. Luckily, it didn't prove too hard to cross-reference a disassembly of the demo executable and identify commonalities between it and the DW1 v1 source code, and from there work out what had changed.

The demo version I used during development was downloaded from Adventure-Treff.

There were actually two demos released for DW1 - this first one contained only the two 'rooms'.. the Fishmonger's and Alley, and the single objective of getting the belt buckle. Please note that this version does not have any sound.

The second demo released apparently contained most of Act I, and featured full voices and SFX. If anyone knows where a copy of that version can be located, I'd be interested in making sure that that version also works with our Tinsel engine.

Discworld 2 support added to SVN

2008-12-02T14:27:00.004+11:00

Well, it's been a little over three months since my last posting, and now my current efforts are finally complete.. as of last night, Discworld 2 support has been committed into the ScummVM trunk.

First things first - setup instructions will probably be added into the main README shortly, but for those of you who just can't wait to get it up and running need to do the following steps:
copy the contents of both CDs into a folder on your hard disk, and rename the ENGLISH.* files on both CDs into ENGLISH1.* and ENGLISH2.* respectively. That goes for other foreign language versions the same, with whatever the language name being.

Remember that Discworld 2 isn't officially supported by ScummVM yet. To play it, you'll need the latest daily build from at least 2/12/08, or build it yourself. We also don't have a Discworld category under the ScummVM Bug Tracker, so any issues would be best discussed in the IRC channel or on the forums.

Special thanks go to DrMcCoy, who pitched in and implemented the changes to the sound/music handling for Discworld 2. Also to the rest of the team who participated in the Discworld 1 development, who provided the occasional code additions and testing during integration of the Discworld 2 code.

In case anyone's been wondering about my progress on the M4 engine, well, my apologies, but wouldn't you know that barely a few days had passed after my last blog posting when we got the Discworld 2 source. So I never really had a chance to delve much further into the graphics system like I said I would. Never fret, though, it's still a long term goal to finish disassembling Rex Nebular and start adding support for it, although I'll probably put it on hiatus again if the team gets given the source code for any other game that needs implementing. As fun as reverse engineering is for me, it's nice to work on a project where you can see rapid results from your work.

What's next for me? Now that the code's been publically released, I plan to move onto adding support for the Discworld 1 demo. I've already been spending some off time working on a disassembly of the executable, cross-referencing methods against the Discworld 1 source, and it shouldn't be long before I can start adding demo-specific code to the Tinsel engine.

Oh, and of course, I've still got copies of LittleBigPlanet and Fallout 3 on the shelf waiting to be played. They've been calling out to me in the middle of the night, saying 'Play me, play me', but I've been firm.. I wasn't going to play them until Discworld 2 was finished. Now that it has, I expect to spend some quality time this Christmas traversing the wasteland, and blowing the crap out of stuff. :)

2008-07-23T19:47:00.003+10:00

Well, it's been a little under two months, and now the Tinsel engine has been released to an eagerly awaiting public. I just hope people don't get too distracted playing it that they forego testing the other games for the 0.12 release. :) Currently Discworld 1 is supported, but not the demo. Kudos to everyone else involved in bringing the conversion to ScummVM to fruition so quickly.

On a more technical note, those of you who are proficient with using the debugging console may wish to experiment with the console in this game.. there are several commands, such as "scene", which allows you to jump to any particular scene number. Another nice one is the "sound" command.. I particularly like "sound 5931". :)

Given the amount of time spent on the Tinsel engine, I haven't had much time to work on the disassembly of Rex Nebular. So there hasn't been much to talk about on my blog. But now I'm back into it again. When last I left, I was starting to plunge into the depths of the dialog display code. This actually proved a useful starting point because:

1) The difficulty dialog is the first thing displayed by the game as it starts, which makes it easy and quick to trace to when running the game in the debugger.

2) We already more or less know both the screen and sprite format used by the engine, which makes it easier to understand and decode the related routines.

3) It also uses font display, which again, allows me to quickly identify the appropriate routines.

4) It allows me a starting point for identifying various other routines, such as mouse/cursor handling, event loops, and graphics routines.

As you can see, the dialog code provides a useful starting point for understanding core functionality used by the game. By identifying these routines, it will make it easier later on to understand other game routines that make use of them. The only problem is that the dialog display code seems to be unreasonably complicated - it seems they tried to used the same code to potentially handle various dialog text, such as headers, control options, and savegame names.

2008-05-23T19:40:00.003+10:00

It's a real pleasure to be able to be able to return to proper disassembly work after the interval I had to spend writing the delinker tool. Subjectively, I know that writing the tool was a necessary step that would save me issues in the long run, but it was still annoying to have to write and then spend all the time fiddling with it to get it work, particularly since I'd never really bothered to look at the structure of executables, and was learning from scratch.

Getting stuck into Rex Nebular, I've started debugging it using the DOSBOX debugger. I was a bit unsure about using it at first, but it's turned out to be an excellent tool. Back when I was disassembling Lure of the Temptress, I made do with DEBUG, which wasn't bad, but obviously had issues with trying to debug graphics programs.. I'd actually had to hack the game executable to stop it setting graphics mode so that I could use DEBUG to trace through all the startup code. Now with DOSBOX, I can trace anywhere in the program, since the debugger itself is in a separate window, which is wonderful.

About the only real annoyance remaining is the RTLink library itself. Although I do have a full disassembly based on a processed executable, I still need to do my debugging on the original executable (if the game worked without the extra space it provides, they wouldn't have needed it in the first place). The issue is that due to the way RTLink works, a call through a proxy stub can actually unload the calling code, which then gets reloaded when the called function exits.. but not necessarily in the same place.

Which means that you might try stepping over a function call, and suddenly the entire game sequence will play out, because the location returned to after the function call may now be at another segment address. Confused me at first, until I happened to trace out of a method and realised what was happening. :)

My first posting

2008-05-14T18:47:00.004+10:00

Welcome, everyone, to my new blog. I figured that since blogs have been becoming popular in the ScummVM community lately, I might as well start my own as well. My interests lie in disassembling and reverse engineering old adventure games, and then re-implementing them in ScummVM.

For this inaugural blog post, I'd like to talk about one of my latest projects - working on the MADS/M4 engine. I'm concentrating mainly on the MADS side of the engine at the moment, which was the game engine used for the three games - Rex Nebular, Dragonsphere, or Return of the Phantom, what the current status is of the engine.

I've been concentrating primarily on disassembling Rex Nebular - the game itself is split into a series of executables to save space:

mainmenu.exe - Which contains the logic for the main menu
animview.exe - Which contains the logic for displaying cutscenes, like the intro and ending animations
textview.exe - Which contains the logic for displaying scrolling text, as in the credits and quotes.
nebular.exe - The actual game itself.

So far, I've spent time disassembling the various executables, excluding the game executable, which I'll get onto in a moment, and along with work done by MD5 and others, we've got the main menus showing for Rex Nebular and Dragonsphere, as well as the scrolling credits and partial support for the cut-scene animations.

The next big step now was to start disassembling the main game executables to figure out all the in-game logic. This was when I'd hit a big stumbling block.. In order to properly understand a game, you have to be able to disassemble it in a dissassembler in it's entirety, so you can start identifying methods and variables, and gradually build up an understanding of how the game works. In the case of the actual game executable, though, all three games had been compiled with an overlay manager called RTLink/Plus, which was designed to save memory by swapping parts of the program code in and out of memory on the fly at runtime.

This therefore made the game difficult to disassemble, because:
1) The functions calls and jumps between code in different segments went via an intermediate table which was dynamically filled in at runtime
2) The data segment was in the middle of the program - this posed a problem because programs typically have the data segment at the end of the program, to allow for uninitialised data areas - space for variables which don't have an explicit value set at startup, so don't need to be part of the executable image.

As such, I was forced to write a program, which I've named rtlink_decode to handle these executables. I'm really quite proud of it - it takes in an executable, and presuming that it detects that is was compled with RTLink/Plus, creates a new executable with the following changes:
1) It detects all the dynamic segments and adds in relocation entries to the executable so that all segment references within these segments are treated like normal code segments
2) Processes the table used for inter-segment calls to fix them to go to the correct segment (what RTLink would have done dynamically at runtime wherever it loaded a segment)
3) Shifts the data segment to the end of the executable, and adjusts any data segment references in the executable to point to the new position.

The result is a new executable which was able to be fully disassembled by IDA Free - which I strongly recommend as a wonderful disassembler for anyone who's thinking about experimenting with reverse engineering.

Now that I've got a clean complete disassembly to work with, I can start figuring out the game logic. This time, though, I won't be starting from scratch - there is shared code between the game executables and the menu, credits, and animation executables, so I've already been able to identify a lot of runtime library methods, as well as other MADS methods I'd previously already figured out.

Nevertheless, it's likely still going to be a while before there's enough disassembled that any major functionality can be implemented into ScummVM, so you'll need to be patient.