Rose Tattoo is in progress

Hi all,

Work is progressing well on Rose Tattoo, and as of tonight I hit a major milestone.. the entire game intro sequence is now completable. See the screenshots below:

There are still some minor graphic glitches at various points, and one of the scenes which is a double-side scene isn't properly scrolled horizontally yet, but even so, it's a great step forward in supporting the game. We're actually lucky in that Rose Tattoo implemented all of the introduction using standard game scenes. So it saved a lot of effort implementing manual introduction code like had to be done for Scalpel.

Now that the introduction is working, more or less, I'll be devoting more time to implementing the game-play. I'd already been spending some of my working on it, so some interaction is possible.. you can look at objects, open up the inventory, and conversations with characters are partially working. The game map is also already implemented, so you can get to other game locations as well.

On another subject, I had good luck implementing the original EA logo at the start of Serrated Scalpel. I was able to complete support for it in the TsAGE engine, and then used that as a basis for copying necessary code into the Sherlock engine. With some most welcome assistance from others, the EA logo at the start of the game now displays when you start the game, just like the original does.

Finally, on yet another tack, there have been some promising first steps towards supporting the 3DO version of Serrated Scalpel by m-kiewitz, with some assistance from clone2727.  The 3DO was a superior version of game, and included 16-bit color, video portraits, and full speech for every conversation in the game. Supporting this would be great. It would be nice if, one day, the 3DO version could be re-released with ScummVM, so a wider audience could properly enjoy the game.

Happy Birthday, Sir Arthur Conan Doyle

Hey everyone.

Looking back it seems, to my chagrin, that it's now been over a year since my last news posting was over a year ago. Whoops :P. Not that I haven't been keeping busy over the last year or so, with the release of Voyeur, Amazon - Guardians of Eden, Rex Nebular (finally), and of course, the newest game.. The Lost Files of Sherlock Holmes: Case of the Serrated Scalpel.

Many thanks go to EA for providing us access to the original source for this game. Also to forum user sirlemonhead, and to James Ferguson, who patiently over the last few years tried to make this happen. I've always been a big Sherlock Holmes fan, so it was fun to work on this project.  It feels fitting to merge the game into master on the 22nd May, which is the birthday of the character's creator, Sir Arthur Conan Doyle. The engine isn't quite ready for serious testing yet.. it's still missing music playback, and there's also a starting logo animation that's not present. It should be finished soon, though, so expect to see an official testing announcement in the near future.

So.. as things stand, what I am up to right now?

The Logo
Apart from the game proper, one of the more interesting things about the game is at the very beginning, where the publisher EA logo is shown. This logo display was actually implemented in a separate executable using the TsAGE engine, of all things. Original source for this couldn't be located, so it means that I'm having to reverse engineer it. Luckily, since we've already had experience with several other TsAGE titles, I've been able to make excellent progress in figuring out all the various TsAGE classes and their methods within the executable.

At the current point in time, I've identified the bulk of the core TsAGE classes, and the custom logic for the "game", which is contained a single scene class. This scene class consists of several scene objects, a few palette containers, and an "Action" class for coordinating what happens in the logo display. There's only a minor variation in how object sprites are loaded compared to the games that I still need to figure out.

I've already started implementing a new sub-module within TsAGE for the game logo. Once I finish that, it will be easier to analyse all the movement and frame changes of the images with the scene.  Hopefully, based on that, I'll be able to simulate a similar sequence in our new Sherlock engine using the bare necessities from TsAGE - likely just the RLB archive manager and sprite loader. Particularly given the thoughtfulness of EA in providing us access to the original source, it would be nice to give them (the company) proper attribution by showing their logo just like the original game does.

The Sequel
Apart from that, we have also been given access to source for the sequel, The Case of the Rose Tattoo. Implementing this is likely to be much more challenging, as the sequel changed over to a 640x480 display, and significantly altered the user interface. As such, it's likely it will need a lot of re-factoring of the code base to add support for it to the existing engine. If you thought a lot of re-factoring was done during the pull request, you 'aint seen nothing yet. :)

I'm also somewhat constrained by the fact that the original uses DOS4GW and a 32-bit code segment. Whilst we do have the original source, I need to be able to run the game in DosBox so I can actually see the code running, and check registers and memory contents at given points in the program. I've had some significant trouble with the DosBox debugger, trying to set breakpoints in the code so I can inspect the game's state.  Doing so crashes either crashes DosBox, or the game executable, or the breakpoints simply aren't hit.

So far, I've only done some preliminary loading of scene resources in the second game, and the lack of a way to display the program state meant that I had to take a more laborious route of poring over the various resource structures and scene loading code in both games, to try and figure out what the differences were between the two, so my code can support it. Likely, as I proceed with implementing more of the game, this will cause real issues that will make finding bugs a lot harder.

World of Xeen
It's been somewhat on a back-burner since I started work on the Sherlock Holmes games, but I had previously made real progress on re-implementing World of Xeen using the ScummVM framework. See below:

As you can see, I have much of the game interface implemented. You can move around, fight monsters (with a few minor glitches), and even leave the town. There are really only a few main areas left to implement, which includes sound support, logic for all the various spells, savegames, and the intro/ending cut-scenes. I probably won't return to working on it until after Rose Tattoo is finished, though. But when I do, I don't anticipate it will take long to finish the remaining areas, and then it would simply be a matter of playing the game through in earnest, identifying and fixing minor bugs as they're identified.

Of course, as an RPG, World of Xeen is a bit outside the scope of ScummVM proper. At that point, it may be time to finally launch the RPG sister project Strangerke and I have been wanting to do. :)

Thanksgiving with a Ring.

"They said it would it would never be done. They were wrong." After a slight hiatus at the start of the year, work has finally finished on our next TsAGE game, Return to Ringworld. The only remaining work still in progress is a card game available on the in-game consoles, and is completely unrelated to the main story.

This is a direct sequel to the original Ringworld game.. after saving the puppeteer race from extermination and uncovering some powerful ancient technology on the first game, Quinn, Seeker of Vengeance and Miranda Rees find themselves searched for as fugitives by all three major species, so they plan to go to Ringworld to hide. But it turns that their ship has suffered problems. And fixing them and getting to Ringworld will only be the start of their problems..

The ScummVM Team is proud to announce that Return to Ringworld is now playable in ScummVM using the latest daily builds, and ready for testing. As usual, all bugs should be reported to our bug tracker following our bug submission guidelines. While you play through the game, we would also love it if you could take some screenshots for us.

Just what am I up to

Well, after months of work, the secret Tony Tough project has finally hit the trunk. It's nice to see yet another game engine that I worked so hard on make it into the trunk. The only downside is that now that it's been announced, I feel obligated to finally make a blog posting. Whilst the project was on, I could procrastinate on making any postings, since most of my time went into working on it, and it was a secret. Now that's it's public, I've lost my good excuse to be lazy. Even so, I've still managed to procrastinate on making this blog posting for a few weeks. :) . Looking back, it looks like it's been nearly a year since my last post. Definitely past time.

So, I've finally got myself together to talk about what I've been up to, and what's happening in the near future.

Tony Tough:
My work on the Tony Tough engine has more or less finished now that we've got the game in a pretty solid state. In addition to other members who've made/making commits to polish up the last rough edges, kudos. Tony Tough actually took longer to convert than might otherwise have been the case because the game was originally multi-threaded. Each in-game action was done on a separate thread, which could be interrupted as necessary. So, for example, if you started an action to talk to a parrot, that action could be temporarily blocked if the parrot was currently already speaking, and then resume once it was finished. This caused issues for support in ScummVM, given the large numbers of systems we have to support.

As such, I utilised the existing coroutine framework that we set up for the Tinsel engine. Our coroutine framework is like a poor man's threading system. Using it, I was able to gradually convert the engine source to a single thread application. This did mean, though, that each and every method used within threads had to be converted to our coroutine macros. This took quite a bit of work to get done, in addition to expanding the function of our coroutine manager to also simulate the semaphores and other synchronisation mechanisms the game used. But now, particularly thanks to some recent polishing up by Fuzzie, we've got the running reliably running in a single thread.

Geekwad 2:
I've been spending some time working on another of the TsAGE game titles, Geekwd 2: Games of the Galaxy. Whilst it's more a set of mini-games that an adventure, it utilises the common TsAGE core. I've currently been focusing on the first minigame, Spaced Solicitors (aka Space Invaders) to get the feel of how the minigames have been set up, and am currently at the stage that the minigame is playable. It still needs some tweaks to control the speed, and for when the player either wins or loses the game. Once I've finished with that minigame, I anticipate the remaining minigames will be somewhat easier to implement, being abler to use that first minigame as a guide for how their minigame scenes work.

Return to Ringworld:
I admittantly hadn't done much work on the R2R sub-engine, since the bulk of my time previously was working on the Tony tough engine. I had spent a bit of time occassionally working on the dissassembly though. Our work on the game is likely to be a longer term project.. several of the remaining areas of the game are fairly complicated mazes and simulators, so the code associated with them will have to be painfully and gradually disassembled and re-implemented. Likely, most of my efforts on it in the near future will be invisible work on decoding the scenes in IDA, rather than making regular code commits.

Hopkins FBI:
Hopkins FBI is the new game I'm also currently experimenting with. Strangerke identified that the publically available Linux demo was actually compiled with some debug information embedded, which gave both method names and the names of global variables. Both the gdb debugger and IDA were able to read it directly, so IDA was able to produce an annotated disassembly. From this, the Hex-rays tool was able to provide a C disassembly of the entire demo. It's not as a good as having original source, since we're missing method parameter names, structures, and any formatting and comments the original had, but it's a great base to work from. The full game executable, whilist not having any debug information, is similiar in size, so we're hopeful that if we can properly support the demo, the full game would only require minor changes.

I've been slowly going through the program flow, converting over methods as I go. The disassembly isn't perfect, and sometimes gets things wrong. This is particularly the case for structures, since it doesn't realise that they're present.. I have to replace all the raw memory accesses to proper array indexes and structure offsets, gradually building up my understanding of the structures as I go. Overall, though, the decompiled code has proven a great aid in understanding the overall game, and particuarly dealing with all the smaller methods.

In the first week alone, I already had the first title screen showing! This is a pretty big deal, considering that I hadn't even looked at the game's internals until Strangerke brought it to my attention. Just showing the screen required conversion and fixing up a lot of the game's graphics framework to use the ScummVM framework, since that also included all of the game's initial setup code.. not just a quick hack to showing a particular screen. Following that, I started working on the animation manager, and again after about a week's worth of work, I had the basic animations of the introduction sequence playing, although it will still need a bit of work to exactly match the speed of the original, and have a proper display of frames within ScummVM. Following that, I put asside fine tuning the animation in favour of working on the main menu, so I'd have something interactive to show for my efforts, and now have the basic menu up and running.

Pretty great progress overall for just a few weeks worth of effort in my spare time. What's next? As fun as it is getting rapid results in what I've done so far, I think I'll spend some time powering through the initial conversion of the bulk of the remaining game methods, except for sound, which I still hate. :), and worry about testing the code afterwards. That way I can be more confident that any bugs aren't just that necessary code somewhere hasn't been implemented yet.

DreamMaster.

Two milestones reached

Yesterday I was chatting with Md5 about the recent progress in Blue Force, and I thought to myself that I should make a blog posting. Looking back, it seems it was June when I made the previously posting. I hadn't realised that it was that long.

As those of who frequent the IRC channel or forums would be aware, Ringworld was completed, and will be part of the upcoming official 1.4 release. We'd already had a successful testing period, with several bugs identified and fixed. It'll be nice to see another game I've worked on added to the trunk and become officially supported.

Now, onto more recent developments. As of yesterday, Strangerke and I have reached two more milestones in the progress of the TsAGE engine.

The first is that I completed the first complete play-through of Blue Force. This had been a gradual process over the last few weeks, as we identified game stopping bugs and fixed them. The save format also changed several times during this period, as we implemented the final few game scenes, and added missing game state variables. Yes, clone2727, the game is completable. So nyaaa. :)

The next step will be for us to start playing the game again from the beginning, comparing it against the original and looking for any minor graphics glitches or problems that we missed. We will also ensure that none of the fixes done for the later game scenes had any adverse effect on the earlier scenes. This includes a variety of minor issues that we identified during the first play through, but were deemed too minor to immediately worry about.

One of the more amusing glitches I came across was a scene in day 2 of Blue Force where a police inspection is done. This particular scene seems to use animation sprites different from anywhere else in both Blue Force and Ringworld. Likely some as yet un-handled flag or sprite offset - the result being that all the officers get drawn with their bodies on top of their head. Picture if you will headless officers walking around balancing on their own heads. :)


The second milestone is the commit of the beginnings of the ‘Return to Ringworld’ game - the third and final 2-D TsAGE adventure. I’d previously spent some time on the side starting to look at a disassembly of the game, and identifying all the common engine functions against my disassemblies of Ringworld and Blue Force, and got to the point where all the core functionality had been identified. I started on adding the game classes about a week ago, and have got to the point where I can at least display the first in-game scene.

I had been holding off on committing this to trunk until after the branch, since it affects a lot of the files, but now the 1.4 branch has been done. It also invalidates the current Blue Force savegame format (but not RIngworld) - since both of them share mostly the same UI code, it made sense to encapsulate it in a common base class. Given that Blue Force isn't official yet, it was easier to break the existing savegame format for it as part of this, then to try and maintain the savegames intact. So it was good that I’ve now completed the first play through of Blue Force, and can start from the beginning of the game again.


I have a tendency, for games I plan to work on disassembling, not to play them through using the original executables. That way I can have the fun of gradually playing through the game as I implement the code for it in ScummVM, and get the enjoyment from playing it for the first time in code that I’ve written. It also serves as an inducement to work hard, so I can see what happens in the game. I’m eager to see what happens to Quinn, Miranda, and Seeker in their second foray to the Ringworld.

Once we've cleaned up the Blue Force code-base a bit more, then work on Return to Ringworld will begin in earnest. Hopefully there won't be too many core engine changes, and work can proceed on it as rapidly as was done for Blue Force.

On a final note, for those of you wondering about Rex Nebular, I did start to return to looking at it, but I've been away from working on it in earnest for too long. There are still parts of the game not yet dis-assembled, and I'd need time to complete that and re-familiarise with the existing code. So it really needs some time when I'm not actively working on some other game. I'll have to see how I do when the TsAGE games are complete.

Let there be sound

The TsAGE engine for Ringworld now has sound playing! I'd be hard to pressed to express just how happy I was when I ran ScummVM and finally had correct sound playing back. I was like, Oh My God! it's finally working. After all the time spent re-implementing all the sound code from the original, and then spending nearly as much time slowly tracing through the execution of the original versus ScummVM to try and figuring out the cause of differences, it was a relief to finally have it working.

As such, the Ringworld game is that much closer to being complete. There are still a few areas left to look into, and polish up, such as:
* Sound fading isn't yet working correctly
* I need to hook the sound manager into the savegame format, and ensure playing sounds are persisted
* There are certain suspect areas in the sound manager's main _sfRethinkVoiceTypes method to review; at least one variable is being assigned to at one point and not currently being used, so I need to double check if that's really the case.
* The sound manager could use an overall clean-up.

It won't be too much longer until it's ready for public testing. Soon Larry Niven fans will be able to explore the Ringworld, and search for a way to defeat the bloodthirsty Kzins.

So what's coming next? Well, I've promised myself some time to work on Rex Nebular again, which I've somewhat neglected in my zest to have Ringworld completed. Strangerke is also enthusiastic about working on the other games that used the TsAGE engine, so I'll probably be spending some time on the game Blue Ice to determine what changes have been made to the engine, and provide a basis for being able to re-implement the game scene logic similar to what was done for Ringworld. Finally, I may offer to help out somewhat on the other games the team has received original source for. Gotta keep busy after all. :)

TsAGE sound support progressing

Work on the TsAGE sound has been progressing. I had originally hoped that the raw format used by the game would be a standard Adlib raw format that I could feed into one of the existing Adlib Sound FX classes. I tried to pass the data to the player class in the Cruise engine, which I had previously worked on, but unfortunately, whilst it didn't crash, it didn't play anything. And I'm not familiar enough with sound formats to have been able to diagnose why.

As such, I've spent some time powering through implementing all the relevant sound code directly from the demo executable. With the symbols identified, I at least had a heads up for the original engine names for a lot of the methods. Plus, we were able to garner some additional information from a demo version of the game Protostar. Whilst it's not an adventure game, the sound code has a lot in common with Ringworld; and not only did it have method names, it also had a lot of field information for the classes and structures. As with the Ringworld demo, it didn't have it for all the classes, but I was at least able to name fields in the sound classes and the sound manager.

From this basis I've now implemented what I think are all the relevant methods of the demo for both the Adlib sound driver and the sound manager classes - I'm currently using the OPL functionality with OPLWriteReg in place of direct port access. The resulting code I've completed doesn't actually play anything back yet, though. Making a log of the underlying writes to the Adlib port versus my ScummVM code, the results are different, so there's obviously some incorrectly implemented code.

My current major suspect is a method called _sfRethinkVoiceTypes - it's a enormously big method with over 500 lines, responsible for updating the voice channels based on queued sounds. It's one big mess, with a large pre-processing, post-processing, and main processing areas. I'll have to slowly trace through the execution both in ScummVM and the demo in the DOSBox Debugger to identify the problem. Hopefully once I've identified the differences, I'll finally hear some sound output.

Fingers crossed. :)

Most wonderful Ringworld demo

Hopefully everyone had a happy Easter weekend; or for those of other religions, were able to enjoy a long weekend wherever you are. :)

It's been a little over a week since the TsAGE engine hit the trunk, and a big thanks go out to everyone who's helped out with commits - fixing compilation problems, and cleaning up the code. Special thanks again go to Strangerke, who has been busy working his way through the Ringworld game to fix bugs and make the game completable. I too have spent most of my time fixing some of the outstanding bugs, as well as newly identified bugs in the core engine.

On another topic, we've had a stroke of exceptional luck. I had been starting to get bogged down in all the various classes related to sound handling, and I wasn't looking forward to trying to figure out what it all did. But then Strangerke discovered that one of the two available Ringworld demos actually had embedded symbol information for most of the variables and methods of the demo!

Since the full game engine isn't anticipated to be much different from that used in the demo, it likewise means we've suddenly got a huge jump forward in understanding the remaining sound-related parts of the engine. Fuzzie was able to extract the information into a text file listing, and I wrote a quick and dirty script in IDA's IDC language to load the symbol information into a new IDB file for the demo executable.

This isn't quite as good as getting the full original source for the game, but it's still a major find. The biggest limitation of the symbol information is that such symbol tables don't include information for fields within structures. So, for example, the 'ASound' class has 282h bytes of information, but the instance of the class only has the symbol for the first byte, not all of the bytes. Therefore, I'll still need to spend a reasonable amount of time figuring out the contents of the various methods. But at least with the method names known, I can more easily figure out what each method actually does.

In the space of under two days, I've already implemented support for the two Ringworld demos. Luckily, the demos use a single script sequence rather than direct code, which made it very easy to add support for. All that still needs to be implemented for the demo are a few custom dialogs, and ensuring none of the full game functionality, like Saving and Restoring, is enabled for it.

Implementing support for the demo also identified a key difference between the two versions of the demo - the second demo was similar to the CD version, in that they changed the format of how the data for priority regions and walkable areas was stored. Having a disassembly of the demo with symbols made it easier to identify the relevant areas of code, and figure out the differences.

In addition to polishing up support for the demo, I'll likely making a branch to work on the sound classes separately. We currently have some hard-coded logic in the existing engine to properly signal actions that any played sound has immediately ended, so I don't want to interface with that whilst working on a proper implementation of the sound system.

So things are definitely rapidly moving forward. :)

TsAGE has now hit the trunk

Hi all,

I'd been meaning to write a new blog post for a while, but progress was progressing so quickly, that I never quite got around to it. Now, though, work has progressed far enough that the tSage engine has now been merged into trunk. Huzzah! :)

Many thanks go to Strangerke, who joined me in implementing all the various game scenes of the Ringworld game. Together, we've implemented all the game logic.

The following is the state of the game:
* All the game scenes for the floppy version of the game are implemented, although some still need some bug-fixing, so the game isn't completable.
* Sound isn't supported yet, and still needs to be reverse engineered in the original game.

So we have the following goals for the near future:
* We'll be working on fixing the remaining scene bugs, and ensuring the game runs smoothly from start to finish.
* Implementing the sound code.
* I also want to implement support for the original game's cheating/debug functionality.
* Add support for the CD version of the game, which has some minor differences.
* And add support for the demo version, if possible.

All in all, it's a celebration for another engine being added to the ScummVM universe. :)

tSage engine proceeding quickly

Hi all,

I've really been making some great strides forward with my implementation of the tSage engine. I offer the two screenshots below. One is from the first game scene when run in DOSBox, the other under ScummVM:



Hard to tell them apart, isn't it? :). The one on the left is from DOSBox, and the one on the right is ScummVM. All the initial hotspots in the first game room are now set up, are being drawn on the screen, and have correct look descriptions. The only exception is a minor graphic glitch with the doorway - the current drawing code doesn't yet support clipping objects based on priority calculations, so the top of the doorway isn't correctly clipped off.

One of the next major steps from here is to delve further into the movement classes and code. Whilst the player is now drawn on-screen, I don't yet have any code responsible for moving the player, such as when you manually walk somewhere, or when the computer moves the player to a set location, such as when you use the door beam.

From what little I've looked into the movement code so far, the game maintains a set of classes responsible for moving objects within the scene. For the player specifically, there's a lot of code that seems to make use of special 'walk regions', which indicate where on the screen the user can walk. The movement code is tied into this, and uses them to determine both where the player can move to, and for pathfinding to move the player to the given destination.

Given the pace I've been having recently in other areas of the engine, I don't anticipate that it will take too long. Hopefully, soon I'll be able to move the player about on the screen. Then I can start delving into implementing all the logic involved in the first cut-scene, where the Kzin comes to the door and speaks with the player. It's so much more fun working on an engine when you can almost day-by-day make visible progress in it's implementation.

PS: I've been making such rapid progress with tSage, I've been somewhat neglecting work on the MADS engine. I hope any fans of Rex Nebular will be patient with me.. I will get back to work on it eventually. :)

Roadblocks in the path of tSage

Those of you who have been following the IRC channels will know that I made some real progress in the disassembly of the tSage engine, used by the Ringworld - Revenge of the Patriarch game, as well as several others. I hadn't really gotten into any of the game logic, but I'd made great strides in decoding the image formats and graphics display, so much so that my engine on Scummvm-misc could display test dialogs, as well as the right-click game dialog with proper highlighting and everything. Sample screenshots below:





However, given the perversity of life, I should have known better than to announce on the channel that I was making real progress. :(. Because as I was getting into the game execution logic, I began to realise just how complicated they made the engine. Whilst using C++ has it's advantages from the perspective of RE, since it does group related functions close together with the variables they use, in this case, they seemed to go overboard with it.

A preliminary look at the main event loop of the game shows they implemented a stack of "action objects" - multiple independent event handlers can be registered and control swapped form one to the other. I'm not sure, but I presume that this is to allow for things like cutscenes, where standard game operation is superseded.

For even more complexity, they went with a class hierarchy for the action objects which is at least four levels deep!.
The classes, as far as I've been able to determine, are as follows:

• They have a core 'SavedObject' class, which has some internal logic for registering each object instance in a central list. I presume this is for automating saving and loading.
• A child class which I'll call "SmartPointer". This class seems to encapsulate a pointer to a secondary object, and provides core logic for 'fixing' up the pointer reference. I can't be sure yet, but there's a lot of 'fixup' code in the engine, which I presume handles restoring pointer references between objects when a savegame is loaded, and all the objects are re-constructed. This class also implement a basic virtual table of processing and dispatch methods that automatically defer to the foreign object if one is set
• Yet another child class which introduces basic event manager functionality amongst other things
• A concrete fourth child class which seems to further override some of the event manager functionality, and is what gets added to the game's "action list".
  

As you can imagine, all this makes it somewhat different to understand what the blazes is going on. I've also examined the engine from some other points, such as scene loading, and even that seems to be overly complex, with method calling method - they really went in the direction of generalised/generic functionality, with the result that the code seems to be much more complex than it should otherwise need to be.

</whinge>

In any case, the tSage engine was only meant to be a part-time RE effort for when I needed a break from the MADS engine. I did somewhat get carried away with it for the last few weeks, but that was only because the areas I was looking at proved so easy to reverse engineer. Now that I'm running into more complex code, I've decided that the break's over, and I'm returning to work on the MADS engine again. For that, at least, the code seems somewhat simpler, even if it can be a pain to run in the DOSBOX debugger because the segments keep getting shifted around in memory. ;)

For those of you who are Ringworld fans, don't worry, I'll still be working on the engine. It's just there won't be as rapid a progress on it as I'd recently hoped.