Happy Birthday, Sir Arthur Conan Doyle

Hey everyone.

Looking back it seems, to my chagrin, that it's now been over a year since my last news posting was over a year ago. Whoops :P. Not that I haven't been keeping busy over the last year or so, with the release of Voyeur, Amazon - Guardians of Eden, Rex Nebular (finally), and of course, the newest game.. The Lost Files of Sherlock Holmes: Case of the Serrated Scalpel.

Many thanks go to EA for providing us access to the original source for this game. Also to forum user sirlemonhead, and to James Ferguson, who patiently over the last few years tried to make this happen. I've always been a big Sherlock Holmes fan, so it was fun to work on this project.  It feels fitting to merge the game into master on the 22nd May, which is the birthday of the character's creator, Sir Arthur Conan Doyle. The engine isn't quite ready for serious testing yet.. it's still missing music playback, and there's also a starting logo animation that's not present. It should be finished soon, though, so expect to see an official testing announcement in the near future.

So.. as things stand, what I am up to right now?

The Logo
Apart from the game proper, one of the more interesting things about the game is at the very beginning, where the publisher EA logo is shown. This logo display was actually implemented in a separate executable using the TsAGE engine, of all things. Original source for this couldn't be located, so it means that I'm having to reverse engineer it. Luckily, since we've already had experience with several other TsAGE titles, I've been able to make excellent progress in figuring out all the various TsAGE classes and their methods within the executable.

At the current point in time, I've identified the bulk of the core TsAGE classes, and the custom logic for the "game", which is contained a single scene class. This scene class consists of several scene objects, a few palette containers, and an "Action" class for coordinating what happens in the logo display. There's only a minor variation in how object sprites are loaded compared to the games that I still need to figure out.

I've already started implementing a new sub-module within TsAGE for the game logo. Once I finish that, it will be easier to analyse all the movement and frame changes of the images with the scene.  Hopefully, based on that, I'll be able to simulate a similar sequence in our new Sherlock engine using the bare necessities from TsAGE - likely just the RLB archive manager and sprite loader. Particularly given the thoughtfulness of EA in providing us access to the original source, it would be nice to give them (the company) proper attribution by showing their logo just like the original game does.

The Sequel
Apart from that, we have also been given access to source for the sequel, The Case of the Rose Tattoo. Implementing this is likely to be much more challenging, as the sequel changed over to a 640x480 display, and significantly altered the user interface. As such, it's likely it will need a lot of re-factoring of the code base to add support for it to the existing engine. If you thought a lot of re-factoring was done during the pull request, you 'aint seen nothing yet. :)

I'm also somewhat constrained by the fact that the original uses DOS4GW and a 32-bit code segment. Whilst we do have the original source, I need to be able to run the game in DosBox so I can actually see the code running, and check registers and memory contents at given points in the program. I've had some significant trouble with the DosBox debugger, trying to set breakpoints in the code so I can inspect the game's state.  Doing so crashes either crashes DosBox, or the game executable, or the breakpoints simply aren't hit.

So far, I've only done some preliminary loading of scene resources in the second game, and the lack of a way to display the program state meant that I had to take a more laborious route of poring over the various resource structures and scene loading code in both games, to try and figure out what the differences were between the two, so my code can support it. Likely, as I proceed with implementing more of the game, this will cause real issues that will make finding bugs a lot harder.

World of Xeen
It's been somewhat on a back-burner since I started work on the Sherlock Holmes games, but I had previously made real progress on re-implementing World of Xeen using the ScummVM framework. See below:

As you can see, I have much of the game interface implemented. You can move around, fight monsters (with a few minor glitches), and even leave the town. There are really only a few main areas left to implement, which includes sound support, logic for all the various spells, savegames, and the intro/ending cut-scenes. I probably won't return to working on it until after Rose Tattoo is finished, though. But when I do, I don't anticipate it will take long to finish the remaining areas, and then it would simply be a matter of playing the game through in earnest, identifying and fixing minor bugs as they're identified.

Of course, as an RPG, World of Xeen is a bit outside the scope of ScummVM proper. At that point, it may be time to finally launch the RPG sister project Strangerke and I have been wanting to do. :)

Rex-tacular

What a difference a mere week can make in terms of progress. At the beginning of last week only the basic background from one scene was showing, although there was more code implemented in the engine that wasn't yet working. Now, a week later, behold the progress of Rex Nebular on ScummVM:

That's right, in a rush of effort this week, the Rex Nebular engine has already passed several milestones in development:

• The user interface is now partly implemented. As you can see from the above picture, the user interface background is showing, with the sections for the actions, items, and the animated inventory item showing.
• The scene background is now displaying, along with basic sprite animations within the scene. The fires are burning, and the hamster "auxiliary power" :) is running.
• Basic player display. Rex is in his default position for the scene, facing the door.
• Behind the scenes, we've already completed the logic for a couple of the game scenes, although we're obviously not able to test them properly yet.

The focus for the upcoming week will now be on the beginnings of user interaction and action handling. This means getting walking working, selections in the user interface, and actually being able to execute actions. A lot of the needed code is already in place, but will require some heavy duty debugging and comparison against the original to figure out where things are going wrong. We're somewhat hampered by the complexity of the core engine when doing debugging, but we'll get there eventually. :)

Rexing it Up

Work has been progressing furiously with the MADS engine, and Rex Nebular support. Whilst I was able to us some code from the previously engine version, a lot of it turned out to be more M4 specific than I'd anticipated, and had to be freshly disassembled and/or rewritten. Additionally, with my greater experience writing engines, I've been able to lay out a cleaner separation of methods into classes than previously, particularly keeping in mind extensibility for having separate game logic for the other MADS games. As a result, the new MADS engine is already significantly different from the old M4 engine.

After a great deal of implementation, not to mention debugging of my code and comparison against the original running in DOSBox, I finally have the background of my test scene showing! See below for the first 'new engine' view of Rex Nebular in all it's glory:

Particularly auspicious considering I see that Rex Nebular has just been released for sale on GOG. So presuming that the engine gets finished this time, people will easily be able to obtain a copy. :)

What's next? In order to reach my original milestone of having an animation sequence playing correctly, it meant that I needed to properly implement the entire frame step and rendering logic from the game, which had a great deal of code, and many different secondary methods that I've encapsulated into a multitude of classes for sprite sets, pending sprites, active sequences, text display, and lots of other things. Not to mention a pretty complicated precursor to M4's RGBList, where resources loaded in for both the scene and for sprites are allocated chunks of the palette space.

I'm currently focusing on debugging the standard sprite drawing, which is used by the animation class. I'm hoping with a bit more work, I can it to properly show the animation sequence. Doing so will also help ensure that all the sprite display logic for the scene also works correctly. This will make it easier to start work later on for full blown game scenes, since all the necessary sprite display will already be done, and I can concentrate on things like player movement, action handling, etc.

Explosive decompression

Looking back, despite my earlier promises of posting more regularly, it seems I've been a bit tardy in posting updates, and quite a bit has happened. So lets get up to date with what's been happening with me. Firstly, we finally got some proper bug reports for Return to Ringworld, and have been working to fix them. Well, moreso Strangerke than me, so thanks go to him for his work in fixing bugs when he has the time.  As of right now, the card mini-game still isn't fully functional, but at least the main game is that much more polished now than it was.

Secondly, as of Monday, I finally merged the next game I was working on, Voyeur, into the project's master branch. This is a weird little game that Strangerke put me onto, because the original DOS executable had debug information embedded in it, which made it somewhat easier to implement. Still a lot of work to understand the contents of methods, reimplement them, and get the game in a stable state. But it made a nice change of pace from all the TsAGE work, where we had to disassemble everything from scratch. As of right now, the Voyeur engine only supports the DOS version. Apart from DOS, there were also CDi and Macintosh versions released. The Mac version at least may be supported at some point in the future, although there is some extra complexities we'd have to worry about due to the Mac-specific data in it, such as rasterized fonts that the PC version doesn't have. Adding support for it might make a nice future mini-project for a Mac enthusiast.

So what's next? Apart from further bugfixes for both Return to Ringworld and Voyeur, I'm already onto my next project.. once again tackling the white elephant of Nex Nebular, for the 3rd time. This time, though, things will hopefully be different. Using my experience with disassembling the TsAGE sound system, I tackled the sound system first, which was the major previous remaining stumbling block. And as of last Sunday, I finally got it working! That's right, my new embryonic MADS engine was able to play back the explosive decompression sound you get when you fail the copy protection check. It was a wonderful feeling, after spending a week reverse engineering and then implementing the code, then having spent an afternoon trying to debug my code and compare bytes being written to the ScummVM FM_OPL driver against the port writes in the original running under DosBox, to finally get everything right, and hear the sound coming out of my speakers.

 It turns out that despite some funky assembly tricks in the player that I had to work around, the player itself is actually a fairly compact, simple implementation. I'm no sound expert (as my frequent prior complaints make clear :) ), but I understood enough to give the various methods in my code halfway decent names, and name at least some of the fields. Hopefully a further analysis, now that I have cleaner C++ code, will help me better understand what's going on, and give the rest of the fields more proper names.

Now that the issue of sound support is taken care of, how am I going to proceed from here? I have a lot of code back from the earlier combined m4 engine which had code from both Rex Nebular and Orion Burger. Part of the trouble with the earlier attempts, I think, was that we were overcomplicating matters. As such, I'll only gradually put in code as I need it, and not worry so much about M4. That'll help me keep the focus more on getting Rex Nebular to work. Not that I'm going to just chuck out all M4 code from the engine. I've already started work on making the core classes, such as the previously named 'M4Surface' class, having a cleaner structure more suited for multiple games. For example, it'll now have it's own class factory, which will abstract having a base MSurface class that will have different descendants that implement the various game specific load/display logic. That seems cleaner than littering the code with "if isM4()" checks, and having methods like "rexLoadBackground". As far as a caller is concerned, they'll just initialise a surface, and the factory will take care of giving them the right kind for their game.

With a cleaner separation of game logic into descendent classes, I'll be able to, for example, keep in place special cases we'd already figured out for Riddle of Master Lu, and make it that much easier to implement support for it later. So that should keep a certain team member who shall remain nameless happy.  :)

At the moment, I'm currently aiming for getting all the initial engine setup sorted out, and playing back the animation sequence for when you get the copy protection answer wrong. As I mentioned above, this will involve pulling some code from the old project to save time, but I'm also already doing a fair amount of refactoring and cleanup, such as changing char *'s to Common::String, using Common::Point's, properly commenting the code, and so on. Given the amount of previous work done on the old M4 engine and the mostly complete disassembly I had of Rex Nebular, we can hopefully expect rapid progress in functionality, and eventually enjoy Rex's escapdes as he quests for the vase he's been sent to find.

Thanksgiving with a Ring.

"They said it would it would never be done. They were wrong." After a slight hiatus at the start of the year, work has finally finished on our next TsAGE game, Return to Ringworld. The only remaining work still in progress is a card game available on the in-game consoles, and is completely unrelated to the main story.

This is a direct sequel to the original Ringworld game.. after saving the puppeteer race from extermination and uncovering some powerful ancient technology on the first game, Quinn, Seeker of Vengeance and Miranda Rees find themselves searched for as fugitives by all three major species, so they plan to go to Ringworld to hide. But it turns that their ship has suffered problems. And fixing them and getting to Ringworld will only be the start of their problems..

The ScummVM Team is proud to announce that Return to Ringworld is now playable in ScummVM using the latest daily builds, and ready for testing. As usual, all bugs should be reported to our bug tracker following our bug submission guidelines. While you play through the game, we would also love it if you could take some screenshots for us.

Return to Ringworld code play-through complete

A great milestone has been reached with support of the Return to Ringworld game by the TsAGE engine. As of today, I've completed my first code play-through of the entire game! That is, my style of implementing games is to implement game scenes in the same order as the game is played. And although the savegame format has changed many times over the course of implementing the game, I've had a continuous savegame sequence for an entire play-through of the game.

The game is likely now completable, although given the number of changes that have been over time, it'll need further testing to ensure that the game has no newly introduced blocking bugs.

So what's the plan next? My initial play-through concentrated more on ensuring the actions needed to complete the game were working. I had a tendency to ignore miscellaneous actions that could be done in a scene, not to mention certain graphic glitches that occurred in some scenes, as well as some special effects and character shading that the game introduced that I hadn't implemented.

So now I plan to start working my way through the game again, doing further cleanup of the scenes code, and concentrating on ensuring each scene is graphically correct. And I may even put in some more options to skip some of the more annoying mini-games as well. :)

DreamMaster.

Hopkins FBI Linux completable

Another two weeks, and another milestone has been reached with the Hopkins FBI engine. The first play-through of the full Linux game has now been completed! All the blocking bugs have been resolved, and I've been able to finish the game.

That's not to say there aren't still some various graphical errors and things that still need to be tackled. Only just last night, for example, I fixed a particularly nasty series of graphical glitches with a lift scene that turned out to be a simple case of not clearing a flag when the animation manager class is created. So much grief trying to track down what looked like massive frame refresh issues or bugs in the animation player code. Thank goodness I was able to play the original inside gdb and easily set breakpoints to find out what values were set in the screen buffers, and when.

One of the biggest remaining graphical issues occurs near the end of the game. I'm not going to spoil anything, but you end up controlling multiple characters. There are currently issues with the icons for changing character not appearing all the time and, likewise, the chacters not always appearing on screen when your player character is in the same scene as them. I'm likely going to look into fixing them next.

After that it will be time to begin a more detailed play through to see if any noticeable differences can be seen between the original running in Linux against our game engine.. identifying and fixing any remaining minor issues as they occur. There may even be some further chance to fix issues in the original. For example, certain special scenes when you can access your inventory via the icon in the top left corner, the original doesn't let you use the 'I' key like you can in most other scenes in the game. That particular problem made me think there was something wrong with the ScummVM engine, before I realised you couldn't do it in the original, either. :)

Apart from that, we still need to start looking into supporting the PC version, since it's a lot more common than the Linux version. Hopefully we can get the sound decoder issues sorted out, and start testing that version in earnest as well.

Speaking of the PC version, I've mentioned previously that whilst I had the game, I'd never previously gotten around to playing it. And playing it now as the ScummVM engine was developed, how bizarre I found the game. What starts as what seems to be a straightforward cop-style investigation game rapidly becomes more and more bizarre. That was brought home when I was watching a play through of the PC version from Youtube, which I was using to check how the graphics are meant to look. It seems like the original PC version, in an underwater base you have to explore, they grafted on a clone of Wolfenstein 3D!

That's right. Whereas in the Linux version you are just shown a map and can select from various destinations, it looks like the PC version forces the player into a first person shooter and makes them battle their way from area to area. I've gotta say that whilst overall the game is fairly good, it's no longer just merely weird.. it's down-right freaky. :) Needless to say, there's every likelihood that we won't be implementing it. We'll likely just use the existing Linux version code, and provide a similiar map image as the Linux version.

And of course, there'll still be various cleanups to do. There's various memory leaks identified by Valgrind that we'll need to look into. And there's also general code cleanup, which both Strangerke and myself have made somewhat of a start on.

DreamMaster.