Hopkins FBI Linux completable

Another two weeks, and another milestone has been reached with the Hopkins FBI engine. The first play-through of the full Linux game has now been completed! All the blocking bugs have been resolved, and I've been able to finish the game.

That's not to say there aren't still some various graphical errors and things that still need to be tackled. Only just last night, for example, I fixed a particularly nasty series of graphical glitches with a lift scene that turned out to be a simple case of not clearing a flag when the animation manager class is created. So much grief trying to track down what looked like massive frame refresh issues or bugs in the animation player code. Thank goodness I was able to play the original inside gdb and easily set breakpoints to find out what values were set in the screen buffers, and when.

One of the biggest remaining graphical issues occurs near the end of the game. I'm not going to spoil anything, but you end up controlling multiple characters. There are currently issues with the icons for changing character not appearing all the time and, likewise, the chacters not always appearing on screen when your player character is in the same scene as them. I'm likely going to look into fixing them next.

After that it will be time to begin a more detailed play through to see if any noticeable differences can be seen between the original running in Linux against our game engine.. identifying and fixing any remaining minor issues as they occur. There may even be some further chance to fix issues in the original. For example, certain special scenes when you can access your inventory via the icon in the top left corner, the original doesn't let you use the 'I' key like you can in most other scenes in the game. That particular problem made me think there was something wrong with the ScummVM engine, before I realised you couldn't do it in the original, either. :)

Apart from that, we still need to start looking into supporting the PC version, since it's a lot more common than the Linux version. Hopefully we can get the sound decoder issues sorted out, and start testing that version in earnest as well.

Speaking of the PC version, I've mentioned previously that whilst I had the game, I'd never previously gotten around to playing it. And playing it now as the ScummVM engine was developed, how bizarre I found the game. What starts as what seems to be a straightforward cop-style investigation game rapidly becomes more and more bizarre. That was brought home when I was watching a play through of the PC version from Youtube, which I was using to check how the graphics are meant to look. It seems like the original PC version, in an underwater base you have to explore, they grafted on a clone of Wolfenstein 3D!

That's right. Whereas in the Linux version you are just shown a map and can select from various destinations, it looks like the PC version forces the player into a first person shooter and makes them battle their way from area to area. I've gotta say that whilst overall the game is fairly good, it's no longer just merely weird.. it's down-right freaky. :) Needless to say, there's every likelihood that we won't be implementing it. We'll likely just use the existing Linux version code, and provide a similiar map image as the Linux version.

And of course, there'll still be various cleanups to do. There's various memory leaks identified by Valgrind that we'll need to look into. And there's also general code cleanup, which both Strangerke and myself have made somewhat of a start on.

DreamMaster.

Hopkins FBI is WAVing

Another two weeks have gone by, and we've seen further development in the Hopkins FBI engine. The first is that the entire sound system is now implemented. In fact, the bulk of it was implemented in only a single day. Despite my concerns to the contrary, the implementation of sound in the game was extremely simple.. everything is implemented as WAV files, even the music.

Which made it  very easy to code for, since I could use the existing ScummVM WAV audio stream class to implement the playback. Most of the complexity, in fact, came to setting up the necessary sound arrays and queing/status code that the game uses, rather than raw sound code itself . In fact, the game uses an interesting compression tactic for music playback.. it splits up a given music track into multiple different WAV files, and has an index of the order to play the set of fragments in, allowing it to repeat the same WAV file multiple times during the song playback.

So now sound is working, and you can hear all the music and in-game sounds just like in the original. :). The only minor downside is that it's revealed that the different versions of the demo and full game (Windows demo, and both Linux and PC full versions) use different sets of music files, at least for the initial introduction sequence and menu. So we're going to have to have different versions of the different start-up code.
 
This is what Strangerke is currently concentrating on. He's been able to obtain the Linux full game, and has just finished implementing all the remaining methods that it uses that the demo didn't have. He's now going to start working his way through the game to see if everything works. We'll likewise have to determine any changes for the Windows version in the near future as well.

As for me, I'll be doing some further playing around with the demo to identify any minor remaining issues and trying to fix them as I find them. And of course assisting Strangerke as necessary. Hopefully we'll soon have a completable full game with full sound support. ;)


DreamMaster.

Hopkins FBI Linux Demo is now completable

Well, another milestone has been reached in the development of the Hopkins engine. The Linux demo, upon which I've been basing the engine implementation, is now officially completable! Yes, you can now guide the intrepid FBI agent through the start of his case, and deal with a bank robbery. I think it's a considerable happy achievement, given how recently I started work on it, and how little time I've had available to spend on it, given work commitments,

So what comes next?

Well, there are still some graphic glitches to sort out. For example, implementing proper savegame thumbnails, and the cursor drawing of actively selected inventory objects isn't clearing itself correctly when the cursor moves. I'll be concentrating next on identifying the cause of these problems and resolving them, which hopefully shouldn't take long.

There is also a need for refactoring. Now that the engine has stabilised into a usable state, I need to start reviewing code and giving proper names to all the structures that currently have names like 'field2' and 'field4'. Likewise, method parameter values and locals will also need better names. I also need to review the current separation of code I've done into various manager classes and move methods more appropriate to different managers into their appropriate manager.

Those two are the immediate short-term goals. Once they're done, there are still the two big ones remaining:

Firstly, supporting the actual full game. Whilst the game does have some basic scripting mechanisms, the bulk of the code seems to be in a massive method called 'Traduction', at least in the demo. This method consists essentially of a massive switch statement that handles all the game logic. It'll be interesting to see if this is done the same way in the full game as well. Strangerke is currently looking for someone with a copy of the Linux version of the game, that we can use as a basis of comparing against the demo version to get the changes. Hopefully, the core engine will remain pretty much identical, and we can just drop in a full game 'Traduction' method to properly support the full game.

And secondly, of course, there's the old hairy chestnut of sound support. I'm hopeful that the music, voice, and sound effects are in a standard format and that, similiar to what was done in Tony (and other) engines, I can use existing ScummVM functionality for sound playback. I really don't relish an extended period trying to re-implement sound drivers for Hopkins, like I had to do for tSage.

So, all in all, we should hopefully soon have Hopkins in a fit state as yet another game playable under ScummVM. :)

Making progress in Hopkins

Well, the last few weeks have been very busy for me at work, and it looks like things will be getting even busier as we get towards the end of the year. Despite this, I've made rapid progress in converting over the disassembled source code. I've now reached a point where I've done an initial conversion/implementation of all the method relied on by the demo, with the exception of the sound routines, which I tend to leave to last, since large chunks can frequently be replaced by existing ScummVM playback code.

There is, however, a bunch of code left in the executable that isn't directly called. This is particularly noticeable due to the way I did the conversion - starting with the decompiled code, I converted the single main method, removing it from the decompiled source file,, then added it into the ScummVM project and created stubs for any sub-method that that method called. I then proceeded to gradually convert those stubbed methods, creating new stubs as necessary. That way, for each method I converted, I could ensure that the code correctly compiled before moving onto the next.

The result of this is that at the end of this process, I have a decompiled source file that still has a lot of code remaining in it, even ignoring all the sound methods that I haven't implemented. I haven't done any analysis of it yet, but I'm presuming that some of the code is related to the full game executable, and whatever compiler they used didn't actually remove it when the executable was required.

Since I've spent the last week cramming the conversion, I figure to leave the unused code alone for now, and return to working on running the code and fixing bugs in the result. I'll also start doing renaming of the data structures as I understand things better. I'm hoping that despite my haste in converting the methods over, I'll be be able to rapidly see some in-game results.

I'll try to make more frequent progress reports to make up for the previous absence of postings. :)

Just what am I up to

Well, after months of work, the secret Tony Tough project has finally hit the trunk. It's nice to see yet another game engine that I worked so hard on make it into the trunk. The only downside is that now that it's been announced, I feel obligated to finally make a blog posting. Whilst the project was on, I could procrastinate on making any postings, since most of my time went into working on it, and it was a secret. Now that's it's public, I've lost my good excuse to be lazy. Even so, I've still managed to procrastinate on making this blog posting for a few weeks. :) . Looking back, it looks like it's been nearly a year since my last post. Definitely past time.

So, I've finally got myself together to talk about what I've been up to, and what's happening in the near future.

Tony Tough:
My work on the Tony Tough engine has more or less finished now that we've got the game in a pretty solid state. In addition to other members who've made/making commits to polish up the last rough edges, kudos. Tony Tough actually took longer to convert than might otherwise have been the case because the game was originally multi-threaded. Each in-game action was done on a separate thread, which could be interrupted as necessary. So, for example, if you started an action to talk to a parrot, that action could be temporarily blocked if the parrot was currently already speaking, and then resume once it was finished. This caused issues for support in ScummVM, given the large numbers of systems we have to support.

As such, I utilised the existing coroutine framework that we set up for the Tinsel engine. Our coroutine framework is like a poor man's threading system. Using it, I was able to gradually convert the engine source to a single thread application. This did mean, though, that each and every method used within threads had to be converted to our coroutine macros. This took quite a bit of work to get done, in addition to expanding the function of our coroutine manager to also simulate the semaphores and other synchronisation mechanisms the game used. But now, particularly thanks to some recent polishing up by Fuzzie, we've got the running reliably running in a single thread.

Geekwad 2:
I've been spending some time working on another of the TsAGE game titles, Geekwd 2: Games of the Galaxy. Whilst it's more a set of mini-games that an adventure, it utilises the common TsAGE core. I've currently been focusing on the first minigame, Spaced Solicitors (aka Space Invaders) to get the feel of how the minigames have been set up, and am currently at the stage that the minigame is playable. It still needs some tweaks to control the speed, and for when the player either wins or loses the game. Once I've finished with that minigame, I anticipate the remaining minigames will be somewhat easier to implement, being abler to use that first minigame as a guide for how their minigame scenes work.

Return to Ringworld:
I admittantly hadn't done much work on the R2R sub-engine, since the bulk of my time previously was working on the Tony tough engine. I had spent a bit of time occassionally working on the dissassembly though. Our work on the game is likely to be a longer term project.. several of the remaining areas of the game are fairly complicated mazes and simulators, so the code associated with them will have to be painfully and gradually disassembled and re-implemented. Likely, most of my efforts on it in the near future will be invisible work on decoding the scenes in IDA, rather than making regular code commits.

Hopkins FBI:
Hopkins FBI is the new game I'm also currently experimenting with. Strangerke identified that the publically available Linux demo was actually compiled with some debug information embedded, which gave both method names and the names of global variables. Both the gdb debugger and IDA were able to read it directly, so IDA was able to produce an annotated disassembly. From this, the Hex-rays tool was able to provide a C disassembly of the entire demo. It's not as a good as having original source, since we're missing method parameter names, structures, and any formatting and comments the original had, but it's a great base to work from. The full game executable, whilist not having any debug information, is similiar in size, so we're hopeful that if we can properly support the demo, the full game would only require minor changes.

I've been slowly going through the program flow, converting over methods as I go. The disassembly isn't perfect, and sometimes gets things wrong. This is particularly the case for structures, since it doesn't realise that they're present.. I have to replace all the raw memory accesses to proper array indexes and structure offsets, gradually building up my understanding of the structures as I go. Overall, though, the decompiled code has proven a great aid in understanding the overall game, and particuarly dealing with all the smaller methods.

In the first week alone, I already had the first title screen showing! This is a pretty big deal, considering that I hadn't even looked at the game's internals until Strangerke brought it to my attention. Just showing the screen required conversion and fixing up a lot of the game's graphics framework to use the ScummVM framework, since that also included all of the game's initial setup code.. not just a quick hack to showing a particular screen. Following that, I started working on the animation manager, and again after about a week's worth of work, I had the basic animations of the introduction sequence playing, although it will still need a bit of work to exactly match the speed of the original, and have a proper display of frames within ScummVM. Following that, I put asside fine tuning the animation in favour of working on the main menu, so I'd have something interactive to show for my efforts, and now have the basic menu up and running.

Pretty great progress overall for just a few weeks worth of effort in my spare time. What's next? As fun as it is getting rapid results in what I've done so far, I think I'll spend some time powering through the initial conversion of the bulk of the remaining game methods, except for sound, which I still hate. :), and worry about testing the code afterwards. That way I can be more confident that any bugs aren't just that necessary code somewhere hasn't been implemented yet.

DreamMaster.