It's ScummVM ULTIMAte

A little over a week ago, it finally happened. I had combined my various work on the Ultima games into a single branch, and it was merged into master. This means Ultima VI: The False Prophet (from Nuvie), and Ultima VIII: Pagan (from Pentagram) are now playable in daily builds.  \o/. Ultima 1, as before, is startable, but still a work in progress, and work on that is on hiatus right now. The Ultima8 sub-engine already has a splash-screen, provided by Dominus Dragon, added to give credit to the Pentagram team that wrote the original code:

Nuvie will similarly get it's own splash image eventually to recognise it as well.

However, I'll hasten to say, that any Ultima fans should not necessary come running to play either game just yet. I'm still not quite ready to announce an official testing period for either game just yet. I plan to be doing my own testing of the games first, and it's always possible that there might be minor issues or changes to the savegame formats. So any experimenters be warned to play at your own risk.

I'd originally intended to move straight onto testing of Ultima VIII post-merge, but I got slightly sidetracked adding new functionality to the base Engine class that all game engines derive from. Functionality that future game engines will benefit from, including the Ultima engine.

1) The first of these is extended savegames. There was a recently added enhancement to savegames to take care of the drudgework of storing the savegames' names, thumbnails, total play time, etc. However, it still required engine writers to manually call various methods manually in their savegame code. With the addition of new methods in the Engine class that builds on this, this is all now taken care of automatically now, as well as opening and closing savegame files in general. All that needs to be done in future engines is overriding of two methods: loadGameStream and saveGameStream, which get passed a handle to a savefile for reading/writing it's contents, and the Engine takes care of everything else, including the savegame description and thumbnail.

2) The second is improved handling for autosaves, still under code review before inclusion. It adds autosave support to almost all the game engines. ScummVM has a global option for an Autosave interval, but it previously required each engine to implement their own code for autosaves. Because of this, only about 1/5th of the engines actually implemented autosave. When these changes are integrated autosaves, when turned on, are provided by practically all game engines. Only a few engines, like DreamWeb, that don't allow saving the from Global Main Menu won't support autosaves.

3) The third one, also under review, is an improvement to engine debuggers, the console window that pops up in many engines when you do Ctrl-D. Engines now just have to create their own debugger and pass it to the Engine to take care of, and it handles all the work for opening it when necessary, and updating it. One of the benefits of this, apart from standardization & centralization, is that when errors occur now, the debugger window is more guaranteed to open up to show it. Even if the engine doesn't have it's own debugger, the base Engine class will create one on the fly. This means it will be easier for users to see fatal errors, rather than ScummVM simply terminating.

With these three done, I think my "base Engine binge" is fully tapped out. Barring any further issues or suggestions coming up in the pull requests for points 2 and 3, I'll probably take a few days to relax. Maybe do some further messing around with cleaning up Starship Titanic code, before I move back to my original plan of doing a test play-through of Ultima VIII.
.

It has to be Xeen to be believed

Hi everyone,

Been a while since my last blog posting, despite regular protestations that I'd be posting more frequently. My bad. At least my development work has been proceeding with it's usual speed in the interim. The just released ScummVM 2.0 spurred me to write a blog posting.

So what's been happening? First of all, as most will know, I finished work on Starship Titanic. So with the new release, both the English and German versions of the game are now officially supported. I've been informed that there's also a French version, but it isn't available commercially (GOG only has the English and German versions). Given that, for now I'm unwilling to spend the multiple weeks worth of effort to add support for it. Maybe if GOG adds it in sometime in the future, support for it can be revisited.

Next, let's move onto what I'm currently working on, now that Titanic is done.. World of Xeen. Support for the game has already improved enormously over the last few weeks. You can walk around the starting town, fight monsters, then leave the town and wander around the outdoors. Even go to the Dwarf Mine and listen to the dwarf's cheesy intro. Not all maps are working yet, but at least it's getting there.

Still lots of other UI bugs to be looked into. I spent much of the past week cramming to implement all the various in-game cutscenes without testing them yet, so there's bound to be problems. The intro/endgame cutscenes for the games are also still only partly implemented. I'll also likely need to do further work on the savegame code, so the engine definitely isn't ready for any serious playing yet.

Anyway, with Christmas holidays coming up, I'll likely take a brief break for a few weeks from my resumed work on Xeen. Much of the work for reversing the game had already been previously done by the excellent work of WizardStan, and then further by myself. As such, a lot of the time I've spent simply implementing remaining missing code in ScummVM and lots of debugging. Since I've already spent so much time in debugging Starship Titanic code to get it working, I'm looking for a more relaxed, cerebral exercise of reverse engineering something fresh over the holidays.

I'll probably return to my preliminary work on the Legend entertainment games, Companions of Xanth and Gateway. Both share a lot of common engine though Xanth, the more recent, has a horrible overlay manager that makes it a pain to debug. As such, I've found working on the earlier game makes things easier to understand in many places. Though since the game uses EGA graphics, understanding how things are being written to the screen does get problematic in places. So a balanced effort reversing both at the same time is called for. An excellent mental challenge.

Everyone have a Happy Holidays/Year end/Christmas or whatever, as 2017 comes to a close. :)

The player is finally home.. the long way round

It's taken so many nights spent slowly debugging the original executable versus my ScummVM implementation, but the final starfield puzzle of Starship Titanic is finally working.

I'm not 100% happy with how the starfield rotates to selected markers when you've locked them in, but frankly, given that I've spent multiple months on disassembling, implementing, and fixing just this one puzzle, I'm just happy at this point that it works at all.

So now with the starfield puzzle finally completable, I was able to initiate the endgame, and see the ending video and credits:

I have to say, there were times when I grew weary of implementing and testing all the matrix code that the puzzle required. But I guess I'm just too stubborn not to see it all the way through.

So what's happening next? Firstly, there are various minor bugs that I was aware of, but hadn't previously gotten around to fixing. I'm currently working into fixing them now For example, I fixed some jerking of text in the end credits, and some black boxes that briefly appeared over the flames in the canal. I've got some outstanding issues with NPC idle animations to look into. The Bellbot also won't currently bugger off if you tell him goodbye :) Once that's done, I'll give it another playthrough just to make sure before it's announced for public testing. So expect it to be soon. For those of you that don't have the game already, the current GOG sale has it discounted. So it's a very opportune time to pick it up.

On a final note, there are couple of things associated with the game that I don't have any immediate plans to spend time on:
* The QSound library the game uses for simulating sounds in a 3D space using standard stereo output. Many know my distaste of working sound code. So I'll leave it as a future exercise for someone else to work on. I've implemented the low level sound calls using mostly the same interface as QSound exposes, so it should prove convenient for anyone who chooses to do so
* The Indeo 4 decoder still doesn't handle cases where transparency information is embedded directly into the video frames, rather than as a separate video track. Since codecs are installed directly into Windows, I'm not even sure which DLL implements the decoder. I'll try and spend a bit of time trying to figure it out, but worst case, it may be something the game just has to live with. The game currently has "best guess" code that estimates what the transparencies should be. It's not perfect, but it's reasonably servicable for now.
* I don't have any near-term plans to do any further work on the German version. I'm simply too burned out over the game, and want to move on from it. I may return to it one day; I'd also welcome anyone else who wants to look into it themselves.

DreamMaster.

There's been a Titanic amount of progress

Since my prior posting earlier in the year, there's been a great deal of progress in Starship Titanic. I decided to put aside the problem of reverse engineering all the Star Map classes until I had the rest of the game working better. In that respect, I've made great strides since, as of last weekend, I was able to complete the entire "prologue" of the game  That included using the computer, experience the crash, talking to the Doorbot, entering the ship, and viewing the Credits. Huzzah. \o/

I was going to prepare a video showing the intro, but with the most recent changes, there seems to be some instability showing up. It seems like something that was already present, just coincidental that the newer changes result in more frequent crashes. It's kind of hard to narrow down the cause, as there's also a problem with the implementation of the Indeo video decoder we're using for NPC videos like for the Doorbot, where it's reading past the end of the frame data. So it's difficult to track down the memory corruption, as warnings about the decoder are completely overwhelming everything else.

So for now, I'll present a screenshot of the amazing multi-color Doorbot :)



After thinking over matters, I've decided to keep progressing into the game, and come back to look at the problem later on. Part of the trouble I'd been having with the code was the sheer length of the intro as I got further and further into it.. requiring me to wait through several minutes of cutscene & conversation every time I made any changes or bugfixes. Even if the intro has suddenly become unstable, I still have savegames I made from beyond it, so I'm using them as a starting point to make further progress testing into the game.

Speaking of testing, I've had a major boon to my efforts to track down bugs in the code. I was previously stymied trying to test the original Windows executable in the IDA debugger, since it kept crashing on me. Plus running in compatibility mode full-screen didn't help either. And without the ability to see "valid" values in the original executable, I anticipated it would be difficult to track down errors in my code, since I wouldn't know whether values/state at any point in time were already wrong on not.

Luckily, though, I stumbled on a solution. Using the Visual Studio "Attach to Process" allowed me to attach to the game executable without it crashing, unlike IDA. At least, for the majority of the time. Though switching from the game to the debugger and back again caused severe corruption of the full-screen display. Luckily, though, there had been some previous discussions about running the game in a window - I was able to use a utility called DXWnd that intercepted the game's DirectX calls and forced it to run a window. The result wasn't perfect, in my opinion, for anyone wanting to play through the game, but it's worked well enough for my purposes, in conjunction with Visual Studio.

As a result, I'm now making much better progress than I had anticipated, and hunting down bugs is in general much easier than I'd anticipated. Let's hope that stays the case.. my next major gameplay milestone is to complete more extensive conversation with the Deskbot to get myself a room. The basic yes/no detection for the Doorbot worked pretty smoothly first time I tried it. The Deskbot, though, is using more of the conversation parser - I've already located and fixed some problems with it. Let's hope that there won't end up being too many.

On a final note, the one downside of my surging progress with Titanic is that I'm currently spending less time working on finishing my Xeen engine. I'd originally anticipated the frequent roadblocks trying to hunt down bugs in Titanic would have me growling in frustration, and switching to Xeen for awhile to unwind a bit. Now with the ability to debug the original executable, that hasn't really happened so far, and hopefully won't happen. I'll probably end up spending more time right now focused solely on Titanic, and see if I can't get the bulk of the game with the exception of the final starmap working by the end of the year. Then I'll be in a better position to alternate between working on Xeen and trying to disassemble the remainder of the Starmap classes.

DreamMaster.

PET is PrETty much done

Hi all. Been a while since my last posting; and a hectic last couple of weeks. After flying down to Florida for a 4th July long weekend getaway, I was barely back before I had to get ready for a work trip to the west coast. Then I get back to two days of company photo-shoots, which kind of disrupted getting back to normal work; but no sooner than they were done, I come down with a cold and had to stay home for several days. And was too sick to even really use my computer until now. :(

Nevertheless, despite all of that, work has been progressing on Starship Titanic since my last posting, and I've made a lot of progress. Firstly, there's the PET User Interface. This was one of the major areas that still had to be investigated last time. Since then, I've spent a lot of time figuring out how everything works, and re-implementing it all in my ScummVM engine. Behold the fruits of my labor:




Code for all the various areas has been pretty much all implemented. You can enter text in the conversation tab, and click stuff in all the others. There's just a few minor visual and functionality glitches here and there, such as the dials in the above image. Some of the tabs, like the Remote and Rooms tabs, are mostly untested at the moment.. it will be easier to test when I have more of the game logic implemented, and can properly test them with live data. Saving and Loading is also waiting on me to finish the saving and loading code.

As before, I was aided somewhat in implementing the PET by how clean the class hierarchy was designed, and the fact there were various internal error messages scattered about the code to give an indication of the classes' original names and purposes. For example, the code has a single common class CPetGlyph for the square glyphs used in all the different PET tabs, and CPetGlyphs for the collection of glyphs, with all the logic for scrolling and rendering. Individual PET areas simply derived from that, and instantiated whatever glyphs were needed, be they action icons like for Save, Load, and Quit, or inventory items that can be selected.

Once the PET was pretty much done, I started looking into the conversation handling and text parsing. To begin with, I worked one of the easier, self contained areas of vocabulary loading. All the vocab data for Starship Titanic is held in a resource inside the game executable called STVOCAB.TXT. It holds the data for an impressive 4700 words! A lot of these come down to synonyms for common words, and in fact even includes foreign language words, like "ein" as a synonym for "a". Each word entry in the file also contains data for what kind of word it is (such as action, adjective, etc.), as well extra information that is used by the parser to uniquely identify the words. The vocabulary loader builds up  the data for each word in turn, building a master list of overall words support by the game, and a linked list of synonyms for each word.

Once I did that, I started looking into the input processing code that gets called when a line of text is entered in the PET. First of all, the game calls some pre-parse methods. These take care of a bunch of standard processing of the line, including:

• lowercasing everything and removing extra spaces between words
• Conversion of numbers from textual formats into decimal. That was surprising.. the game has special code in place so you can enter numbers like '42', 'forty two', or even 'LXXIII'. That's right.. the parser is so sophisticated, it even handles roman numerals. :)
• Expanding common contractions like "it's" to "it is".

Once preprocessing is done, it then breaks down the entered sentence into a series of words. Each word is checked against the word synonym lists to identify the known word for each entered word. Here again, the parser has extra complexities that surprised me. It can not only find word matches based on exact word matches, it has two fairly complicated routines that can identify various forms of pluralization, as well as common word prefixes like 'super', 'anti', 'counter', etc. and find matches on the word without the prefix. At the end of this process, the entire input has been broken done into a linked chain of recognized words, and the real handling of figuring out the player's input can be done using custom logic for each NPC script.

At this point, whilst I had made some inroads on NPC script handling, I somewhat started drifting on what area I worked on. Strangerke had been looking at some of the core game scripts to add me in writing game logic, so to make things simpler in the long-run I went back to the main CGameObject base class, and spent some time figuring out all the remaining unknown methods. Quite a few of the methods were basically stubs calling PET Control methods, so my prior work on the PET was a great help.

Next, there was a bunch of CGameObject methods calling movie code, so I decided it was finally time to figure out what it was doing. I was able to properly implement the game's OSMovie class which handles movies, and it's AVISurface which handles the low level AVI files. It was actually interesting in that some videos have a second video track - I haven't 100% disassembled the movie drawing code, but my impression is that pixels in one are transparent, and in the other are not.  This required me to add an extension to our AVIDecoder to allow a callback method for selecting which streams the AVIDecoder would use. That way, I could set up two decoders in the AVISurface class, one with the audio and first video track, and the second with the second video track, if present. This neatly avoids the limitations of one video track per decoder. The replacement movie code is all implemented now, but apart from cursor loading working (the cursors are stored frame by frame as a video), movie playback is still crashing, and will have to be debugged further. But at least I know what the bulk of the movie methods are now.

Finally, I moved onto the Star Control class. This is what implements the game's starmap control.. with PET mostly implemented, NPC scripts partially implemented, and movie handling more or less done, only it and sound are left unlooked at. And as I've mentioned previously, sound handling has a lot of spatial processing for sounds that I may simply not implement - the AVIDecoder already handles video sound, and I'm hoping to hook up the other sound methods to simply use standard ScummVM sound decoders. So that makes the star control the only remaining large area to look into.

Oh boy, and what a large area it is. Even basic identification of classes was already bumping up to nearly 30 classes all specific to just the star control. I felt greater and greater dismay the further I looked into it. Honestly, it was like Hopkins FBI all over again, where they included a full Wolfeinstein 3D style minigame in the PC release. In that case we were lucky, and could simply ignore it. Not so much in this case - all the classes will have to be implemented.

As I've looked into it further, and worked on classes that don't depend on others (so are self contained, and in theory easier to figure out), some of them have started to make sense. There's matrix and vector classes.. somewhat surprisingly, two versions of each, one for floating point numbers and one for doubles. I can only assume back in the day there was space vs speed considerations for having both types; I've not yet reversed all the methods, so I don't know if they can be merged in the future. I've also identified two "star points" classes that use large lists of spatial data that was hard-coded in the executable. Hopefully, the other remaining classes won't be too hard to figure out likewise.

So, all in all, things are going good. I'm starting to feel better, and I'm making inroads on the last major area of the game. Once I get that out of the way, it'll be a matter of finishing off the NPC scripts, implementing miscellaneous methods here and there such as sound handling, and then implement the game logic. Which, with all the core engine implemented, will hopefully be a straightforward, if slow process.

Implementing Starship Titanic in ScummVM

What began as a casual lunch-time activity (spread across many, many lunches), my work on Douglas Adam's Starship Titanic is finally starting to see significant results. See below for an example of the current progress from the start of the game: https://www.youtube.com/watch?v=8ypLR4fS6vE

You can't get as far as the ship crashing into your house yet, but at least you can fiddle around with the computer and, more importantly, use the television, and see Douglas Adm's visage scolding you to get on with the game. :)

I was originally attracted to working on the game more for technical reasons then the actual gameplay for several reasons. Firstly, because it was a Windows game.. my previous disassembly work has all been on DOS games, so I thought it would make a nice change of pace. Secondly, the original executable relies on compatibility tweaks to run on modern Windows systems and, according to the GOG forums, multiple people have had trouble getting it to run at all. And thirdly, the ease of disassembly.

The game has, in my opinion, the cleanest and most well thought class structures of any game I've worked on. Part of this clean hierarchy involved the bulk of classes in the game deriving from a common "CSaveableObject" base class that defines, amongst other things, the name of the class. The entire game state is laid out as a tree sturcture, with CRoomItem objects for rooms, CNodeItem objects for positions within a room, CViewItem for the different directional views within a node, and game objects under each view. Saving and loading games, including loading the initial game state for new games, is then a simple matter of dumping the hierarchy to and from disk.

Because of this, it's made it somewhat easier to reverse engineer how the classes are implemented. Since the game loading code needs to be able to locate and create classes by their textual name, it meant that I was able to properly identify and name all the classes the same way they were in the original source code (which I don't have access to). Since the original uses C++ classes and inheritance, it's meant that when I've identified the meaning of a virtual method, I could apply the same name to the same method in all the other objects. So, for exmaple, when I identified the methods for saving and loading an item's fields, I was able to focus on those same methods in all the other classes to handle loading the entire game state.

So as you can see from the above video, progress on the engine is going well. I have all the core logic in place for loading the initial game state, displaying views, and moving between them. I've also been able to graft some of the original's movie class, that handles playing AVI videos for all the game's animations, to use the existing ScummVM AviDecoder. Although there's still parts of CMovie that I don't understand. And there also isn't any background sound yet. Apart from that, there's two main areas left to be handled: firstly, all the hardcoded game logic. If there's one thing that I find a pity, it's that rather than using some kind of scripting system, they chose to implement all the game logic in code. So all the various interactable objects in the game have their own code that will need to be slowly implemented.

The other main area remaining to be figured out is the PET control that provides the game's user interface. Particularly the conversation system, where you can converse with the characters within the game. I've already made some minor in-roads into implementing display of the PET, and various error messages in the original executable have given me some ideas of various classes in the conversation system. But it will still take a while to fully implement the game. Plus of course, dispute recently working on it quite a bit in my off hours, work on this game has primarily been a lunch-time diversion, so it's somewhat limited by my work's totally unreasonable policy that lunch should be limited to only a single hour each day :)