Saturday, March 24, 2018

Xeen playthrough in progress

The Xeen playthrough is finally in progress, having been restarted last night. And by the time I went to work this morning, the first town of Clouds, Vertigo, was completable. Huzzah. \o/. I did identify and fix some further problems, such as armor not equipping, reading the note in the Vertigo warehouse, and getting the experience from the mayor. Next stop will be the Dwarf Mines, which I don't anticipate having much, if any, problems. Particularly since I've done some previous testing in them, though I've not yet been beyond the first level.

This is a good omen for completing the rest of the game. Particularly given the amount of care I've already given to fixing bugs I'd introduced in my reimplementation of the combat system. And the fact that all the various towns share the same code base for implementing the different tavern, guild, etc. locations. I'm expecting/hoping that the rest of the game will be straightforward. Indeed, most of the fixes I did last night were fairly straightforward to fix once I'd identified the bugs in question.

Of course, it's been years since I played the game, and honestly I only really remember some of the highlights of the Dark Side. So this playthrough will be a chance to re-experience the games once again. And my unfamiliarity will ensure I go poking into every nook and cranny. Though the downside is that I may not notice minor things not working, such as searching specific locations not giving the expected items, monsters, or loot. Once I've finished my first playthrough, hopefully some more experienced Xeen players can be enticed to play through the game as well.

Well, better get back to work. It may be Friday, but I still have the entirety of the work day to get through before I can get back to my playtesting :)

Thursday, March 22, 2018

Xeen there, done that

After all the time spent both in development, and in hiatus, work on the Xeen engine nears it's completion. Last weekend, I finally considered the engine finally stable enough to start my first playthrough of the game. Though, unfortunately, that idea rapidly crashed and burned.. I immediately discovered a variety of bugs in my exploration of Vertigo that my previous casual testing hadn't revealed, particularly with the Guild display and purchase of spells.

Thankfully, I fixed all those, and over the course of the week so far I've also since discovered further minor issues that I've also been fixing as I identify them. As of today, I'm once again at a point where I'm not aware of any other bugs, so I'm once again going to restart my playthrough and start playing my way through the game. As last weekend showed, it's better for me to finish my own playthrough first before I announce official testing, since it will be a chance to fix all of the more obvious errors that crop up, and ensure anyone else that tests the game will have a smoother, more enjoyable experience. And speaking of testing, some good news.. rather than concentrating on just getting World of Xeen working, I've also implemented the necessary extra functionality and main menus for Clouds and Dark Side individually, as well as for Swords of Xeen. That means three separate games as well as the combined World will be playable using this new engine.

Recent discussions on the GOG forums have also got me to thinking about what the future could hold for this engine. Whilst I'm definitely going to move onto working on other games after this, and likely take a break to play some games, there are some fertile areas someone with a knowledge of C++ could work on and submit patches to the ScummVM group for. First of all, this GOG thread talks about "Ludmeister's mod". It seems to have been abandoned by the author without ever having an official release. But it did have a description of all the changes it introduced. Since ScummVM also allows for game-specific options in the launcher dialog, someone could add checkboxes for enabling/disabling the functionality and add relevant code to the engine to implement them. Some of the mod's changes sounded nice, particularly regarding the passage of time.

Another thing I'd love to see is for this to revive some interest in creating new areas and/or content. Not to spoil things too much, but I couldn't resist adding some new content of my own. I'll leave it up to players to see if they can discover it. If we're lucky, maybe it will help spur others to create further new content for the game. After all, Swords of Xeen was original a mod done by fans. With the engine source code as a guide, maybe someone can cannibalize the scene rendering code and build a visual world editing tool around it.

Thursday, January 11, 2018

Christmas holiday round-up

Over the course of the Christmas holidays, I took the opportunity to take a break from working on completing the Might & Magic Xeen engine to do some further work reverse engineering and implementing code for my Legend Entertainment games engine in ScummVM. The result can be seen in a demonstration video on Youtube.

For now, I've been focusing on disassembling two of the Legend Entertainment games, Frederick Pohl's Gateway, and Companions of Xanth. The former is one of the earlier games with a combined picture and text parser, whilst the latter is the more standard point-and-click type of adventure game. Both share a lot of code in common; though as I've discussed previously in details, the two have some key differences.. Xanth is a full 8-bit point-and click game, but it uses a very annoying overlay manager that makes dissasembling and debugging the code really trouble. Gateway, being an earlier EGA game, is much easier to do.

Over the week I spent on the games, and my engine, there were two main advances. Firstly, I increased my understanding of how the original did the core event handling and renderings. Renderings in particular were somewhat messy, with different parts of UI elements being drawn in different methods; some when the game window is first opened, and others as event changes occur. Using the listbox in particular as an example, I tracked down where the different parts were being drawn and cleanly re-implemented them together into their own draw method of a Listbox class.

Likewise, the Commset also proved useful in being a separate custom screen which nevertheless used many of the core engine's methods for creating windows, defining interactable regions of the screen, drawing, and event processing. I was almost to the point of implementing the initial text display within the Commset when the week ended.

The second major area was fleshing out my engine's event handling. As can be seen from the example video, game views can now be rendered, and the listbox interacted with. This represents a lot of modular code in my engine taking care of managing what the current view is, as well as gathering mouse events and dispatching them into a hierarchy of parent/child game views that can represent the game screen, and then individual elements within it like the Listbox. A lot of this draws inspiration from the Titanic engine, which heavily uses a hierarchy of game views, as well as message handling for passing messages between elements.

The result is that I'll be able to avoid, for example, a lot of hardcoding the original has for handling the two different listboxes. Instead, I can set up the listbox to generate a message when an item is clicked, and then have processing code in a parent "User Interface" class for the overall game screen that will handle replacing the listbox contents as necessary, as well as passing on selected words and/or phrases to the text entry widget where the player inputs their commands.

All this is to keep hardcoding to a minimum, particularly once I've got a better feel for the engine, and start working on Companions of Xanth in greater detail. The idea is to eventually make it easy to support other games, like Death Gate\, so the less hard-coding there is for specific games the better. This current design allows for a modular build up of game content, with an overall "project item" that contains the different views like the game interface, conversations, and so on. And visual elements like the game interface can be further build up of individual UI components like the game scene, inventory, command buttons, and so on. Much of which will be able to be cleanly re-used between the games.

Anyway, now that the holidays are over, I've returned to work on the Might & Magic Worlds of Xeen engine. I'm wary of calling it "nearly completed", but it certainly has a lot of functionality in it already. So it deserves to be finished, fun as continuing to work on the Legend games might be.

Once that's done, though, who knows when I'll next return to work on the Legend games. I may well do it directly after. Although, the recent discussions of AGS have also piqued my interest as well. Depending on whether or not anyone else has taken up the reins of working on it to any degree, I may refocus my effort onto it. And of course, there are the later MADS games like Return of the Phantom & Dragonsphere that could be worked on. Not to mention the Access engine, with Martian Memorandum. Plus various different other RPGs. It's always nice to have options :)


Monday, December 18, 2017

It has to be Xeen to be believed

Hi everyone,

Been a while since my last blog posting, despite regular protestations that I'd be posting more frequently. My bad. At least my development work has been proceeding with it's usual speed in the interim. The just released ScummVM 2.0 spurred me to write a blog posting.

So what's been happening? First of all, as most will know, I finished work on Starship Titanic. So with the new release, both the English and German versions of the game are now officially supported. I've been informed that there's also a French version, but it isn't available commercially (GOG only has the English and German versions). Given that, for now I'm unwilling to spend the multiple weeks worth of effort to add support for it. Maybe if GOG adds it in sometime in the future, support for it can be revisited.

Next, let's move onto what I'm currently working on, now that Titanic is done.. World of Xeen. Support for the game has already improved enormously over the last few weeks. You can walk around the starting town, fight monsters, then leave the town and wander around the outdoors. Even go to the Dwarf Mine and listen to the dwarf's cheesy intro. Not all maps are working yet, but at least it's getting there.

Still lots of other UI bugs to be looked into. I spent much of the past week cramming to implement all the various in-game cutscenes without testing them yet, so there's bound to be problems. The intro/endgame cutscenes for the games are also still only partly implemented. I'll also likely need to do further work on the savegame code, so the engine definitely isn't ready for any serious playing yet.

Anyway, with Christmas holidays coming up, I'll likely take a brief break for a few weeks from my resumed work on Xeen. Much of the work for reversing the game had already been previously done by the excellent work of WizardStan, and then further by myself. As such, a lot of the time I've spent simply implementing remaining missing code in ScummVM and lots of debugging. Since I've already spent so much time in debugging Starship Titanic code to get it working, I'm looking for a more relaxed, cerebral exercise of reverse engineering something fresh over the holidays.

I'll probably return to my preliminary work on the Legend entertainment games, Companions of Xanth and Gateway. Both share a lot of common engine though Xanth, the more recent, has a horrible overlay manager that makes it a pain to debug. As such, I've found working on the earlier game makes things easier to understand in many places. Though since the game uses EGA graphics, understanding how things are being written to the screen does get problematic in places. So a balanced effort reversing both at the same time is called for. An excellent mental challenge.

Everyone have a Happy Holidays/Year end/Christmas or whatever, as 2017 comes to a close. :)

Monday, June 12, 2017

The player is finally home.. the long way round

It's taken so many nights spent slowly debugging the original executable versus my ScummVM implementation, but the final starfield puzzle of Starship Titanic is finally working.

I'm not 100% happy with how the starfield rotates to selected markers when you've locked them in, but frankly, given that I've spent multiple months on disassembling, implementing, and fixing just this one puzzle, I'm just happy at this point that it works at all.

So now with the starfield puzzle finally completable, I was able to initiate the endgame, and see the ending video and credits:
I have to say, there were times when I grew weary of implementing and testing all the matrix code that the puzzle required. But I guess I'm just too stubborn not to see it all the way through.

So what's happening next? Firstly, there are various minor bugs that I was aware of, but hadn't previously gotten around to fixing. I'm currently working into fixing them now For example, I fixed some jerking of text in the end credits, and some black boxes that briefly appeared over the flames in the canal. I've got some outstanding issues with NPC idle animations to look into. The Bellbot also won't currently bugger off if you tell him goodbye :) Once that's done, I'll give it another playthrough just to make sure before it's announced for public testing. So expect it to be soon. For those of you that don't have the game already, the current GOG sale has it discounted. So it's a very opportune time to pick it up.

On a final note, there are couple of things associated with the game that I don't have any immediate plans to spend time on:
* The QSound library the game uses for simulating sounds in a 3D space using standard stereo output. Many know my distaste of working sound code. So I'll leave it as a future exercise for someone else to work on. I've implemented the low level sound calls using mostly the same interface as QSound exposes, so it should prove convenient for anyone who chooses to do so
* The Indeo 4 decoder still doesn't handle cases where transparency information is embedded directly into the video frames, rather than as a separate video track. Since codecs are installed directly into Windows, I'm not even sure which DLL implements the decoder. I'll try and spend a bit of time trying to figure it out, but worst case, it may be something the game just has to live with. The game currently has "best guess" code that estimates what the transparencies should be. It's not perfect, but it's reasonably servicable for now.
* I don't have any near-term plans to do any further work on the German version. I'm simply too burned out over the game, and want to move on from it. I may return to it one day; I'd also welcome anyone else who wants to look into it themselves.


Saturday, November 5, 2016

There's been a Titanic amount of progress

Since my prior posting earlier in the year, there's been a great deal of progress in Starship Titanic. I decided to put aside the problem of reverse engineering all the Star Map classes until I had the rest of the game working better. In that respect, I've made great strides since, as of last weekend, I was able to complete the entire "prologue" of the game  That included using the computer, experience the crash, talking to the Doorbot, entering the ship, and viewing the Credits. Huzzah. \o/

I was going to prepare a video showing the intro, but with the most recent changes, there seems to be some instability showing up. It seems like something that was already present, just coincidental that the newer changes result in more frequent crashes. It's kind of hard to narrow down the cause, as there's also a problem with the implementation of the Indeo video decoder we're using for NPC videos like for the Doorbot, where it's reading past the end of the frame data. So it's difficult to track down the memory corruption, as warnings about the decoder are completely overwhelming everything else.

So for now, I'll present a screenshot of the amazing multi-color Doorbot :)

After thinking over matters, I've decided to keep progressing into the game, and come back to look at the problem later on. Part of the trouble I'd been having with the code was the sheer length of the intro as I got further and further into it.. requiring me to wait through several minutes of cutscene & conversation every time I made any changes or bugfixes. Even if the intro has suddenly become unstable, I still have savegames I made from beyond it, so I'm using them as a starting point to make further progress testing into the game.

Speaking of testing, I've had a major boon to my efforts to track down bugs in the code. I was previously stymied trying to test the original Windows executable in the IDA debugger, since it kept crashing on me. Plus running in compatibility mode full-screen didn't help either. And without the ability to see "valid" values in the original executable, I anticipated it would be difficult to track down errors in my code, since I wouldn't know whether values/state at any point in time were already wrong on not.

Luckily, though, I stumbled on a solution. Using the Visual Studio "Attach to Process" allowed me to attach to the game executable without it crashing, unlike IDA. At least, for the majority of the time. Though switching from the game to the debugger and back again caused severe corruption of the full-screen display. Luckily, though, there had been some previous discussions about running the game in a window - I was able to use a utility called DXWnd that intercepted the game's DirectX calls and forced it to run a window. The result wasn't perfect, in my opinion, for anyone wanting to play through the game, but it's worked well enough for my purposes, in conjunction with Visual Studio.

As a result, I'm now making much better progress than I had anticipated, and hunting down bugs is in general much easier than I'd anticipated. Let's hope that stays the case.. my next major gameplay milestone is to complete more extensive conversation with the Deskbot to get myself a room. The basic yes/no detection for the Doorbot worked pretty smoothly first time I tried it. The Deskbot, though, is using more of the conversation parser - I've already located and fixed some problems with it. Let's hope that there won't end up being too many.

On a final note, the one downside of my surging progress with Titanic is that I'm currently spending less time working on finishing my Xeen engine. I'd originally anticipated the frequent roadblocks trying to hunt down bugs in Titanic would have me growling in frustration, and switching to Xeen for awhile to unwind a bit. Now with the ability to debug the original executable, that hasn't really happened so far, and hopefully won't happen. I'll probably end up spending more time right now focused solely on Titanic, and see if I can't get the bulk of the game with the exception of the final starmap working by the end of the year. Then I'll be in a better position to alternate between working on Xeen and trying to disassemble the remainder of the Starmap classes.


Sunday, July 17, 2016

PET is PrETty much done

Hi all. Been a while since my last posting; and a hectic last couple of weeks. After flying down to Florida for a 4th July long weekend getaway, I was barely back before I had to get ready for a work trip to the west coast. Then I get back to two days of company photo-shoots, which kind of disrupted getting back to normal work; but no sooner than they were done, I come down with a cold and had to stay home for several days. And was too sick to even really use my computer until now. :(

Nevertheless, despite all of that, work has been progressing on Starship Titanic since my last posting, and I've made a lot of progress. Firstly, there's the PET User Interface. This was one of the major areas that still had to be investigated last time. Since then, I've spent a lot of time figuring out how everything works, and re-implementing it all in my ScummVM engine. Behold the fruits of my labor:

Code for all the various areas has been pretty much all implemented. You can enter text in the conversation tab, and click stuff in all the others. There's just a few minor visual and functionality glitches here and there, such as the dials in the above image. Some of the tabs, like the Remote and Rooms tabs, are mostly untested at the moment.. it will be easier to test when I have more of the game logic implemented, and can properly test them with live data. Saving and Loading is also waiting on me to finish the saving and loading code.

As before, I was aided somewhat in implementing the PET by how clean the class hierarchy was designed, and the fact there were various internal error messages scattered about the code to give an indication of the classes' original names and purposes. For example, the code has a single common class CPetGlyph for the square glyphs used in all the different PET tabs, and CPetGlyphs for the collection of glyphs, with all the logic for scrolling and rendering. Individual PET areas simply derived from that, and instantiated whatever glyphs were needed, be they action icons like for Save, Load, and Quit, or inventory items that can be selected.

Once the PET was pretty much done, I started looking into the conversation handling and text parsing. To begin with, I worked one of the easier, self contained areas of vocabulary loading. All the vocab data for Starship Titanic is held in a resource inside the game executable called STVOCAB.TXT. It holds the data for an impressive 4700 words! A lot of these come down to synonyms for common words, and in fact even includes foreign language words, like "ein" as a synonym for "a". Each word entry in the file also contains data for what kind of word it is (such as action, adjective, etc.), as well extra information that is used by the parser to uniquely identify the words. The vocabulary loader builds up  the data for each word in turn, building a master list of overall words support by the game, and a linked list of synonyms for each word.

Once I did that, I started looking into the input processing code that gets called when a line of text is entered in the PET. First of all, the game calls some pre-parse methods. These take care of a bunch of standard processing of the line, including:

  • lowercasing everything and removing extra spaces between words
  • Conversion of numbers from textual formats into decimal. That was surprising.. the game has special code in place so you can enter numbers like '42', 'forty two', or even 'LXXIII'. That's right.. the parser is so sophisticated, it even handles roman numerals. :)
  • Expanding common contractions like "it's" to "it is".

Once preprocessing is done, it then breaks down the entered sentence into a series of words. Each word is checked against the word synonym lists to identify the known word for each entered word. Here again, the parser has extra complexities that surprised me. It can not only find word matches based on exact word matches, it has two fairly complicated routines that can identify various forms of pluralization, as well as common word prefixes like 'super', 'anti', 'counter', etc. and find matches on the word without the prefix. At the end of this process, the entire input has been broken done into a linked chain of recognized words, and the real handling of figuring out the player's input can be done using custom logic for each NPC script.

At this point, whilst I had made some inroads on NPC script handling, I somewhat started drifting on what area I worked on. Strangerke had been looking at some of the core game scripts to add me in writing game logic, so to make things simpler in the long-run I went back to the main CGameObject base class, and spent some time figuring out all the remaining unknown methods. Quite a few of the methods were basically stubs calling PET Control methods, so my prior work on the PET was a great help.

Next, there was a bunch of CGameObject methods calling movie code, so I decided it was finally time to figure out what it was doing. I was able to properly implement the game's OSMovie class which handles movies, and it's AVISurface which handles the low level AVI files. It was actually interesting in that some videos have a second video track - I haven't 100% disassembled the movie drawing code, but my impression is that pixels in one are transparent, and in the other are not.  This required me to add an extension to our AVIDecoder to allow a callback method for selecting which streams the AVIDecoder would use. That way, I could set up two decoders in the AVISurface class, one with the audio and first video track, and the second with the second video track, if present. This neatly avoids the limitations of one video track per decoder. The replacement movie code is all implemented now, but apart from cursor loading working (the cursors are stored frame by frame as a video), movie playback is still crashing, and will have to be debugged further. But at least I know what the bulk of the movie methods are now.

Finally, I moved onto the Star Control class. This is what implements the game's starmap control.. with PET mostly implemented, NPC scripts partially implemented, and movie handling more or less done, only it and sound are left unlooked at. And as I've mentioned previously, sound handling has a lot of spatial processing for sounds that I may simply not implement - the AVIDecoder already handles video sound, and I'm hoping to hook up the other sound methods to simply use standard ScummVM sound decoders. So that makes the star control the only remaining large area to look into.

Oh boy, and what a large area it is. Even basic identification of classes was already bumping up to nearly 30 classes all specific to just the star control. I felt greater and greater dismay the further I looked into it. Honestly, it was like Hopkins FBI all over again, where they included a full Wolfeinstein 3D style minigame in the PC release. In that case we were lucky, and could simply ignore it. Not so much in this case - all the classes will have to be implemented.

As I've looked into it further, and worked on classes that don't depend on others (so are self contained, and in theory easier to figure out), some of them have started to make sense. There's matrix and vector classes.. somewhat surprisingly, two versions of each, one for floating point numbers and one for doubles. I can only assume back in the day there was space vs speed considerations for having both types; I've not yet reversed all the methods, so I don't know if they can be merged in the future. I've also identified two "star points" classes that use large lists of spatial data that was hard-coded in the executable. Hopefully, the other remaining classes won't be too hard to figure out likewise.

So, all in all, things are going good. I'm starting to feel better, and I'm making inroads on the last major area of the game. Once I get that out of the way, it'll be a matter of finishing off the NPC scripts, implementing miscellaneous methods here and there such as sound handling, and then implement the game logic. Which, with all the core engine implemented, will hopefully be a straightforward, if slow process.