Seasons greeting and off on holiday

Seasons greetings to all, and I hope everyone has a happy time as we finish 2018 and look forward to the beginning of 2019. As I sit in the Los Angeles airport writing this, I'm reminded yet again of how much fun I have doing reverse engineering, since even the smallest thread of work can lead you in unexpected directions as you trace the flow of what methods call a given method, and what methods it calls.

For something to do, I opened up my Companions of Xanth in progress disassembly to do a bit of pottering around with it. Just by chance, scrolling through the disassembly, I noticed a call to a method I was pretty sure took in a string, but in this case was passing in a value. I then quickly realised it was an offset n the data segment that hadn't been properly resolved, so I did that. The message in question turned out to be a generic "insufficient memory". So from it's usage, I was able to identify the entire method it was in.. it's called during startup, and ensures that there's enough free memory to play the game. Ironically, it looks like the code to actually display the message if there isn't enough was broken, as evidenced when I ran the game in DosBox debugger.

From there, identified a method just before the message call that took in a number and passed out a string.. it wasn't too hard to identify it as a number formatting routine. I confirmed it in the DosBox debugger by putting a breakpoint in the machine code, and seeing the result from the method. I then noticed that there was one other method calling it, so I was intrigued to find out what it was. It turned out it too was a block of code in a method that printed out the free memory, in a method that looked like a big switch based on a single value. It looked like a keypress. Playing the game, I was able to confirm that pressing 'M' on the keyboard did call the code, which printed out the amount of free memory in the status area.

This was a good woo-hoo method. I had identified the method for handling keypresses. Likely, some of the other keypresses do fairly identifiable things, like saving and loading, so knowing where each key's code is, I'll be able to trace into what they do, and what methods they call. In fact, the free memory printing for the 'M' key already prove dividends. From it, I was able to identify the method that cleared the status area, and a version of 'sprintf' for printing text in the status area. In the future I'll be able to look further into other places that call the method to add lines to the status area, and start identifying other parts of the game code. For example, looking at an item produces a description in the status area, so by setting a breakpoint in the status text writing code, I'll be able to identify what method calls it, and start identifying the structure of items that likely have a "description" as one of it's fields.

All this progress today came from a minor diversion to pass the time sitting in the airport. The notice of an incorrect parameter to a method call has unraveled more of the game's secrets, and giving me multiple further areas I can investigate. A nice way to start the holidays.

It's time for a very verbose change of pace

One of the things I like about my work on ScummVM is that it's a hobby rather than a profession, which gives me the freedom to switch what I'm working on when I desire. I've previously had encouragement to resurrect my Gargoyle engine work from about a decade ago. It was an early attempt to add support for Frotz into ScummVM, and at the time it was rejected due to multiple reasons. It was specific only for Infocom/zcode games, lacked clipboard, truetype fonts, and Unicode support. Most of these things have since been solved by the ScummVM framework itself, which now supports all of them. So I thought, why the heck not, and decided to work on it again.

However, my original Gargoyle engine being limited to just Zcode was a problem. Not only wasn't it extensible to other IF systems, but it would also take work to properly reimplement it to support Unicode. In the end, it was just simpler to scrap it all, and start from scratch. Luckily, there's the very convenient Glk specification, that was specifically created for easing porting the various Interactive Fiction interpreters to other systems. I knew that by supporting it, it would make it easier to create a framework that could support multiple different interpreters as sub-engines.

So I created a new Gargoyle engine, which I've since renamed it to ScummGlk to avoid ambiguity with the stand-alone Gargoyle project which also implements the Glk specification. After over a month of work, I've created an engine that implements the bulk of the API, and implements two sub-engines under it so far.. Scott for Scott Adams games, and Frotz for playing Z-Code/Infocom games. Scott was useful as a test case, due to it's simplicity. With it as the first sub-engine, I was able to focus on getting the ScummGlk core working. Then Frotz was a fairly obvious second choice, given it's popularity and large number of available fan games.

As of last weekend, the review period was completed for the engine, and it was merged into master. Daily builds now include support for both sub-engines. I've spend the last week fleshing out detection entries for all the ZCode games on the if-archive, so all the games will be automatically detected by the launcher. I'm going to wait on announcing an official testing period until I finish support in Frotz for V6 games. But if you can't wait, you're certainly welcome to jump the gun and try playing some games. Maybe one of the early Zork from GOG, or the immortal classic Freefall :)

Having the engine included into master will be a real nostalgia trip for me. Even prior to my original Gargoyle module, and before I even started working on implementing adventures in ScummVM, I spent several years as part of the Interactive Fiction community. Indeed, the Hitchhiker's Guide to the Galaxy was one of the first computer games I ever had. I spent way too many hours getting frustrated with the game's puzzles.

It's an UltimaTE unwinding

After all this time, support for the Might & Magic Xeen games is finally close to being finished. It's currently the official testing period for all the Xeen games, and I've already had a whole bunch of useful bug reports, which I've all taken care of. At this point, it's almost certain that the Xeen engine, and games, will be supported in the next official ScummVM release. It's a nice feeling. More than any other game engine I've worked on, the Xeen engine was the one that got put on hiatus the most number of times. It's very satisfying to finally complete it.

What's in store for me now? Well, I do want to spend some time properly unwinding and actually playing some games. They've only just come out with the sequel to Pillars of Eternity, which I'm now getting into, and plus I have a backlog of other RPG and adventure games that I want to have a chance to properly play. So that will probably keep me occupied for a few months at least. I've also got a move from the US East Coast to West Coast coming up next month. So various preparations are consuming my time, and it's all to the good that work on Xeen has already finished.

Not that I'm totally stopping work on ScummVM stuff. As with the early days of work on Starship Titanic, I started doing some casual reverse engineering during my lunch breaks. By happenstance, I experimented with the Ultima 1 executables, and become enamored by how simple they were to reverse. Both due to the simplicity of the game (considering how old it was), and the way the code was implemented. It was just the thing for doing casual reversing work in chunks of an hour at a time.

As a result of this, I've already implemented a great deal of foundation code in ScummVM for supporting the game, with a long term eye for having a clean engine structure that could support all the early Ultima games.  Here's some screenshots of the player within the overworld, a city, and within the dungeons:

At this point, I've got basic movement functionality working. The player can roam around the map, entering cities, castles, and dungeons. Dungeon movement also works, as does going up and down ladders, so the dungeon can be further ventured down into, or left entirely. That's it at the moment, though; it doesn't yet have any combat, magic, inventory, character interaction, etc. See this Youtube video for an example walk-about that covers venturing into a dungeon, and then heading over to the city of Britian.

Don't expect too much more work on this in the near term, though. As I said, this is more a side thing, and I definitely do want to spend some time unwinding and playing some games. When I get my fill, later in the year, then we'll see. At the moment, it's looking strongly like I'll keep working further on this. I've some crazy ideas about not only adding support for the game, but also reusing the Ultima VI tileset to create an "enhanced" Ultima 1 experience that's more visually pleasing, and rework the controls and user interface to be likewise more player friendly. I know that oh, pretty much everyone who ever played their way through the space simulation section would much appreciate mouse control for aiming at enemy vessels. :)

I kind of consider the RPG series of Ultima, Might & Magic, and Wizardry the triumvirate of major RPGs from the last century and, now that ScummVM is embracing RPGs in addition to adventures, I'd love to see them all under the ScummVM banner. Now that at least some of the earlier Might & Magic games are supported, it may well be time to start in on Ultima support :) Of course, I'm not entirely abandoning working on adventure games. I'd previously made some promising initial progress in reversing some of the Legend Entertainment games.. Companions of Xanth and Gateway. Irrespective of how much time I end up devoting to working on Ultima I, I may end up also devoting some of my time to those games as well.

Anyway, that's probably enough for now. I still want to get in some Pillars of Eternity 2 before it's time for bed. :)

Working my way through Dark Side

I've spent the last week with an in progress working my way through Dark Side of Xeen. As of last night, I've mapped out the bulk of the southern side of the map up to the mountains, released the first stage of Queen Kallindra's castle, finished the Temple of Bark, and am just winding up clearing Sandcaster and it's sewers of enemies. I've identified and fixed multiple problems as I went, and not all of them were specific to Dark Side.

For example, one bug I'd been meaning to fix for a while was finally resolved - when monsters were triggered to hate & attack the entire party at the same time, they were originally only ever attacking the first party member. This made the Dragon Caves on the Clouds side *so* much easier.. all the dragons would wail away on my first character, even after he was dead, leaving the rest of my party to kill them at their leisure. By the time I left the dragon caves, my first party member was below -5000HP :) Another, ironically, coincidental script problem meant the dragon containers didn't tax me when I walked past them, so I didn't even have to worry about losing gold and gems.

I also fixed some bugs in the sound system that could cause crashes when multiple sound effects were played in rapid succession, and a bug in the combat system that could case a crash when recalculating the order that characters and monsters fight in based on their speed. The later I'm particularly hoping will help resolve the outstanding rare crashes I was previously getting with extended combat. It may be naive.. I'll have to wait and see whether any further crashes occur.

So yeh, things are progressing, and the engine is getting more and more stable; I'll keep working my way through the game. I've got a long weekend coming up tomorrow, but honestly, I'm not sure how much time over the weekend I'll end up playing the game. I think I'll keep the bulk of it for some R&R and doing other things. We'll see how it goes.

Clouds of Xeen is now completable

Good news on the progress of the Xeen engine in ScummVM. As of the weekend, I finally finished my first playthrough of Clouds of Xeen! I finally got to give smackdown on Lord Xeen, and watch the ending cutscene. Which I've admittedly previously seen multiple times when I was first implementing the code, but there was the emotional satisfaction of seeing it properly, rather than just using the mirror cheats. Though I will confess that I got so overeager to finally finish the game, that I lacked the patience to do it properly (with well/fountain stat raises), and instead used my debugger cheats instead. But it doesn't take away from the fact that the first game is confirmed as completable. \o/

Now, it's onto Dark Side. There were actually more bugs due to changes done for Dark Side (code I'd implemented but not yet tested) than I anticipated. I already fixed multiple different bugs on the weekend that were causing hangs and crashes. Currently I'm looking into a somewhat complicated problem with how wall decorations are handled.. it causes a crash that makes Ellinger's Tower impossible to complete. So Dark Side definitely isn't completable, and needs further testing. On the plus side, though, it shouldn't take as long to test and finish Dark Side than my Clouds testing took, since Dark Side benefits from all the fixes I've already done for Clouds.

At this rate, my initial play through & tests of both Dark Side and Swords should be done within the next month, and  the game will likely be stable enough for a public testing period. Not long now :)

Work on Xeen is up in the air

As tempting as it would be to write this post yesterday on April Fools, I've left it to today to make a status update on my playthrough through Xeen. Work has been proceeding well. I've fixed lots of bugs that were identified by both myself and Dark-Star, who's been assisting me with preliminary testing. As of yesterday, I've:
- Finished the Dwarf Mines
- Mostly finished Rivercity (except for the Knights behind the training area)
- Completed the Witch Tower. Hence the post title.. the levitate spell works, and I've been walking around on the clouds :)

Many of the bugs were crashes and hangs, so it highlights that the game still isn't ready for general public testing. The Witch Tower, for example, required two fixes to get it to work.. one where I wasn't setting whether the party had the key (so it was never letting you in), and one where I incorrectly implemented the "return from subroutine" script opcode, so trying to go back into the Witch Tower from the stairs in the Witch Tower clouds hung the game. Ironic that both ways into the tower were glitched. :)

I think combat is also more settled now. Previously, there were various crashes, and none of the combat spells worked. Now, though, I'm able to properly cast them, so the monsters around Rivercity are much more manageable to defeat. Though I did still experience a problem once where the party wasn't immediately registered as dead after all being killed, so obviously still needs a bit of polishing to fix such remaining issues. Speaking of spells, I decided it was acceptable, given the circumstances, to cheat to give myself extra funds to purchases all the spells for each character. There was little point in delaying matters, trying to accumulate wealth. It also meant I was able to buy pathfinding and mountaineering skills for my party so I wouldn't be constrained where I could travel. It already proved useful, as I was able to identify and fix several bugs with the close-up rendering of forests, where they weren't properly being scaled, and were drawing outside of the scene area.

From here, I'll keep on working my way through the game, fixing bugs as I go. Given I'm already significantly into the game (at least for Clouds), I don't anticipate there'll be any major impediments to my progress. Apart from tracking down remaining issues with combat, hopefully the rest of the game will just be fixing minor problems, like that which prevented me from entering the Witch Tower.

Xeen playthrough in progress

The Xeen playthrough is finally in progress, having been restarted last night. And by the time I went to work this morning, the first town of Clouds, Vertigo, was completable. Huzzah. \o/. I did identify and fix some further problems, such as armor not equipping, reading the note in the Vertigo warehouse, and getting the experience from the mayor. Next stop will be the Dwarf Mines, which I don't anticipate having much, if any, problems. Particularly since I've done some previous testing in them, though I've not yet been beyond the first level.

This is a good omen for completing the rest of the game. Particularly given the amount of care I've already given to fixing bugs I'd introduced in my reimplementation of the combat system. And the fact that all the various towns share the same code base for implementing the different tavern, guild, etc. locations. I'm expecting/hoping that the rest of the game will be straightforward. Indeed, most of the fixes I did last night were fairly straightforward to fix once I'd identified the bugs in question.

Of course, it's been years since I played the game, and honestly I only really remember some of the highlights of the Dark Side. So this playthrough will be a chance to re-experience the games once again. And my unfamiliarity will ensure I go poking into every nook and cranny. Though the downside is that I may not notice minor things not working, such as searching specific locations not giving the expected items, monsters, or loot. Once I've finished my first playthrough, hopefully some more experienced Xeen players can be enticed to play through the game as well.

Well, better get back to work. It may be Friday, but I still have the entirety of the work day to get through before I can get back to my playtesting :)

Xeen there, done that

After all the time spent both in development, and in hiatus, work on the Xeen engine nears it's completion. Last weekend, I finally considered the engine finally stable enough to start my first playthrough of the game. Though, unfortunately, that idea rapidly crashed and burned.. I immediately discovered a variety of bugs in my exploration of Vertigo that my previous casual testing hadn't revealed, particularly with the Guild display and purchase of spells.

Thankfully, I fixed all those, and over the course of the week so far I've also since discovered further minor issues that I've also been fixing as I identify them. As of today, I'm once again at a point where I'm not aware of any other bugs, so I'm once again going to restart my playthrough and start playing my way through the game. As last weekend showed, it's better for me to finish my own playthrough first before I announce official testing, since it will be a chance to fix all of the more obvious errors that crop up, and ensure anyone else that tests the game will have a smoother, more enjoyable experience. And speaking of testing, some good news.. rather than concentrating on just getting World of Xeen working, I've also implemented the necessary extra functionality and main menus for Clouds and Dark Side individually, as well as for Swords of Xeen. That means three separate games as well as the combined World will be playable using this new engine.

Recent discussions on the GOG forums have also got me to thinking about what the future could hold for this engine. Whilst I'm definitely going to move onto working on other games after this, and likely take a break to play some games, there are some fertile areas someone with a knowledge of C++ could work on and submit patches to the ScummVM group for. First of all, this GOG thread talks about "Ludmeister's mod". It seems to have been abandoned by the author without ever having an official release. But it did have a description of all the changes it introduced. Since ScummVM also allows for game-specific options in the launcher dialog, someone could add checkboxes for enabling/disabling the functionality and add relevant code to the engine to implement them. Some of the mod's changes sounded nice, particularly regarding the passage of time.

Another thing I'd love to see is for this to revive some interest in creating new areas and/or content. Not to spoil things too much, but I couldn't resist adding some new content of my own. I'll leave it up to players to see if they can discover it. If we're lucky, maybe it will help spur others to create further new content for the game. After all, Swords of Xeen was original a mod done by fans. With the engine source code as a guide, maybe someone can cannibalize the scene rendering code and build a visual world editing tool around it.

Christmas holiday round-up

Over the course of the Christmas holidays, I took the opportunity to take a break from working on completing the Might & Magic Xeen engine to do some further work reverse engineering and implementing code for my Legend Entertainment games engine in ScummVM. The result can be seen in a demonstration video on Youtube.

For now, I've been focusing on disassembling two of the Legend Entertainment games, Frederick Pohl's Gateway, and Companions of Xanth. The former is one of the earlier games with a combined picture and text parser, whilst the latter is the more standard point-and-click type of adventure game. Both share a lot of code in common; though as I've discussed previously in details, the two have some key differences.. Xanth is a full 8-bit point-and click game, but it uses a very annoying overlay manager that makes dissasembling and debugging the code really trouble. Gateway, being an earlier EGA game, is much easier to do.

Over the week I spent on the games, and my engine, there were two main advances. Firstly, I increased my understanding of how the original did the core event handling and renderings. Renderings in particular were somewhat messy, with different parts of UI elements being drawn in different methods; some when the game window is first opened, and others as event changes occur. Using the listbox in particular as an example, I tracked down where the different parts were being drawn and cleanly re-implemented them together into their own draw method of a Listbox class.

Likewise, the Commset also proved useful in being a separate custom screen which nevertheless used many of the core engine's methods for creating windows, defining interactable regions of the screen, drawing, and event processing. I was almost to the point of implementing the initial text display within the Commset when the week ended.

The second major area was fleshing out my engine's event handling. As can be seen from the example video, game views can now be rendered, and the listbox interacted with. This represents a lot of modular code in my engine taking care of managing what the current view is, as well as gathering mouse events and dispatching them into a hierarchy of parent/child game views that can represent the game screen, and then individual elements within it like the Listbox. A lot of this draws inspiration from the Titanic engine, which heavily uses a hierarchy of game views, as well as message handling for passing messages between elements.

The result is that I'll be able to avoid, for example, a lot of hardcoding the original has for handling the two different listboxes. Instead, I can set up the listbox to generate a message when an item is clicked, and then have processing code in a parent "User Interface" class for the overall game screen that will handle replacing the listbox contents as necessary, as well as passing on selected words and/or phrases to the text entry widget where the player inputs their commands.

All this is to keep hardcoding to a minimum, particularly once I've got a better feel for the engine, and start working on Companions of Xanth in greater detail. The idea is to eventually make it easy to support other games, like Death Gate\, so the less hard-coding there is for specific games the better. This current design allows for a modular build up of game content, with an overall "project item" that contains the different views like the game interface, conversations, and so on. And visual elements like the game interface can be further build up of individual UI components like the game scene, inventory, command buttons, and so on. Much of which will be able to be cleanly re-used between the games.

Anyway, now that the holidays are over, I've returned to work on the Might & Magic Worlds of Xeen engine. I'm wary of calling it "nearly completed", but it certainly has a lot of functionality in it already. So it deserves to be finished, fun as continuing to work on the Legend games might be.

Once that's done, though, who knows when I'll next return to work on the Legend games. I may well do it directly after. Although, the recent discussions of AGS have also piqued my interest as well. Depending on whether or not anyone else has taken up the reins of working on it to any degree, I may refocus my effort onto it. And of course, there are the later MADS games like Return of the Phantom & Dragonsphere that could be worked on. Not to mention the Access engine, with Martian Memorandum. Plus various different other RPGs. It's always nice to have options :)

DreamMaster.