Thursday, January 24, 2019

Frotz work in ScummGlk is ongoing

Work on the ScummGlk engine is ongoing. Since Christmas holidays I've been primarily focusing on adding support for the version 6 graphical Infocom games to the Frotz sub-engine, with particular focus on Zork Zero to begin with. Whilst I don't count the version 6 games among the best of the original Infocom line-up, ScummVM has always been about providing as good a coverage of old games as possible. So from my point of view it's worth spending the extra time adding support for the remaining Infocom games now rather than later, and only moving onto the next sub-engine once that's done.

Below is a screenshot of Zork Zero as of when I came back on the 7th January:


As you can see from the screenshot, the background is starting to be properly rendered, and the 'A' and location glyphs are properly rendered and interleaved with the text. The status bar still needs some work, though.. although it is being rendered, it's then being completely wiped out by the status bar window, which isn't even displaying any text yet.

Right now, I'm focusing on adding the 8x8 fixed size font used in the original games. If you look closely at the screenshot above, the current resizable TrueType font doesn't really look crisp at that low a resolution, and is somewhat blurry. I'm currently making a tool to use the same font data that Frotz has, and produce bitmaps to add to the fonts Zip file for the different 8x8 font styles (normal, bold, emphasized, and bold & emphasized). It has had it's share of issues:
 Somehow, I don't think I've got it quite right yet.. :)

Anyway, progress is gradually being made, and I hope to have Zork Zero fully supported soon. The other v6 games currently error out immediately during startup, but I'm hoping further adding support for them will be a simpler matter.

Sunday, December 23, 2018

Seasons greeting and off on holiday

Seasons greetings to all, and I hope everyone has a happy time as we finish 2018 and look forward to the beginning of 2019. As I sit in the Los Angeles airport writing this, I'm reminded yet again of how much fun I have doing reverse engineering, since even the smallest thread of work can lead you in unexpected directions as you trace the flow of what methods call a given method, and what methods it calls.

For something to do, I opened up my Companions of Xanth in progress disassembly to do a bit of pottering around with it. Just by chance, scrolling through the disassembly, I noticed a call to a method I was pretty sure took in a string, but in this case was passing in a value. I then quickly realised it was an offset n the data segment that hadn't been properly resolved, so I did that. The message in question turned out to be a generic "insufficient memory". So from it's usage, I was able to identify the entire method it was in.. it's called during startup, and ensures that there's enough free memory to play the game. Ironically, it looks like the code to actually display the message if there isn't enough was broken, as evidenced when I ran the game in DosBox debugger.

From there, identified a method just before the message call that took in a number and passed out a string.. it wasn't too hard to identify it as a number formatting routine. I confirmed it in the DosBox debugger by putting a breakpoint in the machine code, and seeing the result from the method. I then noticed that there was one other method calling it, so I was intrigued to find out what it was. It turned out it too was a block of code in a method that printed out the free memory, in a method that looked like a big switch based on a single value. It looked like a keypress. Playing the game, I was able to confirm that pressing 'M' on the keyboard did call the code, which printed out the amount of free memory in the status area.

This was a good woo-hoo method. I had identified the method for handling keypresses. Likely, some of the other keypresses do fairly identifiable things, like saving and loading, so knowing where each key's code is, I'll be able to trace into what they do, and what methods they call. In fact, the free memory printing for the 'M' key already prove dividends. From it, I was able to identify the method that cleared the status area, and a version of 'sprintf' for printing text in the status area. In the future I'll be able to look further into other places that call the method to add lines to the status area, and start identifying other parts of the game code. For example, looking at an item produces a description in the status area, so by setting a breakpoint in the status text writing code, I'll be able to identify what method calls it, and start identifying the structure of items that likely have a "description" as one of it's fields.

All this progress today came from a minor diversion to pass the time sitting in the airport. The notice of an incorrect parameter to a method call has unraveled more of the game's secrets, and giving me multiple further areas I can investigate. A nice way to start the holidays.


Saturday, December 15, 2018

It's time for a very verbose change of pace

One of the things I like about my work on ScummVM is that it's a hobby rather than a profession, which gives me the freedom to switch what I'm working on when I desire. I've previously had encouragement to resurrect my Gargoyle engine work from about a decade ago. It was an early attempt to add support for Frotz into ScummVM, and at the time it was rejected due to multiple reasons. It was specific only for Infocom/zcode games, lacked clipboard, truetype fonts, and Unicode support. Most of these things have since been solved by the ScummVM framework itself, which now supports all of them. So I thought, why the heck not, and decided to work on it again.

However, my original Gargoyle engine being limited to just Zcode was a problem. Not only wasn't it extensible to other IF systems, but it would also take work to properly reimplement it to support Unicode. In the end, it was just simpler to scrap it all, and start from scratch. Luckily, there's the very convenient Glk specification, that was specifically created for easing porting the various Interactive Fiction interpreters to other systems. I knew that by supporting it, it would make it easier to create a framework that could support multiple different interpreters as sub-engines.

So I created a new Gargoyle engine, which I've since renamed it to ScummGlk to avoid ambiguity with the stand-alone Gargoyle project which also implements the Glk specification. After over a month of work, I've created an engine that implements the bulk of the API, and implements two sub-engines under it so far.. Scott for Scott Adams games, and Frotz for playing Z-Code/Infocom games. Scott was useful as a test case, due to it's simplicity. With it as the first sub-engine, I was able to focus on getting the ScummGlk core working. Then Frotz was a fairly obvious second choice, given it's popularity and large number of available fan games.

As of last weekend, the review period was completed for the engine, and it was merged into master. Daily builds now include support for both sub-engines. I've spend the last week fleshing out detection entries for all the ZCode games on the if-archive, so all the games will be automatically detected by the launcher. I'm going to wait on announcing an official testing period until I finish support in Frotz for V6 games. But if you can't wait, you're certainly welcome to jump the gun and try playing some games. Maybe one of the early Zork from GOG, or the immortal classic Freefall :)

Having the engine included into master will be a real nostalgia trip for me. Even prior to my original Gargoyle module, and before I even started working on implementing adventures in ScummVM, I spent several years as part of the Interactive Fiction community. Indeed, the Hitchhiker's Guide to the Galaxy was one of the first computer games I ever had. I spent way too many hours getting frustrated with the game's puzzles.


Monday, May 21, 2018

It's an UltimaTE unwinding

After all this time, support for the Might & Magic Xeen games is finally close to being finished. It's currently the official testing period for all the Xeen games, and I've already had a whole bunch of useful bug reports, which I've all taken care of. At this point, it's almost certain that the Xeen engine, and games, will be supported in the next official ScummVM release. It's a nice feeling. More than any other game engine I've worked on, the Xeen engine was the one that got put on hiatus the most number of times. It's very satisfying to finally complete it.

What's in store for me now? Well, I do want to spend some time properly unwinding and actually playing some games. They've only just come out with the sequel to Pillars of Eternity, which I'm now getting into, and plus I have a backlog of other RPG and adventure games that I want to have a chance to properly play. So that will probably keep me occupied for a few months at least. I've also got a move from the US East Coast to West Coast coming up next month. So various preparations are consuming my time, and it's all to the good that work on Xeen has already finished.

Not that I'm totally stopping work on ScummVM stuff. As with the early days of work on Starship Titanic, I started doing some casual reverse engineering during my lunch breaks. By happenstance, I experimented with the Ultima 1 executables, and become enamored by how simple they were to reverse. Both due to the simplicity of the game (considering how old it was), and the way the code was implemented. It was just the thing for doing casual reversing work in chunks of an hour at a time.

As a result of this, I've already implemented a great deal of foundation code in ScummVM for supporting the game, with a long term eye for having a clean engine structure that could support all the early Ultima games.  Here's some screenshots of the player within the overworld, a city, and within the dungeons:



At this point, I've got basic movement functionality working. The player can roam around the map, entering cities, castles, and dungeons. Dungeon movement also works, as does going up and down ladders, so the dungeon can be further ventured down into, or left entirely. That's it at the moment, though; it doesn't yet have any combat, magic, inventory, character interaction, etc. See this Youtube video for an example walk-about that covers venturing into a dungeon, and then heading over to the city of Britian.

Don't expect too much more work on this in the near term, though. As I said, this is more a side thing, and I definitely do want to spend some time unwinding and playing some games. When I get my fill, later in the year, then we'll see. At the moment, it's looking strongly like I'll keep working further on this. I've some crazy ideas about not only adding support for the game, but also reusing the Ultima VI tileset to create an "enhanced" Ultima 1 experience that's more visually pleasing, and rework the controls and user interface to be likewise more player friendly. I know that oh, pretty much everyone who ever played their way through the space simulation section would much appreciate mouse control for aiming at enemy vessels. :)

I kind of consider the RPG series of Ultima, Might & Magic, and Wizardry the triumvirate of major RPGs from the last century and, now that ScummVM is embracing RPGs in addition to adventures, I'd love to see them all under the ScummVM banner. Now that at least some of the earlier Might & Magic games are supported, it may well be time to start in on Ultima support :) Of course, I'm not entirely abandoning working on adventure games. I'd previously made some promising initial progress in reversing some of the Legend Entertainment games.. Companions of Xanth and Gateway. Irrespective of how much time I end up devoting to working on Ultima I, I may end up also devoting some of my time to those games as well.

Anyway, that's probably enough for now. I still want to get in some Pillars of Eternity 2 before it's time for bed. :)





Saturday, April 14, 2018

Working my way through Dark Side

I've spent the last week with an in progress working my way through Dark Side of Xeen. As of last night, I've mapped out the bulk of the southern side of the map up to the mountains, released the first stage of Queen Kallindra's castle, finished the Temple of Bark, and am just winding up clearing Sandcaster and it's sewers of enemies. I've identified and fixed multiple problems as I went, and not all of them were specific to Dark Side.

For example, one bug I'd been meaning to fix for a while was finally resolved - when monsters were triggered to hate & attack the entire party at the same time, they were originally only ever attacking the first party member. This made the Dragon Caves on the Clouds side *so* much easier.. all the dragons would wail away on my first character, even after he was dead, leaving the rest of my party to kill them at their leisure. By the time I left the dragon caves, my first party member was below -5000HP :) Another, ironically, coincidental script problem meant the dragon containers didn't tax me when I walked past them, so I didn't even have to worry about losing gold and gems.

I also fixed some bugs in the sound system that could cause crashes when multiple sound effects were played in rapid succession, and a bug in the combat system that could case a crash when recalculating the order that characters and monsters fight in based on their speed. The later I'm particularly hoping will help resolve the outstanding rare crashes I was previously getting with extended combat. It may be naive.. I'll have to wait and see whether any further crashes occur.

So yeh, things are progressing, and the engine is getting more and more stable; I'll keep working my way through the game. I've got a long weekend coming up tomorrow, but honestly, I'm not sure how much time over the weekend I'll end up playing the game. I think I'll keep the bulk of it for some R&R and doing other things. We'll see how it goes.

Tuesday, April 10, 2018

Clouds of Xeen is now completable

Good news on the progress of the Xeen engine in ScummVM. As of the weekend, I finally finished my first playthrough of Clouds of Xeen! I finally got to give smackdown on Lord Xeen, and watch the ending cutscene. Which I've admittedly previously seen multiple times when I was first implementing the code, but there was the emotional satisfaction of seeing it properly, rather than just using the mirror cheats. Though I will confess that I got so overeager to finally finish the game, that I lacked the patience to do it properly (with well/fountain stat raises), and instead used my debugger cheats instead. But it doesn't take away from the fact that the first game is confirmed as completable. \o/

Now, it's onto Dark Side. There were actually more bugs due to changes done for Dark Side (code I'd implemented but not yet tested) than I anticipated. I already fixed multiple different bugs on the weekend that were causing hangs and crashes. Currently I'm looking into a somewhat complicated problem with how wall decorations are handled.. it causes a crash that makes Ellinger's Tower impossible to complete. So Dark Side definitely isn't completable, and needs further testing. On the plus side, though, it shouldn't take as long to test and finish Dark Side than my Clouds testing took, since Dark Side benefits from all the fixes I've already done for Clouds.

At this rate, my initial play through & tests of both Dark Side and Swords should be done within the next month, and  the game will likely be stable enough for a public testing period. Not long now :)



Tuesday, April 3, 2018

Work on Xeen is up in the air

As tempting as it would be to write this post yesterday on April Fools, I've left it to today to make a status update on my playthrough through Xeen. Work has been proceeding well. I've fixed lots of bugs that were identified by both myself and Dark-Star, who's been assisting me with preliminary testing. As of yesterday, I've:
- Finished the Dwarf Mines
- Mostly finished Rivercity (except for the Knights behind the training area)
- Completed the Witch Tower. Hence the post title.. the levitate spell works, and I've been walking around on the clouds :)

Many of the bugs were crashes and hangs, so it highlights that the game still isn't ready for general public testing. The Witch Tower, for example, required two fixes to get it to work.. one where I wasn't setting whether the party had the key (so it was never letting you in), and one where I incorrectly implemented the "return from subroutine" script opcode, so trying to go back into the Witch Tower from the stairs in the Witch Tower clouds hung the game. Ironic that both ways into the tower were glitched. :)

I think combat is also more settled now. Previously, there were various crashes, and none of the combat spells worked. Now, though, I'm able to properly cast them, so the monsters around Rivercity are much more manageable to defeat. Though I did still experience a problem once where the party wasn't immediately registered as dead after all being killed, so obviously still needs a bit of polishing to fix such remaining issues. Speaking of spells, I decided it was acceptable, given the circumstances, to cheat to give myself extra funds to purchases all the spells for each character. There was little point in delaying matters, trying to accumulate wealth. It also meant I was able to buy pathfinding and mountaineering skills for my party so I wouldn't be constrained where I could travel. It already proved useful, as I was able to identify and fix several bugs with the close-up rendering of forests, where they weren't properly being scaled, and were drawing outside of the scene area.

From here, I'll keep on working my way through the game, fixing bugs as I go. Given I'm already significantly into the game (at least for Clouds), I don't anticipate there'll be any major impediments to my progress. Apart from tracking down remaining issues with combat, hopefully the rest of the game will just be fixing minor problems, like that which prevented me from entering the Witch Tower.