Friday, May 29, 2020

A retrospective and a look forward

Hey everyone. I hope everyone has done okay so far this year, considering the circumstances. On a more positive note, you will have seen we've recently announced the testing period for three of the games in the Ultima series of RPGs.. Ultima IV: Quest of the Avatar, Ultima VI: The False Prophet, and Ultima VIII: Pagan. Thanks go to the original projects and all the developers that previously put work into them, as well as in particular to mduggan, who has put a lot of work into Ultima VIII since it was merged in.

So now that we've got three of the Ultima series merged in, are more to follow? That's actually a bit of a tricky question. Of the remaining games
* Ultima IX: This is more in scope for ResidualVM than ScummVM. I remember reading previously that the "Ultima Source Code Offline Archive Project" had obtained the code for Ultima IX. So there may be a future possibility of it being reimplemented in ResidualVM.
* Ultima VII/SE: Exult is currently an independent project with active development and users. So there's no immediate plans for integration until and unless the developers decide they don't plan to do any further development, or decide themselves the time is right to merge it into ScummVM
* Underworlds: I'm vaguely aware that there are projects for the two Underworld games, but I haven't researched how complete they are. Despite their 3d look and feel, these games were done in the DOS days where all 3D was done in software. Because of this, unless the reimplemented code specifically uses 3D APIs, then it would actually be in scope for ScummVM rather than ResidualVM.
* Ultima I: I have previously been working on a reimplementation of Ultima 1. It's currently on hiatus, but I eventually plan to return to work on it. As with Ultima IV, I hope to allow for an enhanced version of the game that uses better tilesets
* Escape from Mt Drash: There is source for a PC reimplementation available here. So support for it could be added. Even if the only real relevance to the Ultima series was a marketing ploy, it is still technically an Ultima game, so support for it should be added I guess for completeness if nothing else.
* The rest. This is where it gets difficult. Apart from Ultima V, the earlier Ultima games were, honestly, less sophisticated, and I'd be less likely to enjoy playing them. So without pre-existing projects for them, it becomes harder for me to justify spending all the time reverse engineering them from scratch for support. It's even more frustrating in that it looks like there's been previous people that reverse engineered the earlier games, such as ultima2.voyd.net. But since they never released source code for their work, it really doesn't help us that much. If any of those authors happen to read this, or anyone knows how to get in touch with them, we'd really appreciate getting our hands on their work. Otherwise, it's unlikely the earlier Ultima games will be supported any time soon. Or if the previously mentioned Offline Archive Project has source for any of the earlier games, we'd welcome getting access to them.

So in any case, what are my plans now that those three games are working their way towards being officially supported? Well, first of all, I kind of fell into working on integrating the Recomprehend engine into ScummVM. Given the simplicity of the engine, it seems like a fun project to unwind with after all the work on the Ultima games. I don't anticipate it'll take all that long. There are apparently some minor issues in the screen rendering that may require reverse engineering bits of the original games for comparison, but again due to the simplicity of the games that shouldn't be too hard.

Next after that, I kind of want to get Glk into a more officially completed state. Technically all the sub-engines in it so far could be considered complete on their own, but the exception is the Frotz interpreter.. the current one didn't properly support all the version 6 graphical games. I would really like to support them properly. So I've tentatively started working on the newest version of Frotz to convert it to run on Glk from scratch. At this point I'm not sure if this new work will be able to be cleanly set up to handle both v6 and non-v6 games.. it may end up being easier to keep the existing version for non-v6, and just focus on v6 games for this. I'll have to wait and see.

After from that, Glk should be suitable for an official release. Though there are other sub-engines I could work on in the future, such as TADS support. The TADS 2 version in the codebase doesn't currently work properly, and when I previously tried starting on converting TADS 3 to compile under ScummVM, it led me down a rabbit hole of compilation issues. Support for both will likely be on hiatus for work again in a future release.

Apart from them, what else this year? Good question.. I had had some tentative plans to look into early AGS games support.. years ago I did a remake of Black Cauldron in AGS, and it'd be very satisfying to finally support it in ScummVM. However, just the other day Strangerke made an intriguing discovery. A non-interactive demo for Shannara, one of the later Legend Entertainment games, has what looks like debug information at the end of the executable (though it's not in a known format IDA can automatically handle). As long time readers will know, supporting the Legend Entertainment games has been one of my long term goals for years. Whilst the demo itself is effectively just a slide show, the debug information makes it look like they implemented the demo using large chunks of the full game's engine for rendering the demo scenes. In fact, there may even be a bunch of code from the main game the demo doesn't even use that wasn't optimized out. As such, it may prove a valuable insight for understanding the full Shannara game, and from that, the earlier games of Companions of Xanth and Death Gate.

So it may well be that I'll end up switching my focus back to the Legend games for a while. We'll have to wait and see. Considering that I ended up working on Comprehend by chance, there's always a possibility that something else interesting may pop up :)



Monday, February 10, 2020

It's ScummVM ULTIMAte

A little over a week ago, it finally happened. I had combined my various work on the Ultima games into a single branch, and it was merged into master. This means Ultima VI: The False Prophet (from Nuvie), and Ultima VIII: Pagan (from Pentagram) are now playable in daily builds.  \o/. Ultima 1, as before, is startable, but still a work in progress, and work on that is on hiatus right now. The Ultima8 sub-engine already has a splash-screen, provided by Dominus Dragon, added to give credit to the Pentagram team that wrote the original code:

Nuvie will similarly get it's own splash image eventually to recognise it as well.

However, I'll hasten to say, that any Ultima fans should not necessary come running to play either game just yet. I'm still not quite ready to announce an official testing period for either game just yet. I plan to be doing my own testing of the games first, and it's always possible that there might be minor issues or changes to the savegame formats. So any experimenters be warned to play at your own risk.

I'd originally intended to move straight onto testing of Ultima VIII post-merge, but I got slightly sidetracked adding new functionality to the base Engine class that all game engines derive from. Functionality that future game engines will benefit from, including the Ultima engine.

1) The first of these is extended savegames. There was a recently added enhancement to savegames to take care of the drudgework of storing the savegames' names, thumbnails, total play time, etc. However, it still required engine writers to manually call various methods manually in their savegame code. With the addition of new methods in the Engine class that builds on this, this is all now taken care of automatically now, as well as opening and closing savegame files in general. All that needs to be done in future engines is overriding of two methods: loadGameStream and saveGameStream, which get passed a handle to a savefile for reading/writing it's contents, and the Engine takes care of everything else, including the savegame description and thumbnail.

2) The second is improved handling for autosaves, still under code review before inclusion. It adds autosave support to almost all the game engines. ScummVM has a global option for an Autosave interval, but it previously required each engine to implement their own code for autosaves. Because of this, only about 1/5th of the engines actually implemented autosave. When these changes are integrated autosaves, when turned on, are provided by practically all game engines. Only a few engines, like DreamWeb, that don't allow saving the from Global Main Menu won't support autosaves.

3) The third one, also under review, is an improvement to engine debuggers, the console window that pops up in many engines when you do Ctrl-D. Engines now just have to create their own debugger and pass it to the Engine to take care of, and it handles all the work for opening it when necessary, and updating it. One of the benefits of this, apart from standardization & centralization, is that when errors occur now, the debugger window is more guaranteed to open up to show it. Even if the engine doesn't have it's own debugger, the base Engine class will create one on the fly. This means it will be easier for users to see fatal errors, rather than ScummVM simply terminating.

With these three done, I think my "base Engine binge" is fully tapped out. Barring any further issues or suggestions coming up in the pull requests for points 2 and 3, I'll probably take a few days to relax. Maybe do some further messing around with cleaning up Starship Titanic code, before I move back to my original plan of doing a test play-through of Ultima VIII.
.

Tuesday, January 7, 2020

Post New Years report

Once again, it seems I fell out of habit of doing regular blog updates. Though hopefully you'll at least be ameliorated by the fact that I have at least been busy over the last few months. With the start of 2020 finally upon us, I thought it would be an opportune time to summarize what I've been up to.

First of all, the ScummVM Glk engine has been going well. Before I put it on hiatus late last year, I'd already converted over 2/3rds of the existing engines to run under ScummVM, with cleanups and hooking into the ScummVM save system, etc. At this point, there's only a few sub-engines from the original Gargoyle suite left, chief among them TADS which, due to it's complexity, will likely take some time to clean up so that it compiles properly as part of ScummVM. I had previously done a preliminary conversion of TADS 2, but there were problems with immediate crashes that I couldn't figure out. I'm hopeful that doing a fresh conversion of both TADS 2 and 3 at the same time may result in a better outcome this time.

Next comes my work on Ultima. Back at the start of December, I'd had plans to keep working on Glk right up to my Christmas holidays. However, after a non-serious prodding from Dominus Dragon about Pentagram (the engine for playing Ultima 8) still not yet being supported, I started looking at what it would take to convert over the engine. And ended up getting hooked on it, to the exclusion of all else for the next several weeks. So below, I present a screenshot of Ultima 8 running in ScummVM:


As of Christmas, the only area I had left to update/correct was that of the music player.. I still need to figure out how to hook it into ScummVM. Otherwise, everything else seems to be working fine.. I was able to play the introduction, get through the docks cutscene, and then enter the city. I haven't yet really put the game through it's paces beyond that. It's better to wait until I'm happy that everything is properly implemented.

And I got distracted yet again, although this time intentionally. At this point, my Christmas holidays had been reached, and I really wanted something a bit more fun to work on than just bugfixing  and playtesting of Ultima 8. Trying to work my way through all those caves and jump platforms wasn't my first choice for my R&R activities.. I much preferred working on the engine itself than actually playing the game.

Back when I started getting hooked on working on Ultima 8, and realized how fun it was, I again set my sights on Nuvie, the engine for Ultima 6 and spinoffs. I'd previously asked the lead developer, Eric Fry, nearly two years back about integrating Nuvie into ScummVM, but at that time he still wanted to work further on Martian Dreams on it with it still being the familiar stand alone project. Trying again in December, I got no response, even after several weeks, and asking a co-developer, again Dominus Dragon, to also try contacting him. Since it had also been several months since any commits had been done, I decided that I'd done my due diligence, and it was finally fair game.

So over the holidays I put Ultima 8 aside, and worked on integrating Nuvie into ScummVM as well, creating a shared "ultima" engine that contains both Ultima 6 and Ultima 8. And, eventually, hopefully the other remaining Ultima games as well. Including my in-progress reimplementation of Ultima 1, which I originally put on hiatus way back when I first got agreement to merge Gargoyle into ScummVM.

So over the holidays I had further fun of converting the Nuvie engine. To make things easy for users, the new ultima6 subengine has two different detection entries, with "original":


 and "enhanced" versions:


As you can see, original gives the look and feel of the original game, whereas the Enhanced version automatically sets all the Nuvie options for enhanced graphics, full screen map, and Ultima 7 style gumps. So users will easily be able to play in an enhanced mode without worrying about fiddling around with configuration options.

As of now, the game starts up and runs. It still needs some work testing the various areas, and making sure it all works properly. In it's case, the sound code was a lot easier to convert than for Pentagram. It actually already used copies of the ScummVM mixer and stream classes. So it was a pretty straightforward process to simply remove them, and remap the code to use the original ScummVM classes instead. Such a relief :).

A downside, though, is parts of the engine, such as the cutscenes, were written in Lua rather than C code. This has disadvantages for properly supporting the game in ScummVM. First of all, I've had to duplicate the entire Lua runtime from Nuvie just to run the scripts, as the version of Lua we have in common/lua/ can't execute the scripts, giving an error when I tried. Plus, when the intro sequence is running, the system isn't properly updated, which means I can't drag the window or, more seriously, exit ScummVM by closing the Window. Long term, I'm likely going to have to rewrite the Lua scripts as C++ code and classes, so Lua can eventually be entirely removed. That will give me greater flexibility to properly handle events for the cutscenes. Also, for enhanced mode, I set the graphic mode to double size of the original, so rewriting the cutscenes in C++ would also make it easier to either stretch the cinematics, or temporarily switch back to 320x200 mode for them.


So where am I going from here? Well, the larger portion of my time is going to return to working on Glk again. Given it's advanced state, I'd be happier getting it finally done and properly supported. At least for the engines that were already part of the Gargoyle suite. Converting other non-Glk-compliant interactive fiction engines can wait until some other time in the future. I'll also spend some smaller portion of my time still working on Ultima 6. For now I'm not worrying about converting the Lua scripts.. I'll first be focusing on ensuring that the rest of the engine is running solidly under ScummVM. After that I can worry about the scripts. We'll see how I go :)

Thursday, January 24, 2019

Frotz work in ScummGlk is ongoing

Work on the ScummGlk engine is ongoing. Since Christmas holidays I've been primarily focusing on adding support for the version 6 graphical Infocom games to the Frotz sub-engine, with particular focus on Zork Zero to begin with. Whilst I don't count the version 6 games among the best of the original Infocom line-up, ScummVM has always been about providing as good a coverage of old games as possible. So from my point of view it's worth spending the extra time adding support for the remaining Infocom games now rather than later, and only moving onto the next sub-engine once that's done.

Below is a screenshot of Zork Zero as of when I came back on the 7th January:


As you can see from the screenshot, the background is starting to be properly rendered, and the 'A' and location glyphs are properly rendered and interleaved with the text. The status bar still needs some work, though.. although it is being rendered, it's then being completely wiped out by the status bar window, which isn't even displaying any text yet.

Right now, I'm focusing on adding the 8x8 fixed size font used in the original games. If you look closely at the screenshot above, the current resizable TrueType font doesn't really look crisp at that low a resolution, and is somewhat blurry. I'm currently making a tool to use the same font data that Frotz has, and produce bitmaps to add to the fonts Zip file for the different 8x8 font styles (normal, bold, emphasized, and bold & emphasized). It has had it's share of issues:
 Somehow, I don't think I've got it quite right yet.. :)

Anyway, progress is gradually being made, and I hope to have Zork Zero fully supported soon. The other v6 games currently error out immediately during startup, but I'm hoping further adding support for them will be a simpler matter.

Sunday, December 23, 2018

Seasons greeting and off on holiday

Seasons greetings to all, and I hope everyone has a happy time as we finish 2018 and look forward to the beginning of 2019. As I sit in the Los Angeles airport writing this, I'm reminded yet again of how much fun I have doing reverse engineering, since even the smallest thread of work can lead you in unexpected directions as you trace the flow of what methods call a given method, and what methods it calls.

For something to do, I opened up my Companions of Xanth in progress disassembly to do a bit of pottering around with it. Just by chance, scrolling through the disassembly, I noticed a call to a method I was pretty sure took in a string, but in this case was passing in a value. I then quickly realised it was an offset n the data segment that hadn't been properly resolved, so I did that. The message in question turned out to be a generic "insufficient memory". So from it's usage, I was able to identify the entire method it was in.. it's called during startup, and ensures that there's enough free memory to play the game. Ironically, it looks like the code to actually display the message if there isn't enough was broken, as evidenced when I ran the game in DosBox debugger.

From there, identified a method just before the message call that took in a number and passed out a string.. it wasn't too hard to identify it as a number formatting routine. I confirmed it in the DosBox debugger by putting a breakpoint in the machine code, and seeing the result from the method. I then noticed that there was one other method calling it, so I was intrigued to find out what it was. It turned out it too was a block of code in a method that printed out the free memory, in a method that looked like a big switch based on a single value. It looked like a keypress. Playing the game, I was able to confirm that pressing 'M' on the keyboard did call the code, which printed out the amount of free memory in the status area.

This was a good woo-hoo method. I had identified the method for handling keypresses. Likely, some of the other keypresses do fairly identifiable things, like saving and loading, so knowing where each key's code is, I'll be able to trace into what they do, and what methods they call. In fact, the free memory printing for the 'M' key already prove dividends. From it, I was able to identify the method that cleared the status area, and a version of 'sprintf' for printing text in the status area. In the future I'll be able to look further into other places that call the method to add lines to the status area, and start identifying other parts of the game code. For example, looking at an item produces a description in the status area, so by setting a breakpoint in the status text writing code, I'll be able to identify what method calls it, and start identifying the structure of items that likely have a "description" as one of it's fields.

All this progress today came from a minor diversion to pass the time sitting in the airport. The notice of an incorrect parameter to a method call has unraveled more of the game's secrets, and giving me multiple further areas I can investigate. A nice way to start the holidays.


Saturday, December 15, 2018

It's time for a very verbose change of pace

One of the things I like about my work on ScummVM is that it's a hobby rather than a profession, which gives me the freedom to switch what I'm working on when I desire. I've previously had encouragement to resurrect my Gargoyle engine work from about a decade ago. It was an early attempt to add support for Frotz into ScummVM, and at the time it was rejected due to multiple reasons. It was specific only for Infocom/zcode games, lacked clipboard, truetype fonts, and Unicode support. Most of these things have since been solved by the ScummVM framework itself, which now supports all of them. So I thought, why the heck not, and decided to work on it again.

However, my original Gargoyle engine being limited to just Zcode was a problem. Not only wasn't it extensible to other IF systems, but it would also take work to properly reimplement it to support Unicode. In the end, it was just simpler to scrap it all, and start from scratch. Luckily, there's the very convenient Glk specification, that was specifically created for easing porting the various Interactive Fiction interpreters to other systems. I knew that by supporting it, it would make it easier to create a framework that could support multiple different interpreters as sub-engines.

So I created a new Gargoyle engine, which I've since renamed it to ScummGlk to avoid ambiguity with the stand-alone Gargoyle project which also implements the Glk specification. After over a month of work, I've created an engine that implements the bulk of the API, and implements two sub-engines under it so far.. Scott for Scott Adams games, and Frotz for playing Z-Code/Infocom games. Scott was useful as a test case, due to it's simplicity. With it as the first sub-engine, I was able to focus on getting the ScummGlk core working. Then Frotz was a fairly obvious second choice, given it's popularity and large number of available fan games.

As of last weekend, the review period was completed for the engine, and it was merged into master. Daily builds now include support for both sub-engines. I've spend the last week fleshing out detection entries for all the ZCode games on the if-archive, so all the games will be automatically detected by the launcher. I'm going to wait on announcing an official testing period until I finish support in Frotz for V6 games. But if you can't wait, you're certainly welcome to jump the gun and try playing some games. Maybe one of the early Zork from GOG, or the immortal classic Freefall :)

Having the engine included into master will be a real nostalgia trip for me. Even prior to my original Gargoyle module, and before I even started working on implementing adventures in ScummVM, I spent several years as part of the Interactive Fiction community. Indeed, the Hitchhiker's Guide to the Galaxy was one of the first computer games I ever had. I spent way too many hours getting frustrated with the game's puzzles.


Monday, May 21, 2018

It's an UltimaTE unwinding

After all this time, support for the Might & Magic Xeen games is finally close to being finished. It's currently the official testing period for all the Xeen games, and I've already had a whole bunch of useful bug reports, which I've all taken care of. At this point, it's almost certain that the Xeen engine, and games, will be supported in the next official ScummVM release. It's a nice feeling. More than any other game engine I've worked on, the Xeen engine was the one that got put on hiatus the most number of times. It's very satisfying to finally complete it.

What's in store for me now? Well, I do want to spend some time properly unwinding and actually playing some games. They've only just come out with the sequel to Pillars of Eternity, which I'm now getting into, and plus I have a backlog of other RPG and adventure games that I want to have a chance to properly play. So that will probably keep me occupied for a few months at least. I've also got a move from the US East Coast to West Coast coming up next month. So various preparations are consuming my time, and it's all to the good that work on Xeen has already finished.

Not that I'm totally stopping work on ScummVM stuff. As with the early days of work on Starship Titanic, I started doing some casual reverse engineering during my lunch breaks. By happenstance, I experimented with the Ultima 1 executables, and become enamored by how simple they were to reverse. Both due to the simplicity of the game (considering how old it was), and the way the code was implemented. It was just the thing for doing casual reversing work in chunks of an hour at a time.

As a result of this, I've already implemented a great deal of foundation code in ScummVM for supporting the game, with a long term eye for having a clean engine structure that could support all the early Ultima games.  Here's some screenshots of the player within the overworld, a city, and within the dungeons:



At this point, I've got basic movement functionality working. The player can roam around the map, entering cities, castles, and dungeons. Dungeon movement also works, as does going up and down ladders, so the dungeon can be further ventured down into, or left entirely. That's it at the moment, though; it doesn't yet have any combat, magic, inventory, character interaction, etc. See this Youtube video for an example walk-about that covers venturing into a dungeon, and then heading over to the city of Britian.

Don't expect too much more work on this in the near term, though. As I said, this is more a side thing, and I definitely do want to spend some time unwinding and playing some games. When I get my fill, later in the year, then we'll see. At the moment, it's looking strongly like I'll keep working further on this. I've some crazy ideas about not only adding support for the game, but also reusing the Ultima VI tileset to create an "enhanced" Ultima 1 experience that's more visually pleasing, and rework the controls and user interface to be likewise more player friendly. I know that oh, pretty much everyone who ever played their way through the space simulation section would much appreciate mouse control for aiming at enemy vessels. :)

I kind of consider the RPG series of Ultima, Might & Magic, and Wizardry the triumvirate of major RPGs from the last century and, now that ScummVM is embracing RPGs in addition to adventures, I'd love to see them all under the ScummVM banner. Now that at least some of the earlier Might & Magic games are supported, it may well be time to start in on Ultima support :) Of course, I'm not entirely abandoning working on adventure games. I'd previously made some promising initial progress in reversing some of the Legend Entertainment games.. Companions of Xanth and Gateway. Irrespective of how much time I end up devoting to working on Ultima I, I may end up also devoting some of my time to those games as well.

Anyway, that's probably enough for now. I still want to get in some Pillars of Eternity 2 before it's time for bed. :)