Tuesday, January 19, 2021

A Great Surprise

 I only realized with the turning of the new year that I have characteristically, once again, fallen behind on posting interesting news in my blog. My bad. As many of you will already know, last year I got sucked into working on the Comprehend series of games as a sub-engine under Glk. They were an interesting early series of combination graphics and text entry games including Transylvania, Crimson Crown, and OO-Topos.

As has become a bit of a custom over the last few years, over the Christmas holidays I scouted around for an existing source code project to work on integrating into ScummVM, as somewhat of a break from the various disassembly-based work I frequently do throughout the year. And behold the result of my work:

See here for a Youtube link of a screen capture I did for the game starting up. The Black Cauldron Remake is a point and click remake of the early Sierra game of the same name which I did over a dozen years ago, before I started working in earnest on the ScummVM project. Having it supported in ScummVM will be satisfying.. harkening back to my earliest days of open source work, and bringing it all full circle. And the important part of the remake is that it was done in AGS.

That's right. There's now an AGS engine for ScummVM that's at least capable of playing one AGS game through in it's entirety. Late last year I broached the subject of ScummVM integration on the AGS Forums, and the response was positive. AGS is currently in transition between 3.5 and a major redesign for 4.0, so there wasn't a rapidly moving codebase to try and keep up with. This engine, like the standalone interpreter it's based on, should eventually support all the AGS games from 2.5 to 3.*. Luckily, this covers the majority of the popular AGS games The ScummVM implementation will be just another implementation of AGS, so it'll be an alternative to the standalone interpreter, rather than a replacement.

The engine isn't yet available on Buildbot, so don't yet go rushing to grab a daily build just yet. I plan to do a pull request shortly, so it'll likely be merged into master in a few weeks once it's approved. There's still a lot of cleanup work to be done, bugfixing for other games, etc. so work on the engine will be ongoing. I added detection already for a few other games, but they crash on startup due to other unimplemented methods.. Rather than importing Allegro and other libraries used by the AGS interpreter in their entirety, I created a mockup layer that either remaps Allegro calls to ScummVM equivalents, or implements a bare-bones equivalent for the project from scratch. But so far I've only focused on what Black Cauldron needed, so other parts will be implemented as I gradually test other games.

On a side note, for all the AGS games to work on non-Windows/Linux/Mac systems, all the various plugins games used will also need to be re-implemented in C++.. I can't just have ScummVM loading in DLLs like the original interpreter does. Thankfully, I've already been able to obtain the original source for the AgsCreditz plugin from it's author,  Hopefully other plugin authors will be likewise generous when I get to them. When done, this'll mean AGS games will be playable on more systems that AGS wasn't previously available for.

One day in the near future, expect to play some of the classic AGS games like the AGDI Sierra remakes in ScummVM :)

Friday, May 29, 2020

A retrospective and a look forward

Hey everyone. I hope everyone has done okay so far this year, considering the circumstances. On a more positive note, you will have seen we've recently announced the testing period for three of the games in the Ultima series of RPGs.. Ultima IV: Quest of the Avatar, Ultima VI: The False Prophet, and Ultima VIII: Pagan. Thanks go to the original projects and all the developers that previously put work into them, as well as in particular to mduggan, who has put a lot of work into Ultima VIII since it was merged in.

So now that we've got three of the Ultima series merged in, are more to follow? That's actually a bit of a tricky question. Of the remaining games
* Ultima IX: This is more in scope for ResidualVM than ScummVM. I remember reading previously that the "Ultima Source Code Offline Archive Project" had obtained the code for Ultima IX. So there may be a future possibility of it being reimplemented in ResidualVM.
* Ultima VII/SE: Exult is currently an independent project with active development and users. So there's no immediate plans for integration until and unless the developers decide they don't plan to do any further development, or decide themselves the time is right to merge it into ScummVM
* Underworlds: I'm vaguely aware that there are projects for the two Underworld games, but I haven't researched how complete they are. Despite their 3d look and feel, these games were done in the DOS days where all 3D was done in software. Because of this, unless the reimplemented code specifically uses 3D APIs, then it would actually be in scope for ScummVM rather than ResidualVM.
* Ultima I: I have previously been working on a reimplementation of Ultima 1. It's currently on hiatus, but I eventually plan to return to work on it. As with Ultima IV, I hope to allow for an enhanced version of the game that uses better tilesets
* Escape from Mt Drash: There is source for a PC reimplementation available here. So support for it could be added. Even if the only real relevance to the Ultima series was a marketing ploy, it is still technically an Ultima game, so support for it should be added I guess for completeness if nothing else.
* The rest. This is where it gets difficult. Apart from Ultima V, the earlier Ultima games were, honestly, less sophisticated, and I'd be less likely to enjoy playing them. So without pre-existing projects for them, it becomes harder for me to justify spending all the time reverse engineering them from scratch for support. It's even more frustrating in that it looks like there's been previous people that reverse engineered the earlier games, such as ultima2.voyd.net. But since they never released source code for their work, it really doesn't help us that much. If any of those authors happen to read this, or anyone knows how to get in touch with them, we'd really appreciate getting our hands on their work. Otherwise, it's unlikely the earlier Ultima games will be supported any time soon. Or if the previously mentioned Offline Archive Project has source for any of the earlier games, we'd welcome getting access to them.

So in any case, what are my plans now that those three games are working their way towards being officially supported? Well, first of all, I kind of fell into working on integrating the Recomprehend engine into ScummVM. Given the simplicity of the engine, it seems like a fun project to unwind with after all the work on the Ultima games. I don't anticipate it'll take all that long. There are apparently some minor issues in the screen rendering that may require reverse engineering bits of the original games for comparison, but again due to the simplicity of the games that shouldn't be too hard.

Next after that, I kind of want to get Glk into a more officially completed state. Technically all the sub-engines in it so far could be considered complete on their own, but the exception is the Frotz interpreter.. the current one didn't properly support all the version 6 graphical games. I would really like to support them properly. So I've tentatively started working on the newest version of Frotz to convert it to run on Glk from scratch. At this point I'm not sure if this new work will be able to be cleanly set up to handle both v6 and non-v6 games.. it may end up being easier to keep the existing version for non-v6, and just focus on v6 games for this. I'll have to wait and see.

After from that, Glk should be suitable for an official release. Though there are other sub-engines I could work on in the future, such as TADS support. The TADS 2 version in the codebase doesn't currently work properly, and when I previously tried starting on converting TADS 3 to compile under ScummVM, it led me down a rabbit hole of compilation issues. Support for both will likely be on hiatus for work again in a future release.

Apart from them, what else this year? Good question.. I had had some tentative plans to look into early AGS games support.. years ago I did a remake of Black Cauldron in AGS, and it'd be very satisfying to finally support it in ScummVM. However, just the other day Strangerke made an intriguing discovery. A non-interactive demo for Shannara, one of the later Legend Entertainment games, has what looks like debug information at the end of the executable (though it's not in a known format IDA can automatically handle). As long time readers will know, supporting the Legend Entertainment games has been one of my long term goals for years. Whilst the demo itself is effectively just a slide show, the debug information makes it look like they implemented the demo using large chunks of the full game's engine for rendering the demo scenes. In fact, there may even be a bunch of code from the main game the demo doesn't even use that wasn't optimized out. As such, it may prove a valuable insight for understanding the full Shannara game, and from that, the earlier games of Companions of Xanth and Death Gate.

So it may well be that I'll end up switching my focus back to the Legend games for a while. We'll have to wait and see. Considering that I ended up working on Comprehend by chance, there's always a possibility that something else interesting may pop up :)

Monday, February 10, 2020

It's ScummVM ULTIMAte

A little over a week ago, it finally happened. I had combined my various work on the Ultima games into a single branch, and it was merged into master. This means Ultima VI: The False Prophet (from Nuvie), and Ultima VIII: Pagan (from Pentagram) are now playable in daily builds.  \o/. Ultima 1, as before, is startable, but still a work in progress, and work on that is on hiatus right now. The Ultima8 sub-engine already has a splash-screen, provided by Dominus Dragon, added to give credit to the Pentagram team that wrote the original code:

Nuvie will similarly get it's own splash image eventually to recognise it as well.

However, I'll hasten to say, that any Ultima fans should not necessary come running to play either game just yet. I'm still not quite ready to announce an official testing period for either game just yet. I plan to be doing my own testing of the games first, and it's always possible that there might be minor issues or changes to the savegame formats. So any experimenters be warned to play at your own risk.

I'd originally intended to move straight onto testing of Ultima VIII post-merge, but I got slightly sidetracked adding new functionality to the base Engine class that all game engines derive from. Functionality that future game engines will benefit from, including the Ultima engine.

1) The first of these is extended savegames. There was a recently added enhancement to savegames to take care of the drudgework of storing the savegames' names, thumbnails, total play time, etc. However, it still required engine writers to manually call various methods manually in their savegame code. With the addition of new methods in the Engine class that builds on this, this is all now taken care of automatically now, as well as opening and closing savegame files in general. All that needs to be done in future engines is overriding of two methods: loadGameStream and saveGameStream, which get passed a handle to a savefile for reading/writing it's contents, and the Engine takes care of everything else, including the savegame description and thumbnail.

2) The second is improved handling for autosaves, still under code review before inclusion. It adds autosave support to almost all the game engines. ScummVM has a global option for an Autosave interval, but it previously required each engine to implement their own code for autosaves. Because of this, only about 1/5th of the engines actually implemented autosave. When these changes are integrated autosaves, when turned on, are provided by practically all game engines. Only a few engines, like DreamWeb, that don't allow saving the from Global Main Menu won't support autosaves.

3) The third one, also under review, is an improvement to engine debuggers, the console window that pops up in many engines when you do Ctrl-D. Engines now just have to create their own debugger and pass it to the Engine to take care of, and it handles all the work for opening it when necessary, and updating it. One of the benefits of this, apart from standardization & centralization, is that when errors occur now, the debugger window is more guaranteed to open up to show it. Even if the engine doesn't have it's own debugger, the base Engine class will create one on the fly. This means it will be easier for users to see fatal errors, rather than ScummVM simply terminating.

With these three done, I think my "base Engine binge" is fully tapped out. Barring any further issues or suggestions coming up in the pull requests for points 2 and 3, I'll probably take a few days to relax. Maybe do some further messing around with cleaning up Starship Titanic code, before I move back to my original plan of doing a test play-through of Ultima VIII.

Tuesday, January 7, 2020

Post New Years report

Once again, it seems I fell out of habit of doing regular blog updates. Though hopefully you'll at least be ameliorated by the fact that I have at least been busy over the last few months. With the start of 2020 finally upon us, I thought it would be an opportune time to summarize what I've been up to.

First of all, the ScummVM Glk engine has been going well. Before I put it on hiatus late last year, I'd already converted over 2/3rds of the existing engines to run under ScummVM, with cleanups and hooking into the ScummVM save system, etc. At this point, there's only a few sub-engines from the original Gargoyle suite left, chief among them TADS which, due to it's complexity, will likely take some time to clean up so that it compiles properly as part of ScummVM. I had previously done a preliminary conversion of TADS 2, but there were problems with immediate crashes that I couldn't figure out. I'm hopeful that doing a fresh conversion of both TADS 2 and 3 at the same time may result in a better outcome this time.

Next comes my work on Ultima. Back at the start of December, I'd had plans to keep working on Glk right up to my Christmas holidays. However, after a non-serious prodding from Dominus Dragon about Pentagram (the engine for playing Ultima 8) still not yet being supported, I started looking at what it would take to convert over the engine. And ended up getting hooked on it, to the exclusion of all else for the next several weeks. So below, I present a screenshot of Ultima 8 running in ScummVM:

As of Christmas, the only area I had left to update/correct was that of the music player.. I still need to figure out how to hook it into ScummVM. Otherwise, everything else seems to be working fine.. I was able to play the introduction, get through the docks cutscene, and then enter the city. I haven't yet really put the game through it's paces beyond that. It's better to wait until I'm happy that everything is properly implemented.

And I got distracted yet again, although this time intentionally. At this point, my Christmas holidays had been reached, and I really wanted something a bit more fun to work on than just bugfixing  and playtesting of Ultima 8. Trying to work my way through all those caves and jump platforms wasn't my first choice for my R&R activities.. I much preferred working on the engine itself than actually playing the game.

Back when I started getting hooked on working on Ultima 8, and realized how fun it was, I again set my sights on Nuvie, the engine for Ultima 6 and spinoffs. I'd previously asked the lead developer, Eric Fry, nearly two years back about integrating Nuvie into ScummVM, but at that time he still wanted to work further on Martian Dreams on it with it still being the familiar stand alone project. Trying again in December, I got no response, even after several weeks, and asking a co-developer, again Dominus Dragon, to also try contacting him. Since it had also been several months since any commits had been done, I decided that I'd done my due diligence, and it was finally fair game.

So over the holidays I put Ultima 8 aside, and worked on integrating Nuvie into ScummVM as well, creating a shared "ultima" engine that contains both Ultima 6 and Ultima 8. And, eventually, hopefully the other remaining Ultima games as well. Including my in-progress reimplementation of Ultima 1, which I originally put on hiatus way back when I first got agreement to merge Gargoyle into ScummVM.

So over the holidays I had further fun of converting the Nuvie engine. To make things easy for users, the new ultima6 subengine has two different detection entries, with "original":

 and "enhanced" versions:

As you can see, original gives the look and feel of the original game, whereas the Enhanced version automatically sets all the Nuvie options for enhanced graphics, full screen map, and Ultima 7 style gumps. So users will easily be able to play in an enhanced mode without worrying about fiddling around with configuration options.

As of now, the game starts up and runs. It still needs some work testing the various areas, and making sure it all works properly. In it's case, the sound code was a lot easier to convert than for Pentagram. It actually already used copies of the ScummVM mixer and stream classes. So it was a pretty straightforward process to simply remove them, and remap the code to use the original ScummVM classes instead. Such a relief :).

A downside, though, is parts of the engine, such as the cutscenes, were written in Lua rather than C code. This has disadvantages for properly supporting the game in ScummVM. First of all, I've had to duplicate the entire Lua runtime from Nuvie just to run the scripts, as the version of Lua we have in common/lua/ can't execute the scripts, giving an error when I tried. Plus, when the intro sequence is running, the system isn't properly updated, which means I can't drag the window or, more seriously, exit ScummVM by closing the Window. Long term, I'm likely going to have to rewrite the Lua scripts as C++ code and classes, so Lua can eventually be entirely removed. That will give me greater flexibility to properly handle events for the cutscenes. Also, for enhanced mode, I set the graphic mode to double size of the original, so rewriting the cutscenes in C++ would also make it easier to either stretch the cinematics, or temporarily switch back to 320x200 mode for them.

So where am I going from here? Well, the larger portion of my time is going to return to working on Glk again. Given it's advanced state, I'd be happier getting it finally done and properly supported. At least for the engines that were already part of the Gargoyle suite. Converting other non-Glk-compliant interactive fiction engines can wait until some other time in the future. I'll also spend some smaller portion of my time still working on Ultima 6. For now I'm not worrying about converting the Lua scripts.. I'll first be focusing on ensuring that the rest of the engine is running solidly under ScummVM. After that I can worry about the scripts. We'll see how I go :)

Thursday, January 24, 2019

Frotz work in ScummGlk is ongoing

Work on the ScummGlk engine is ongoing. Since Christmas holidays I've been primarily focusing on adding support for the version 6 graphical Infocom games to the Frotz sub-engine, with particular focus on Zork Zero to begin with. Whilst I don't count the version 6 games among the best of the original Infocom line-up, ScummVM has always been about providing as good a coverage of old games as possible. So from my point of view it's worth spending the extra time adding support for the remaining Infocom games now rather than later, and only moving onto the next sub-engine once that's done.

Below is a screenshot of Zork Zero as of when I came back on the 7th January:

As you can see from the screenshot, the background is starting to be properly rendered, and the 'A' and location glyphs are properly rendered and interleaved with the text. The status bar still needs some work, though.. although it is being rendered, it's then being completely wiped out by the status bar window, which isn't even displaying any text yet.

Right now, I'm focusing on adding the 8x8 fixed size font used in the original games. If you look closely at the screenshot above, the current resizable TrueType font doesn't really look crisp at that low a resolution, and is somewhat blurry. I'm currently making a tool to use the same font data that Frotz has, and produce bitmaps to add to the fonts Zip file for the different 8x8 font styles (normal, bold, emphasized, and bold & emphasized). It has had it's share of issues:
 Somehow, I don't think I've got it quite right yet.. :)

Anyway, progress is gradually being made, and I hope to have Zork Zero fully supported soon. The other v6 games currently error out immediately during startup, but I'm hoping further adding support for them will be a simpler matter.

Sunday, December 23, 2018

Seasons greeting and off on holiday

Seasons greetings to all, and I hope everyone has a happy time as we finish 2018 and look forward to the beginning of 2019. As I sit in the Los Angeles airport writing this, I'm reminded yet again of how much fun I have doing reverse engineering, since even the smallest thread of work can lead you in unexpected directions as you trace the flow of what methods call a given method, and what methods it calls.

For something to do, I opened up my Companions of Xanth in progress disassembly to do a bit of pottering around with it. Just by chance, scrolling through the disassembly, I noticed a call to a method I was pretty sure took in a string, but in this case was passing in a value. I then quickly realised it was an offset n the data segment that hadn't been properly resolved, so I did that. The message in question turned out to be a generic "insufficient memory". So from it's usage, I was able to identify the entire method it was in.. it's called during startup, and ensures that there's enough free memory to play the game. Ironically, it looks like the code to actually display the message if there isn't enough was broken, as evidenced when I ran the game in DosBox debugger.

From there, identified a method just before the message call that took in a number and passed out a string.. it wasn't too hard to identify it as a number formatting routine. I confirmed it in the DosBox debugger by putting a breakpoint in the machine code, and seeing the result from the method. I then noticed that there was one other method calling it, so I was intrigued to find out what it was. It turned out it too was a block of code in a method that printed out the free memory, in a method that looked like a big switch based on a single value. It looked like a keypress. Playing the game, I was able to confirm that pressing 'M' on the keyboard did call the code, which printed out the amount of free memory in the status area.

This was a good woo-hoo method. I had identified the method for handling keypresses. Likely, some of the other keypresses do fairly identifiable things, like saving and loading, so knowing where each key's code is, I'll be able to trace into what they do, and what methods they call. In fact, the free memory printing for the 'M' key already prove dividends. From it, I was able to identify the method that cleared the status area, and a version of 'sprintf' for printing text in the status area. In the future I'll be able to look further into other places that call the method to add lines to the status area, and start identifying other parts of the game code. For example, looking at an item produces a description in the status area, so by setting a breakpoint in the status text writing code, I'll be able to identify what method calls it, and start identifying the structure of items that likely have a "description" as one of it's fields.

All this progress today came from a minor diversion to pass the time sitting in the airport. The notice of an incorrect parameter to a method call has unraveled more of the game's secrets, and giving me multiple further areas I can investigate. A nice way to start the holidays.

Saturday, December 15, 2018

It's time for a very verbose change of pace

One of the things I like about my work on ScummVM is that it's a hobby rather than a profession, which gives me the freedom to switch what I'm working on when I desire. I've previously had encouragement to resurrect my Gargoyle engine work from about a decade ago. It was an early attempt to add support for Frotz into ScummVM, and at the time it was rejected due to multiple reasons. It was specific only for Infocom/zcode games, lacked clipboard, truetype fonts, and Unicode support. Most of these things have since been solved by the ScummVM framework itself, which now supports all of them. So I thought, why the heck not, and decided to work on it again.

However, my original Gargoyle engine being limited to just Zcode was a problem. Not only wasn't it extensible to other IF systems, but it would also take work to properly reimplement it to support Unicode. In the end, it was just simpler to scrap it all, and start from scratch. Luckily, there's the very convenient Glk specification, that was specifically created for easing porting the various Interactive Fiction interpreters to other systems. I knew that by supporting it, it would make it easier to create a framework that could support multiple different interpreters as sub-engines.

So I created a new Gargoyle engine, which I've since renamed it to ScummGlk to avoid ambiguity with the stand-alone Gargoyle project which also implements the Glk specification. After over a month of work, I've created an engine that implements the bulk of the API, and implements two sub-engines under it so far.. Scott for Scott Adams games, and Frotz for playing Z-Code/Infocom games. Scott was useful as a test case, due to it's simplicity. With it as the first sub-engine, I was able to focus on getting the ScummGlk core working. Then Frotz was a fairly obvious second choice, given it's popularity and large number of available fan games.

As of last weekend, the review period was completed for the engine, and it was merged into master. Daily builds now include support for both sub-engines. I've spend the last week fleshing out detection entries for all the ZCode games on the if-archive, so all the games will be automatically detected by the launcher. I'm going to wait on announcing an official testing period until I finish support in Frotz for V6 games. But if you can't wait, you're certainly welcome to jump the gun and try playing some games. Maybe one of the early Zork from GOG, or the immortal classic Freefall :)

Having the engine included into master will be a real nostalgia trip for me. Even prior to my original Gargoyle module, and before I even started working on implementing adventures in ScummVM, I spent several years as part of the Interactive Fiction community. Indeed, the Hitchhiker's Guide to the Galaxy was one of the first computer games I ever had. I spent way too many hours getting frustrated with the game's puzzles.