Christmas 2023 holiday roundup

 Seems like it's been a full year since my last blog posting. I think that at this point, I'll just have to accept that I'm not very good at making regular postings. At least you all can take solace that my output of work on implementing support for new games in ScummVM remains unaffected. :)

So, what's been happening in the last year? Well, first of all, I finished implementing support for Might and Magic 1. The game itself was pretty primitive, but it was an excellent candidate for something I'd wanted to do for a while - not just reimplementing a game, but adding an enhanced mode that would breathe new life into it. In this case, I was able to use my experience with working on the Xeen games some years previously to add it's user interface into M&M1 as an option. I think it turned out pretty nicely:

After that, I decided to revisit another game that had been bit-rotting, in this case for nearly twenty years - the M4 engine, which supports the games Orion Burger and Ripley's Believe It Or Not - The Riddle of Master Lu. This engine had special meaning to me, as it was the first engine I worked on collaboratively with others, shortly after I first joined the group. Back then though, things got mixed up with trying to have the one engine support both those games as well as the earlier MADS games like Rex Nebular in a single engine, and work stalled on it. Since that time, I'd re-implemented the MADS engine separately (though currently not all MADS games are supported), so decided it was finally time to tackle the M4 engine. And, just in time for Christmas, I finally finished implementing it with Orion Burger fully playable.

I've just announced an official testing period for the game. All the logic of the game was hardcoded, and had to be manually implemented. As such, my implementation of the various game rooms' logic suffered from many minor errors that had to be identified and fixed. Hopefully this testing period will have others discover any other remaining issues I'd overlooked, and the game will finally be officially supported.

Other than that, on a final note, as with previous years I choose to muck around with something different over the Christmas holidays. In this case, rather than fall back on working on the early Legend games yet again, I decided on the Wasteland engine, in particular disassembling Fountain of Dreams. I'd done some previous examination, and it looked like the codebase was fairly straightforward and simple (at least for the small parts of the executables I investigated). My time was productive, and I was not only able to reverse engineer and reimplement the entirety of the character creation screens, but also get a basic map display going. I give you, Fountain of Dreams startup in ScummVM:

ScummVM Fountain of Dreams

As you'll see from the attached YouTube video, I even added basic mouse control support for the character creation screens, just for the heck of it. At the moment, what's shown is pretty much all there is, but it's a firm starting point that further work can be done on.

Will further work be done? At this point I'm not entirely sure. I haven't firmly committed to working on Fountain of Dreams and/or Wasteland. It may be that I'll just putter around with them between working on other games. After all, I still have the drudgery of implementing all the game scenes' logic for Riddle of Master Lu to look forward to :P.. maybe I can use working on this as a way to unwind as I work my way through it all.

Christmas holidays roundup

 Welcome to 2023, everyone. It's now mid-month in January, and I'm back from my long overdue Christmas holidays. Unfortunately :). Whilst I did end up having to spend some of my time taking care of work, I still got significant time to rest, read some novels and, of course, muck around with ScummVM stuff :). As with several previous years, I mostly ended up fiddling around with my disassembly of the early Legend games, and trying to re-implement them in ScummVM.

Thankfully, I was finally able to make some proper progress this year. Working further on my disassembly of the first Gateway game, I started a new iteration of the Legend engine in ScummVM using the latest meta engine/detection code, and gradually pulled in the work I'd previously done in the prior version of the engine. Then using it as a basis, I started implementing the code for the main text window. This included all the logic for caching text, line wrapping, and the -MORE- indicator. When it was done, I finally had the basis for displaying game messages.

Once that was done, for the first time, I started looking at the actual engine implementation. And I made progress, stumbling on a set of methods that implement what seemed like a basic C++ class hierarchy in C, with a table of 734 "objects" that were each one of eight different categories/types. Each type having a given amount of currently unknown data values. The only field of these structures that is currently known is a "method index" into a table of functions. Whilst the bulk of objects have a unique function, there are a few that don't have one, or share the same method. This is also how I know the code was a manual implementation of classes, since the function for getting the method index is basically a big switch statement that reads the appropriate offset, which is different for each category. If it was proper C++, they'd have a common base class and the field would be the same for all of them.

And these methods being pointed to.. what are they? It seems to be nothing less than all the per-game custom logic. With a bit of experimentation, I found that the first category was for rooms. Changing rooms in game is basically a matter of changing the room object being pointed to, and executing some code in the room for entering it. There's a global "room object num" that has the starting room by default, but by changing the value during startup in the DosBox debugger, I was able to change the room I started in.

Funnily enough, I tried changing the "room" to one of the other category types, and realized that it's type was for items. It seems the core engine has some resiliency, since by setting it to the item, the game told me that I was now "in" the item, as if it were a room :). Presumably, the other categories may represent other types of game entries, like non-inventory scenery or room exits, and their internal functions will have the logic for looking at items, moving to new rooms, etc.

This is all a big deal, since it means that when I have time free to work on it, I now have a solid starting point for figuring out game logic, and the methods they call in the engine to do the various things. Once I spend a bit more time figuring out the sentence parser, I'll be able to rapidly progress in finally supporting the Gateway game and, likely in the future, other early Legend games as well. Though it's going to be a pain having to manually implement and test each and every object function in each game. It's a good thing I have the patience to do that kind of thing. Though come to it, if I figure out all the engine methods the game calls, maybe Ghidra can assist in quickly producing readable C code.

As a final treat, a screenshot of the current engine. The listbox, font drawing, and other elements were pulled in from work done in prior years, and they still need work. Mostly, I focused on the text area, and better understanding how the game works:

Seasons Greetings

 With the end of the year upon us, I thought it was time to do a blog post. And then I double-checked, and realized that I hadn't actually done a blog post since the start of last year. Awkward! So, yes, it's definitely well past time that I did a new post :) Things have obviously moved on since the last time I posted. In that time, the AGS engine has been merged into master, and is now officially part of ScummVM. I also spent some finishing the game Chewy Esc From F5, and it now too is finally part of the official releases.

My passion has always been reverse engineering, and due to certain real world considerations, I wanted something simple to work on for the last six months. I ended up choosing Might and Magic 1 for several reasons. I had already done some minor initial work on it, and found the disassembly areas I'd already looked at fairly straightforward. Plus, we already supported the Xeen games. I figured that if M&M1 were supported, M&M2, which I think is closely based on 1, would also be easy to support in the future. Plus, Xeen was closely based on M&M3, so it would be easier to support that as well. So we could eventually end up supporting all the Xeen games 1 through 5. And since I know there's projects implementing M&M6 and later games, maybe one day they could be merged in, and ScummVM could support the entire series. And then maybe onto Wizardry and Final Fantasy :)

.Plus, given my familiarity with the Xeen engine, I was very interested in the possibility of introducing an "enhanced" mode to M&M1 that brought the user interface of the Xeen games to the primitive user interface of M&M1. With cleaner visuals for the various dialogs and niceties like the minimap, it could encourage people who never played the original to finally try out the game that started the entire series.

So, how is my work on the game progressing? As of today, I'm finally closing in on finishing re-implementing the original. All the user interface and game logic is done and tested, and all that's left is debugging the combat subsystem. Just last night I finished fixing the code relating to party members attacking the monsters, and now I need to start working on the code for when monsters attack the party. Apart from that there'll just be general testing.

Though there are a few other limitations remaining. I wasn't able to figure out the PC speaker code it uses for sound, so the game will be soundless. At least for the original un-enhanced version. Since the original only had sound in a few places, like when bumping into a wall, or a few notes when a party member is killed in combat, I don't consider that as a major loss.  I was also never able to properly figure out entirely how the original did image decoding and handled the EGA palette. The engine has it's own decoding that I figured out from trial and error, but it does mean that the monsters appear with the wrong colours. Unless I can figure out exactly how the original did the image rendering, I may simply have to use one of the online wikis to see the colors of each monster, and set them up in ScummVM to be correct..

Anyway, I'm hoping to at least get the remaining bulk of combat debugged before my Christmas holidays, and then worry about playing around with doing an enhanced version in the new year. As has been my habit in previous years, I'll likely find something else to do during my holidays. I'm kind of thinking to return to work on the Legend engine again. That seems to be my default fallback project. And who knows, one of these days I may even actually finish re-implementing one of the games :)

So seasons greetings to everyone, and I hope both myself and you all will have a good 2023 to look forward to.

A Great Surprise

 I only realized with the turning of the new year that I have characteristically, once again, fallen behind on posting interesting news in my blog. My bad. As many of you will already know, last year I got sucked into working on the Comprehend series of games as a sub-engine under Glk. They were an interesting early series of combination graphics and text entry games including Transylvania, Crimson Crown, and OO-Topos.

As has become a bit of a custom over the last few years, over the Christmas holidays I scouted around for an existing source code project to work on integrating into ScummVM, as somewhat of a break from the various disassembly-based work I frequently do throughout the year. And behold the result of my work:

See here for a Youtube link of a screen capture I did for the game starting up. The Black Cauldron Remake is a point and click remake of the early Sierra game of the same name which I did over a dozen years ago, before I started working in earnest on the ScummVM project. Having it supported in ScummVM will be satisfying.. harkening back to my earliest days of open source work, and bringing it all full circle. And the important part of the remake is that it was done in AGS.

That's right. There's now an AGS engine for ScummVM that's at least capable of playing one AGS game through in it's entirety. Late last year I broached the subject of ScummVM integration on the AGS Forums, and the response was positive. AGS is currently in transition between 3.5 and a major redesign for 4.0, so there wasn't a rapidly moving codebase to try and keep up with. This engine, like the standalone interpreter it's based on, should eventually support all the AGS games from 2.5 to 3.*. Luckily, this covers the majority of the popular AGS games The ScummVM implementation will be just another implementation of AGS, so it'll be an alternative to the standalone interpreter, rather than a replacement.

The engine isn't yet available on Buildbot, so don't yet go rushing to grab a daily build just yet. I plan to do a pull request shortly, so it'll likely be merged into master in a few weeks once it's approved. There's still a lot of cleanup work to be done, bugfixing for other games, etc. so work on the engine will be ongoing. I added detection already for a few other games, but they crash on startup due to other unimplemented methods.. Rather than importing Allegro and other libraries used by the AGS interpreter in their entirety, I created a mockup layer that either remaps Allegro calls to ScummVM equivalents, or implements a bare-bones equivalent for the project from scratch. But so far I've only focused on what Black Cauldron needed, so other parts will be implemented as I gradually test other games.

On a side note, for all the AGS games to work on non-Windows/Linux/Mac systems, all the various plugins games used will also need to be re-implemented in C++.. I can't just have ScummVM loading in DLLs like the original interpreter does. Thankfully, I've already been able to obtain the original source for the AgsCreditz plugin from it's author,  Hopefully other plugin authors will be likewise generous when I get to them. When done, this'll mean AGS games will be playable on more systems that AGS wasn't previously available for.

One day in the near future, expect to play some of the classic AGS games like the AGDI Sierra remakes in ScummVM :)

A retrospective and a look forward

Hey everyone. I hope everyone has done okay so far this year, considering the circumstances. On a more positive note, you will have seen we've recently announced the testing period for three of the games in the Ultima series of RPGs.. Ultima IV: Quest of the Avatar, Ultima VI: The False Prophet, and Ultima VIII: Pagan. Thanks go to the original projects and all the developers that previously put work into them, as well as in particular to mduggan, who has put a lot of work into Ultima VIII since it was merged in.

So now that we've got three of the Ultima series merged in, are more to follow? That's actually a bit of a tricky question. Of the remaining games
* Ultima IX: This is more in scope for ResidualVM than ScummVM. I remember reading previously that the "Ultima Source Code Offline Archive Project" had obtained the code for Ultima IX. So there may be a future possibility of it being reimplemented in ResidualVM.
* Ultima VII/SE: Exult is currently an independent project with active development and users. So there's no immediate plans for integration until and unless the developers decide they don't plan to do any further development, or decide themselves the time is right to merge it into ScummVM
* Underworlds: I'm vaguely aware that there are projects for the two Underworld games, but I haven't researched how complete they are. Despite their 3d look and feel, these games were done in the DOS days where all 3D was done in software. Because of this, unless the reimplemented code specifically uses 3D APIs, then it would actually be in scope for ScummVM rather than ResidualVM.
* Ultima I: I have previously been working on a reimplementation of Ultima 1. It's currently on hiatus, but I eventually plan to return to work on it. As with Ultima IV, I hope to allow for an enhanced version of the game that uses better tilesets
* Escape from Mt Drash: There is source for a PC reimplementation available here. So support for it could be added. Even if the only real relevance to the Ultima series was a marketing ploy, it is still technically an Ultima game, so support for it should be added I guess for completeness if nothing else.
* The rest. This is where it gets difficult. Apart from Ultima V, the earlier Ultima games were, honestly, less sophisticated, and I'd be less likely to enjoy playing them. So without pre-existing projects for them, it becomes harder for me to justify spending all the time reverse engineering them from scratch for support. It's even more frustrating in that it looks like there's been previous people that reverse engineered the earlier games, such as ultima2.voyd.net. But since they never released source code for their work, it really doesn't help us that much. If any of those authors happen to read this, or anyone knows how to get in touch with them, we'd really appreciate getting our hands on their work. Otherwise, it's unlikely the earlier Ultima games will be supported any time soon. Or if the previously mentioned Offline Archive Project has source for any of the earlier games, we'd welcome getting access to them.

So in any case, what are my plans now that those three games are working their way towards being officially supported? Well, first of all, I kind of fell into working on integrating the Recomprehend engine into ScummVM. Given the simplicity of the engine, it seems like a fun project to unwind with after all the work on the Ultima games. I don't anticipate it'll take all that long. There are apparently some minor issues in the screen rendering that may require reverse engineering bits of the original games for comparison, but again due to the simplicity of the games that shouldn't be too hard.

Next after that, I kind of want to get Glk into a more officially completed state. Technically all the sub-engines in it so far could be considered complete on their own, but the exception is the Frotz interpreter.. the current one didn't properly support all the version 6 graphical games. I would really like to support them properly. So I've tentatively started working on the newest version of Frotz to convert it to run on Glk from scratch. At this point I'm not sure if this new work will be able to be cleanly set up to handle both v6 and non-v6 games.. it may end up being easier to keep the existing version for non-v6, and just focus on v6 games for this. I'll have to wait and see.

After from that, Glk should be suitable for an official release. Though there are other sub-engines I could work on in the future, such as TADS support. The TADS 2 version in the codebase doesn't currently work properly, and when I previously tried starting on converting TADS 3 to compile under ScummVM, it led me down a rabbit hole of compilation issues. Support for both will likely be on hiatus for work again in a future release.

Apart from them, what else this year? Good question.. I had had some tentative plans to look into early AGS games support.. years ago I did a remake of Black Cauldron in AGS, and it'd be very satisfying to finally support it in ScummVM. However, just the other day Strangerke made an intriguing discovery. A non-interactive demo for Shannara, one of the later Legend Entertainment games, has what looks like debug information at the end of the executable (though it's not in a known format IDA can automatically handle). As long time readers will know, supporting the Legend Entertainment games has been one of my long term goals for years. Whilst the demo itself is effectively just a slide show, the debug information makes it look like they implemented the demo using large chunks of the full game's engine for rendering the demo scenes. In fact, there may even be a bunch of code from the main game the demo doesn't even use that wasn't optimized out. As such, it may prove a valuable insight for understanding the full Shannara game, and from that, the earlier games of Companions of Xanth and Death Gate.

So it may well be that I'll end up switching my focus back to the Legend games for a while. We'll have to wait and see. Considering that I ended up working on Comprehend by chance, there's always a possibility that something else interesting may pop up :)

It's ScummVM ULTIMAte

A little over a week ago, it finally happened. I had combined my various work on the Ultima games into a single branch, and it was merged into master. This means Ultima VI: The False Prophet (from Nuvie), and Ultima VIII: Pagan (from Pentagram) are now playable in daily builds.  \o/. Ultima 1, as before, is startable, but still a work in progress, and work on that is on hiatus right now. The Ultima8 sub-engine already has a splash-screen, provided by Dominus Dragon, added to give credit to the Pentagram team that wrote the original code:

Nuvie will similarly get it's own splash image eventually to recognise it as well.

However, I'll hasten to say, that any Ultima fans should not necessary come running to play either game just yet. I'm still not quite ready to announce an official testing period for either game just yet. I plan to be doing my own testing of the games first, and it's always possible that there might be minor issues or changes to the savegame formats. So any experimenters be warned to play at your own risk.

I'd originally intended to move straight onto testing of Ultima VIII post-merge, but I got slightly sidetracked adding new functionality to the base Engine class that all game engines derive from. Functionality that future game engines will benefit from, including the Ultima engine.

1) The first of these is extended savegames. There was a recently added enhancement to savegames to take care of the drudgework of storing the savegames' names, thumbnails, total play time, etc. However, it still required engine writers to manually call various methods manually in their savegame code. With the addition of new methods in the Engine class that builds on this, this is all now taken care of automatically now, as well as opening and closing savegame files in general. All that needs to be done in future engines is overriding of two methods: loadGameStream and saveGameStream, which get passed a handle to a savefile for reading/writing it's contents, and the Engine takes care of everything else, including the savegame description and thumbnail.

2) The second is improved handling for autosaves, still under code review before inclusion. It adds autosave support to almost all the game engines. ScummVM has a global option for an Autosave interval, but it previously required each engine to implement their own code for autosaves. Because of this, only about 1/5th of the engines actually implemented autosave. When these changes are integrated autosaves, when turned on, are provided by practically all game engines. Only a few engines, like DreamWeb, that don't allow saving the from Global Main Menu won't support autosaves.

3) The third one, also under review, is an improvement to engine debuggers, the console window that pops up in many engines when you do Ctrl-D. Engines now just have to create their own debugger and pass it to the Engine to take care of, and it handles all the work for opening it when necessary, and updating it. One of the benefits of this, apart from standardization & centralization, is that when errors occur now, the debugger window is more guaranteed to open up to show it. Even if the engine doesn't have it's own debugger, the base Engine class will create one on the fly. This means it will be easier for users to see fatal errors, rather than ScummVM simply terminating.

With these three done, I think my "base Engine binge" is fully tapped out. Barring any further issues or suggestions coming up in the pull requests for points 2 and 3, I'll probably take a few days to relax. Maybe do some further messing around with cleaning up Starship Titanic code, before I move back to my original plan of doing a test play-through of Ultima VIII.
.

Post New Years report

Once again, it seems I fell out of habit of doing regular blog updates. Though hopefully you'll at least be ameliorated by the fact that I have at least been busy over the last few months. With the start of 2020 finally upon us, I thought it would be an opportune time to summarize what I've been up to.

First of all, the ScummVM Glk engine has been going well. Before I put it on hiatus late last year, I'd already converted over 2/3rds of the existing engines to run under ScummVM, with cleanups and hooking into the ScummVM save system, etc. At this point, there's only a few sub-engines from the original Gargoyle suite left, chief among them TADS which, due to it's complexity, will likely take some time to clean up so that it compiles properly as part of ScummVM. I had previously done a preliminary conversion of TADS 2, but there were problems with immediate crashes that I couldn't figure out. I'm hopeful that doing a fresh conversion of both TADS 2 and 3 at the same time may result in a better outcome this time.

Next comes my work on Ultima. Back at the start of December, I'd had plans to keep working on Glk right up to my Christmas holidays. However, after a non-serious prodding from Dominus Dragon about Pentagram (the engine for playing Ultima 8) still not yet being supported, I started looking at what it would take to convert over the engine. And ended up getting hooked on it, to the exclusion of all else for the next several weeks. So below, I present a screenshot of Ultima 8 running in ScummVM:

As of Christmas, the only area I had left to update/correct was that of the music player.. I still need to figure out how to hook it into ScummVM. Otherwise, everything else seems to be working fine.. I was able to play the introduction, get through the docks cutscene, and then enter the city. I haven't yet really put the game through it's paces beyond that. It's better to wait until I'm happy that everything is properly implemented.

And I got distracted yet again, although this time intentionally. At this point, my Christmas holidays had been reached, and I really wanted something a bit more fun to work on than just bugfixing  and playtesting of Ultima 8. Trying to work my way through all those caves and jump platforms wasn't my first choice for my R&R activities.. I much preferred working on the engine itself than actually playing the game.

Back when I started getting hooked on working on Ultima 8, and realized how fun it was, I again set my sights on Nuvie, the engine for Ultima 6 and spinoffs. I'd previously asked the lead developer, Eric Fry, nearly two years back about integrating Nuvie into ScummVM, but at that time he still wanted to work further on Martian Dreams on it with it still being the familiar stand alone project. Trying again in December, I got no response, even after several weeks, and asking a co-developer, again Dominus Dragon, to also try contacting him. Since it had also been several months since any commits had been done, I decided that I'd done my due diligence, and it was finally fair game.

So over the holidays I put Ultima 8 aside, and worked on integrating Nuvie into ScummVM as well, creating a shared "ultima" engine that contains both Ultima 6 and Ultima 8. And, eventually, hopefully the other remaining Ultima games as well. Including my in-progress reimplementation of Ultima 1, which I originally put on hiatus way back when I first got agreement to merge Gargoyle into ScummVM.

So over the holidays I had further fun of converting the Nuvie engine. To make things easy for users, the new ultima6 subengine has two different detection entries, with "original":

 and "enhanced" versions:

As you can see, original gives the look and feel of the original game, whereas the Enhanced version automatically sets all the Nuvie options for enhanced graphics, full screen map, and Ultima 7 style gumps. So users will easily be able to play in an enhanced mode without worrying about fiddling around with configuration options.

As of now, the game starts up and runs. It still needs some work testing the various areas, and making sure it all works properly. In it's case, the sound code was a lot easier to convert than for Pentagram. It actually already used copies of the ScummVM mixer and stream classes. So it was a pretty straightforward process to simply remove them, and remap the code to use the original ScummVM classes instead. Such a relief :).

A downside, though, is parts of the engine, such as the cutscenes, were written in Lua rather than C code. This has disadvantages for properly supporting the game in ScummVM. First of all, I've had to duplicate the entire Lua runtime from Nuvie just to run the scripts, as the version of Lua we have in common/lua/ can't execute the scripts, giving an error when I tried. Plus, when the intro sequence is running, the system isn't properly updated, which means I can't drag the window or, more seriously, exit ScummVM by closing the Window. Long term, I'm likely going to have to rewrite the Lua scripts as C++ code and classes, so Lua can eventually be entirely removed. That will give me greater flexibility to properly handle events for the cutscenes. Also, for enhanced mode, I set the graphic mode to double size of the original, so rewriting the cutscenes in C++ would also make it easier to either stretch the cinematics, or temporarily switch back to 320x200 mode for them.


So where am I going from here? Well, the larger portion of my time is going to return to working on Glk again. Given it's advanced state, I'd be happier getting it finally done and properly supported. At least for the engines that were already part of the Gargoyle suite. Converting other non-Glk-compliant interactive fiction engines can wait until some other time in the future. I'll also spend some smaller portion of my time still working on Ultima 6. For now I'm not worrying about converting the Lua scripts.. I'll first be focusing on ensuring that the rest of the engine is running solidly under ScummVM. After that I can worry about the scripts. We'll see how I go :)