Upscaling Myst III

I have already described at length how much the Myst video games are important to me^[1]. The first game I bought in the series, at random in a store, was Myst III. It managed to capture my imagination as soon as I pored over the box and the fascinating landscapes displayed. While developped by Presto Studios rather than the original creators Cyan Worlds, it remains to this day my favorite game in the franchise, thanks to the atmosphere conveyed by its visuals and its beautiful soundtrack.

Released in 2005, it was following the by-then tried-and-true recipe of point & click pre-rendered 360° panoramas, with inserted full motion video actors and many intricate puzzles. But almost twenty years later, the original renderings are trailing behind my fond memories of my first playthrough. Each panorama was displayed at a resolution of 800x600 pixels, a very small fraction of my current laptop screen.

Upscale all the things

Five years ago, I ran an experiment to check if I could generate higher resolution versions of all Myst III environments using modern upscaling techniques. Panoramas could be conveniently dumped thanks to a tool called Riveal, and run through an out-of-the-box neural upscaler. The panorama faces could thus be converted from 640px to 2560px. The result was suprisingly clean, in part because the initial renderings had a slightly stylized look that fit well with the oversmoothing typical of such upscaling. I ended up processing all panoramas and a few of the videos. But I could not think of a way to feed the upscaled data back into the original game, and I moved on to other projects.

There are obvious questions of copyright on the upscaled data, which still belong to Cyan ; I will not distribute the upscaled data.

Skip ahead a few years, and an open-source re-implementation of the original game is now available in ScummVM, a well-known emulator for puzzle games. While sorting trough a backup of the upscaled data (more than 50GB of high resolution PNGs), I found the motivation to finally try and load the new data in the old game.

Updating the game

The goal was to load the upscaled environments in the game and display them at a higher quality while minimizing modifications in the ScummVM code base. This meant handling most of the changes in the game data packs, and not in its code. The ScummVM codebase was incredibly valuable in understanding how to parse the original data and replace part of it with the upscaled images.

Environments

There are series of data packs, one per region of each level of the game, where all panoramas are stored as cube faces encoded in 640x640 JPEGs. GUI elements are also compressed as regular JPEGs with varying sizes. All of these are stored along with their pixel dimensions, and the game is able to load them no matter the size. I wrote a small tool to open the existing packs, insert the upscaled images inside, update all dimensions and offsets, and write these back to disk. Thankfully ScummVM could already target higher rendering resolution^[2], meaning that the game directly benefited from the increased asset resolution.

Effects

To handle interactions with objects in the scene, small images are composited on top of the panorama faces. For instance, a light projector that is initially turned off can be replaced with the version casting light when the user has clicked on it, by copying from a smaller image that covers this particular spot in the panorama. Sadly, I did not dump these interactive regions in my initial 2019 tests. To alleviate this oversight, I wrote a small tool to extract the corresponding images and upscale them with a basic bicubic filter, before re-inserting them in the data packs. The upscaling quality is subpar, but this allowed me to test the loading modifications.

A more complex effect is used in some panoramas, to apply real-time distortions to part of the scenery. This is mainly used for water surfaces and force fields. The effect is controlled by a mask stored as an additional set of images. The mask also packs additional parameters controlling the type and strength of effect. Because this is bit-packed data, there was not much sense in upscaling it in any way other than a nearest-neighbor filter. I thus chose to keep them at 640p, and adjust the distortion code to work with a mask of a different resolution than the panorama.

Overall, effects were where most of the code modifications were required: interactive regions are placed on top of the panoramas using absolute pixel coordinates that had to be rescaled. Similarly, the code for distortion effects was designed to run at a fixed 640p resolution, something I've fixed. The code is now able to handle mixed resolution smoothly, by broadcasting the computed distortion to all pixels of the panorama that map to a given pixel in the mask.

Videos

When a more complex interaction is triggered, the game can also play back a short video. These are encoded as BINK videos^[3], and as mentioned previously I had dumped some of them as image sequences using ffmpeg. At the time, I'd upscaled a small subset of these but this was an even more time consuming process and the result was very unstable, as can be seen below. When sorting through the backup, I also realized that some of the image sequences were mixed up. Furthermore re-encoding videos from the images proved to be more cumbersome than expected, as the old BINK1 encoder is only available on Windows and quite limited. This is were my motivation faltered and I decided to leave video support for future work.

Improvements and takeaways

The current result has multiple limitations:

• The network was designed for 2D image upscaling, meaning that visual continuity between the faces of a panorama is never guaranteed. In practice this is not too visible thanks to the smooth rendering style, but there are edge cases.
• The same goes for the borders of interactive spots, which mix up with the upscaled background
• Additionally, temporal continuity between successive video frames is not enforced by the network, leading to flickering and blurry patches in the pre-rendered sequences. This also amplifies compression artefacts in the BINK videos.

It would be interesting to run new tests with a video upscaling network, taking this as an opportunity to increase the original 15fps framerate to a more modern 60fps too. Finding a way to enforce continuity constraints between adjacent faces or at the edges of interactive spots is also an interesting problem. This might require re-training a network for this specific task^[4], or finding a way of compositing multiple image patches with soft transitions to hide inconsistencies. Once again, this is left for future me.

Through this project, I have been able to study how ScummVM re-implemented the game functionalities, from asset loading to display and gameplay. Panorama compositing and effect animation are handled entirely on the CPU ; only the resulting face textures are sent to the GPU for display. That way, the implementation is able to run on as many platforms as possible, without too much work to support additional graphics APIs. On the other hand, running visual effects on the 4x upscaled panoramas now requires a decent CPU to reach a satisfying framerate. The data packs have also significantly increased in size, pushing the game install from 2GB to around 10GB (without any upscaled video).

ScummVM provides a great opportunity for this type of modding, as it its codebase is easily extensible. I was for instance able to add a field of view modifier quite quickly along the way. My modifications for packing uspcaled data and displaying it are available on Github.

But the work of the community is also paramount for video game preservation. One of the best examples is how their Myst III implementation has been used as the basis for the re-release of the game on Steam in recent years. I don't know if Cyan was able to compensate the maintainers and volunteers for their time on the project, but I'm guessing most of the original sources and high-resolution assets have been lost to time.