Skip to content

[Bug] [macOS] [cause and fix included] NaN texture coordinates resolve toward max on Apple GPUs breaking ACNH backdrops #476

Description

@mtmasantana

Prerequisites

  • I am confident that my issue has not been reported already in open OR closed issues.
  • I am able to reliably reproduce the issue on the latest stable or canary version.
  • I am not using an unofficial build (Flatpak, Emudeck, etc.).
  • I am a silly goose and didn't read these boxes.
  • I am a silly goose and DID read these boxes.

Description of the issue

Backdrops that should be pure black render white. All interiors are affected with a white void around the room diorama. Scenes like the Tom Nook save management one and the KK Slider intro one are also affected. Geometry, characters and lighting are all correct aside from the bug. See screenshots.

Image Image Image

Reproduction steps

Easiest way to reproduce is to open any save to the main menu and press minus to open the Tom Nook save management screen. Entering any interior works too. Bug doesn't need any specific conditions (aside from using an Apple GPU).

Log file

Log file not particularly informative and I've already thoroughly diagnosed the issue and found a fix. Not necessary.

OS

macOS Tahoe 26.2

Ryujinx version

Canary 1.3.339, also reproduced in Stable 1.3.3

Game version

Animal Crossing: New Horizons 3.0.3

CPU

Apple Silicon M1

GPU

Apple Silicon M1

RAM

8 GB

List of applied mods

No mods applied.

Additional context?

Bug is Mac only. Found posts going back years across multiple Ryujinx versions with none mentioning being on Windows or Linux. I haven't tested on a different OS but you'll see why it's Mac only when I explain the cause.

1: https://www.reddit.com/r/Ryubing/comments/1t07vpy/any_fix_to_this_problem_in_animal_crossing/
2: https://www.reddit.com/r/yuzu/comments/1qndovd/animal_crossing_new_horizons_interior_background/ (yuzu sub but poster is on Ryujinx, mentions being on Mac)
3: https://www.reddit.com/r/Ryujinx/comments/zfuz4f/playing_animal_crossing_on_mac_interior (mentions being on Mac)
4: https://www.reddit.com/r/Ryujinx/comments/1g29bh9/animal_crossing_nh_glitch_why_are_the_interiors/ (commenter mentions having the same issue on Mac)

The white is painted by a single draw: the last full-screen i=3 post-process pass of each frame. Its fragment shader (MoltenVK shader hash bf7f5bae9c901307) is a preserve-chroma tonemapper followed by a 32x32x32 color-grading LUT lookup. The shader's output RGB is nothing but that LUT sample.

The shader computes Rec.601 luminance L from exposure * (scene + bloom), then divides by it. Where the scene is exactly black, L == 0, so the following happens:

1.0 / L -> +Inf
L' = 1 - exp(-L) -> 0
fma(0, -Inf, ...) -> 0 * Inf -> NaN

The NaN propagates through the tonemap blend, the saturation lerp, and the LUT scale-and-bias, so all three LUT sample coordinates arrive as NaN.

Pixels with L > 0 are not affected by the bug because NaN is not produced from the math.

Sampling a texture with NaN coordinates causes different behavior across different GPUs.

Apple AGX resolves them toward max, the LUT's white corner, measured at (251, 251, 251). This makes the backdrop render white. Nvidia/AMD probably resolve NaN somewhere that lands on the black corner, since the same shader produces the same NaN on every platform and the bug is only reported on Mac. I haven't tested that directly.

Bug confirmed outside the emulator with a standalone Metal program (attached).

It fills a 32x32x32 RGBA8 texture with an identity ramp, so the color returned is the resolved sample coordinate, then samples it on my M1 with a linear clamp-to-edge sampler. Each line below is the coordinate passed in followed by the color that came back:

(0, 0, 0) returns (0.0000, 0.0000, 0.0000), which is 8-bit (0, 0, 0), black
(0.5, 0.5, 0.5) returns (0.5000, 0.5000, 0.5000), which is 8-bit (128, 128, 128), gray
(1, 1, 1) returns (1.0000, 1.0000, 1.0000), which is 8-bit (255, 255, 255), white
(NaN, NaN, NaN) returns (1.0000, 1.0000, 1.0000), which is 8-bit (255, 255, 255), white!!!

(NaN, 0, 0) returns (1.0000, 0.0000, 0.0000), which is 8-bit (255, 0, 0)
(0, NaN, 0) returns (0.0000, 1.0000, 0.0000), which is 8-bit (0, 255, 0)
(0, 0, NaN) returns (0.0000, 0.0000, 1.0000), which is 8-bit (0, 0, 255)

(+Inf, +Inf, +Inf) returns (1.0000, 1.0000, 1.0000), which is 8-bit (255, 255, 255)
(-Inf, -Inf, -Inf) returns (0.0000, 0.0000, 0.0000), which is 8-bit (0, 0, 0)

NaN resolves to exactly 1.0, and it does so per component independently. One NaN component does not affect the others.

Infinities clamp correctly in both directions, so the clamp itself is working normally and the behavior is specific to NaN. The result is the same as if NaN compared greater than max, which is what happens to a comparison-based clamp when every comparison against NaN returns false.

This also accounts for the backdrop measuring (251, 251, 251) rather than pure white. The coordinate resolves to exactly 1.0 and the sample lands on the LUT's (1,1,1) texel, which in this game's grade is around 251.

Proposed fix: flush NaN texture coordinates to 0 in the SPIR-V code generator, matching the behavior the guest shader was written against. In src/Ryujinx.Graphics.Shader/CodeGen/Spirv/Instructions.cs, immediately after pCoords is assembled in GenerateTextureSample, emit OpIsNan and OpSelect to substitute 0.0 for any NaN component before the coordinates reach the sample instruction. OpSelect works per component, so a NaN in one axis does not disturb the others. Patch attached. With it applied, the backdrop renders black on the Nook screen and in interiors like it's supposed to. I didn't notice any performance difference.

My fix applies the flush to every floating-point texture sample on all backends. Nvidia and AMD probably don't need it. A flag probably should be added: HasNanTextureCoordQuirk, alongside the existing HasFrontFacingBug/HasVectorIndexingBug in Ryujinx.Graphics.GAL/Capabilities.cs, set when MoltenVK is detected and threaded through IGpuAccessor -> HostCapabilities -> CodeGenContext.

TLDR when luminance of 0 is used for some pixels, math that creates NaN happens. This resolves differently on Nvidia/AMD compared to Apple, creating a Mac-only noticeable visual bug with a pretty simple fix in only a few lines. The fix is flushing NaN coordinates to 0 in the SPIR-V code generator. Not sure if the same bug happens on Nvidia/AMD but it probably doesn't since the bug is pretty noticeable and someone would've probably brought it up if it did.

nan_texcoord_probe.txt

This is supposed to be a .swift file but they aren't supported on GitHub so I uploaded it as a txt. It's the Metal program I used for showing the NaN behavior outside the emulator. Rename it to .swift then build and run with "swiftc -O nan_texcoord_probe.swift -o nan_texcoord_probe && ./nan_texcoord_probe".

nanflush-fix.patch

With this applied, things rendered properly. See screenshot below. If you want to test it yourself, make sure the ACNH shader cache is disabled/purged otherwise the SPIR-V generator doesn't run on the already cached shaders.

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions