Artifacts in 0112, 0113, 0114 verified videos

[blahner]

It seems that for the three viewpoints (0112, 0113, 0114) in the "match blows balloon" scenario the artifact removal step for the "verified" set may have introduced other artifacts (see screenshot of the 'testing' split of 0014). Specifically, a light-colored box around scene probably intended to edit the video to stop the rotating platform in the 'deterministic artifacts' case. I do not see these artifacts for 0057 or 0051 with a similar rotating platform.

• Can you confirm that the artifacts I am seeing in these videos exist in the dataset and it is not a download error on my end? I re-downloaded the set multiple times and still see the artifacts. I do not see the artifacts in the original physics IQ videos.
• If these artifacts do exist in the dataset, would you be able to release a corrected version for these videos? I understand the importance of keeping the physics IQ verified evaluations stable, but I have other use cases for the dataset where having naturalistic videos (no artifacts like this) is important.

Thank you for this dataset and for your help!

[tuliti]

Hi @blahner,

thank you for taking a look at the data!
Our goal was to get the best out of the existing videos.

I will take a detailed look tomorrow in the office.

best Tim

[tuliti]

Hi @blahner,

took a little bit longer than expected to get to the bottom of this.

Re your questions:

• Yes, the artifacts are present and your download is correct.
• Will talk to the team, how we handle this and post here.

First of all, these artifacts are a leftover from the freezing action.
Apparently, as soon as the ballon pops, the original recording camera changes to a darker image and contrasts this freezing area that had really good representations at the beginning of the video.

Here one can see the original vs the verified version (aggregated motion masks).
The motion masks account for the majority of the Verified and likewise Original Score (3 out of 4 metrics).
The motion masks are exactly as intented in the verified version with a focus on the balloon rather than the rotator.

For this project, we pushed the original benchmark as far as we could, but are still limited to the original recordings and some of its shortcomings. There’s much more to come: broader, more detailed, and ultimately stronger work is already on the way.