PROVE: A Perceptual RemOVal cohErence
Benchmark for Visual Media

Abstract

Evaluating object removal in images and videos remains challenging because the task is inherently one-to-many, yet existing metrics frequently disagree with human perception. Full-reference metrics reward copy-paste behaviors over genuine erasure; no-reference metrics suffer from systematic biases such as favoring blurry results; and global temporal metrics are insensitive to localized artifacts within edited regions.

To address these limitations, we propose RC (Removal Coherence), a pair of perception-aligned metrics: RC-S, which measures spatial coherence via sliding-window feature comparison between masked and background regions, and RC-T, which measures temporal consistency via distribution tracking within shared restored regions across adjacent frames. To validate RC and support community benchmarking, we further introduce PROVE-Bench, a two-tier real-world benchmark comprising PROVE-M, an 80-video paired dataset with motion augmentation, and PROVE-H, a 100-video challenging subset without ground truth. Together, RC metrics and PROVE-Bench form the PROVE (Perceptual RemOVal cohErence) evaluation framework for visual media. Experiments across diverse image and video benchmarks demonstrate that RC achieves substantially stronger alignment with human judgments than existing evaluation protocols.

Motivation

Existing evaluation metrics for object removal exhibit systematic biases that conflict with human perception.

Figure 1. Illustrative examples of metric bias in object removal evaluation. (a) Full-reference metrics reward copy-paste behavior over genuine erasure. (b) No-reference metrics favor blurry outputs across diffusion steps. (c) Traditional vs. diffusion-based methods show inconsistencies between metric judgments and visual perception.

Figure 2. RC-S captures locally visible side effects and residual artifacts. Human-perceived ranking (1 = best): D > B > A > C. RC-S ranking: D(1) > B(2) > A(3) > C(4), consistent with human perception. In contrast, ReMOVE ranks A(1) > B(3) > C(4) > D(2), and CFD ranks A(1) > B(2) > C(4) > D(3) — both incorrectly favor the residual-containing result A over the cleanest removal D.

Figure 3. “Blur is Clean” bias analysis. As blur radius increases inside the masked region, ReMOVE scores incorrectly improve and CFD incorrectly decreases (appears better), while RC-S correctly and monotonically degrades — demonstrating its robustness against the blur-favoring bias.

Proposed RC Metrics

We introduce a unified local distribution matching framework in deep semantic feature space, named Removal Coherence (RC), instantiated as two complementary metrics.

Figure 4. Overview of the proposed RC metrics. (a) RC-S measures intra-frame spatial coherence by comparing masked and background feature distributions within sliding windows. (b) RC-T measures inter-frame temporal consistency by comparing restored-region feature distributions across adjacent frames under union-based cropping and intersection-based evaluation.

Key Design Choices

Why Local Cropping Matters

Cropping strategies of ReMOVE vs. RC-S. Green boxes denote crop regions, red areas denote masks. ReMOVE uses a single enlarged crop even when targets are spatially far apart, introducing excessive irrelevant background that dilutes the feature difference. RC-S crops each target independently with local context, enabling fine-grained detection of incoherence.

Counter-intuitive rankings under local perturbations. We apply Gaussian blur or region swap to the masked areas. Human judgment clearly prefers: ori > blur > swap. However, ReMOVE and CFD produce the reverse ordering (red = best-ranked, green = second-best), while RC-S remains consistent with human perception.

PROVE-Bench

A two-tier real-world benchmark for evaluating object removal in video, combining paired evaluability with unconstrained stress testing.

Figure 5. Construction pipeline of PROVE-M: real-world paired capture with controlled conditions, three-stage pairwise quality control, and Ken Burns-style motion augmentation applied synchronously to input–mask–GT triplets.

(a) PROVE-M — Sample frames from the motion-augmented paired benchmark.

(b) PROVE-H — Sample frames from the hard real-world benchmark.

Comparison with Existing Benchmarks

Results

Benchmark Results on PROVE-M

Quantitative evaluation of mainstream video object removal methods on the PROVE-M benchmark. ↓ means lower is better, ↑ means higher is better.

Benchmark Results on PROVE-H

Quantitative evaluation of mainstream video object removal methods on the PROVE-H benchmark. ↓ means lower is better, ↑ means higher is better.

Note: Due to compliance requirements, the open-source data differs slightly from the data used in the paper. The results above are based on the open-source version and may exhibit minor numerical differences from the paper, but the overall trends remain consistent.

Human Correlation Analysis

RC-S achieves the best average correlation with human rankings across six benchmarks, ranking first on five of six benchmarks under both Kendall's τ and Spearman's ρ.

Validation of RC-T

RC-T responds sensitively and monotonically to controlled temporal corruptions, whereas existing temporal metrics (TC, TF) remain largely insensitive.

Figure 6. Sensitivity-based validation of temporal metrics under increasing corruption severity. RC-T exhibits monotonically degrading scores, whereas TC and TF remain insensitive or even improve.

Citation

If you find our work useful for your research, please consider citing our paper: