RL Training Non-convergence on OmniGen2 with EditScore-7B

[xiaoyaopengfei]

Hi authors,

Thanks for your impressive work on EditScore! I am currently reproducing the RL training using omnigen2_edit_rl_single_machine_editscore7b. However, I've encountered some stability issues where the reward_mean fluctuates significantly without a clear upward trend.

After analyzing the training logs and specific tasks, I have two major observations and would love to hear your insights:

1. Reward Sparsity in motion_change Tasks
   I observed that in many motion_change samples , the SC_score is frequently 0.
   even when the model attempts an edit, EditScore often gives a zero score for Semantic Conformity, leading to sparse rewards.
   Do you think this is caused by EditScore being too strict on complex motion, or is it a limitation of the base model's initial exploration? How did you handle these zero-reward samples during your training to avoid gradient instability?

2. Reward Inconsistency in background_change Tasks
   I noticed cases where two images have very similar background fidelity/similarity, yet their rewards differ significantly.
   This high variance in rewards for similar visual outputs seems to introduce a lot of noise into the policy gradient.
   Is this inconsistency a known behavior of the 7B reward model? Or are there other normalization techniques you found effective?

Environment & Hyperparameters:

Base Model: OmniGen2

Reward Model: EditScore-7B

Tasks: rl_abs_9tasks.jsonl

Training setup: Single machine, default parameters from the repo.

I've attached my training curve and some example cases for reference. Looking forward to your guidance!

Best regards,
Spike

[sabulin]

Hi Spike,

Thanks for trying EditScore and for the detailed analysis!

Regarding the instability you observed:

1. Reward variance and sparsity

EditScore-7B is a relatively small generative reward model, so some degree of noise and reward variance is expected. To mitigate this, we propose an inference-time compute scaling strategy in the paper: perform K independent stochastic forward passes of the reward model and use the mean of the K scores as the final reward. This significantly reduces variance in practice.

You might try this strategy during training. Alternatively, using a larger EditScore model or the EditScore variant trained on Qwen3-VL can also improve reward stability.

2. Training curve vs. final policy performance

In our experience, the RL training reward curves do not always show a clear monotonic trend. The more reliable signal is the final performance of the policy model on evaluation benchmarks, such as GEdit, ImgEdit, and Emu-Edit.

We recommend periodically evaluating the policy on these benchmarks to determine whether RL training is actually improving editing quality.

Hope this helps, and thanks again for the careful investigation!