SciDiagBench Leaderboard

Each generated diagram D̂ is evaluated against the original human-created reference diagram D by Gemini 3.1 Pro. Evaluation is conducted using type-specific criteria and weights.

1. 1. Evaluation Dimensions & Criteria

   Evaluation spans three top-level dimensions D = {Structure, Content, Aesthetics}, with Structure and Content decomposed into sub-dimensions:

   Structure Spatial & Topological

   • Visual Organizational Layout (VOL): Layout balance, visual hierarchy, reading path, and clean routing.
   • Logical Accuracy (LA): Topological dependency correctness, execution flow, and algorithmic sequence validity.

   Content Information Fidelity

   • Visual Element Representation (VER): Graphical symbols, domain metaphors, icon clarity, and visual encoding.
   • Content Accuracy (CA): Scientific factual correctness, terminology, and fidelity to paper text and caption.

   Aesthetics Graphic Design Quality

   • Visual Polish: Typography, color harmony, visual contrast, clean alignment, and publication-ready presentation.

   For each diagram type τ, Gemini 3.1 Pro evaluates instances against type-specific criteria C_d,τ defined under these sub-dimensions.

2. 2. Blind Reference-Anchored Scoring

   The judge VLM receives the paper context P, target caption C, human reference D, and generated diagram D̂ in anonymized and randomized order:

   ({s_D̂,c}_{c ∈ Cτ}, {s_D,c}_{c ∈ Cτ}) = JudgeVLM(P, C, D, D̂, C_τ)

   s_·,c denotes the score of a diagram under criterion c ∈ C_τ.

3. 3. Two-Level Weighted Aggregation & Relative Score

   Criterion scores are aggregated into a composite score S(·) using two-level normalized weights:

   • Intra-dimension weights (w_c): Normalized within each top-level dimension (∑_{c ∈ Cd,τ} w_c = 1).
   • Dimension weights (W_d,τ): Type-specific weights across the three dimensions (∑_{d ∈ D} W_d,τ = 1).

   The composite score for either diagram · ∈ {D̂, D} is:

   S(·) = ∑_{d ∈ D} W_d,τ ∑_{c ∈ Cd,τ} w_c · s_·,c (where D = {Structure, Content, Aesthetics})

   The final instance score is computed as a relative ratio against the human reference diagram:

   S_final(D̂) = S(D̂) S(D̂) + S(D)
   [0, 0.5000) Inferior to human reference
   (Reference preferred)

   0.5000 Human reference parity
   (S(D̂) = S(D))

   (0.5000, 1] Superior to human reference
   (Generated preferred)

4. 4. Dimension-Level Breakdown & Benchmark Aggregation

   • Dimension & Sub-dimension Relative Scores: Computed using the relative ratio on the corresponding criterion scores:
     S_d,final(D̂) = S_d(D̂) / [S_d(D̂) + S_d(D)]
   • Diagram-Type Score (10 test instances): Arithmetic mean of relative scores across the 10 test instances of type τ:
     S̄_τ = ¹/₁₀ ∑_i=1¹⁰ S_final⁽ⁱ⁾(D̂)
   • Overall Benchmark Score (110 instances): Mean across all 110 test instances:
     S̄_overall = ¹/₁₁₀ ∑_i=1¹¹⁰ S_final⁽ⁱ⁾(D̂)