DEEP READING ATLAS · ENGLISH
Mechanical drawings break general MLLMs at three levels — dense reading, projection-consistent reasoning, standards-aware judging. MechVQA measures each level separately, and MechVL shows domain post-training closes much of the gap.
Enter the reading atlasWhole-paper map 01 — 03
High annotation density hides decisive cues; weak domain priors mishandle drafting conventions; spatial reasoning fails under strict projection rules. Adjacent benchmarks cover only slices.
PDF §1Expert-filtered drawings, OCR + MLLM metadata with human verification, three QA-generation routes, cross-model question validation, majority-voted answers, and a leakage-controlled 8:1:1 split.
PDF §3, Fig. 2aMechVL-4B: SFT for schema and grounding, then two-stage self-play DAPO with a format/accuracy/quality reward. Ablations isolate stages, RL algorithm, and reward design.
PDF §§4–5, Tables 2–3Reading orientation. General multimodal LLMs fail on mechanical engineering drawings in ways that go beyond recognition: dense annotations hide decisive cues, projection rules demand cross-view correspondence, and drafting standards require domain knowledge. This paper contributes both a yardstick and a climber: MechVQA, a 20,778-pair benchmark built under multi-stage quality gates, and MechVL, a 4B domain-specialized model whose SFT→DAPO training ladder reaches a Total score of 84.85, above every open- and closed-source baseline tested. Original paper: Qian Kou, Xiaofeng Shi (equal contribution, co-corresponding), Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing, MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding (ICML 2026). This reading uses arXiv:2605.30794v2 (7 July 2026), 28 pages: main text §§1–6 plus appendices A–C; Figs. 1–18; Tables 1–7.
One-sentence thread: the authors argue that mechanical drawing understanding fails at three distinct levels — reading dense annotations, reasoning under projection rules, and judging against drafting standards — so they build a benchmark that measures all three levels separately, then show that a small model with domain-targeted post-training can beat much larger general MLLMs on it. (PDF §§1–2, 5)
The problem is not optical character recognition. A mechanical drawing encodes semantics in a compact graphical language: orthographic multi-view projections, dense dimensioning, section views, symbolic notations, and structured text. The paper's diagnosis (PDF §1) names three compounding factors: high annotation density (decisive cues are easy to miss), weak domain priors (datums, geometric tolerances, roughness conventions), and unreliable spatial reasoning under projection rules (the same feature must be tracked across views). Existing adjacent benchmarks cover only slices of this space — rulebook QA, blueprint symbols, AEC floor plans, or physics puzzles — leaving no unified evaluation of recognition, reasoning, and judgment on real part and assembly drawings. (PDF §1, Table 1)
Where the evidence enters: Table 2 ranks 17 general MLLMs plus MechVL on the test split; Figure 3 breaks scores down by difficulty; Table 3 runs three controlled ablations (training stages, RL algorithm, reward design); Table 5 quantifies how much expert verification changes the metadata the whole pipeline depends on. (PDF §5, App. B.2)
Original Figure 1 answers two orientation questions at once. Panel (a), left column: what makes mechanical drawings hard — high-density annotations, cross-view spatial correspondence, and a specialized symbol vocabulary (datum and geometric tolerance, parameter tables, limit dimensions, roughness, technical requirements, title blocks, section symbols). Panel (b), right and bottom: how MechVQA's ten subtasks map onto the three capability axes, each card showing one real example question and its answer — read one card per subtask to calibrate what the benchmark actually asks. (PDF p. 2, Fig. 1)
A benchmark's claims are only as strong as its construction. MechVQA's pipeline is built around a deliberate trade: answerability and verifiability over brute-force scale. Every stage has a quality gate with a discard path, and anything that cannot be verified is revised or removed rather than kept for volume. (PDF §3.3)
Pre-processing secures trustworthy drawings and metadata; three generation routes produce candidate questions; two voting/validation gates keep only uniquely answerable pairs.
Collect part and assembly drawings from public textbooks, professional handbooks, and design platforms; domain experts remove low-quality, incomplete, or poorly scanned sheets.
An OCR model (MinerU2.5) extracts textual content; strong closed-source MLLMs infer other metadata fields; mechanically trained graduate students then verify every field under an internal annotation handbook.
Three sources: (I) free generation — a closed-source MLLM assesses drawing complexity and drafts subtask-labeled questions; (II) template-based without ground truth — subtask-specific templates bind symbols to features; (III) template-based with ground truth — metadata-grounded deterministic templates, expert-built 2D/3D view matching, and CAD-edited anomaly questions.
In iterative rounds, a different model acts as validator: each question is accepted, rewritten to fix violations, or rejected as unanswerable, checking grounding, unique answerability, and subtask consistency.
Multiple strong models answer each question; a strong LLM judge compares core semantics and keeps only pairs with a clear majority. All answers carry a detailed explanation plus a short final answer.
Split 8:1:1 at drawing-group level — all QAs from one drawing stay in one split. Fused CLIP representations (mean text + image embeddings) are clustered so near-duplicate drawing variants land in the same split, stratified over source, subtask, and difficulty; t-SNE visually confirms matched coverage.
Original Fig. 2a draws this pipeline as three gated zones. Read left to right: the purple pre-processing zone contains Quality Gate 1 (image filtering, with a discard bin) and Quality Gate 2 (experts check and refine OCR + model-extracted metadata); the green zone is iterative question generation with the Fail/Refine loop feeding back into question generation and Quality Gate 3 discarding failed questions; the orange zone is answer-side validation, where multi-model generation goes through response validation and voting before anything enters the high-quality dataset that feeds SFT and RL training. Note that both the question side and the answer side have their own discard paths — nothing passes on a single model's say-so. (PDF p. 5, Fig. 2a)
Is the expert-verification gate doing real work? Table 5 says yes, with numbers. Comparing model-extracted metadata against expert-corrected metadata on the audited groups: view counts are corrected in 41.6% of cases (models usually undercount), section views in 37.8% (confused with section-like or directional views), side/top views in 33.0%/31.8% (systematically over-labeled), technical requirements in 43.7% (surface treatments, heat-treatment parameters, chamfer notation), while coarse part-category labels change in under 1% of cases. The error patterns are systematic, not random — exactly the kind a handbook-guided human pass can fix. (PDF App. B.2, Table 5)
What do the source drawings actually look like? Original Fig. 5 shows typical examples — assembly sheets with numbered part callouts and BOM tables, spring drawings with technical-requirement blocks, and multi-view part drawings dense with dimensional chains. This is the visual regime the benchmark stresses: decisive information is present, but spread across views, tables, and symbols at high density. (PDF p. 13, Fig. 5)
Ten subtasks, three axes, three difficulties. Recognition covers explicit information: Identification & Counting (IC), Dimension & Annotation (DA), Text & Table (TT), Item Localization (IL). Reasoning covers inference beyond direct reading: Structure Understanding (SU), Geometric Calculation (GC), Assembly Relationship (AR), Projection & Multi-view (PM). Judging covers decisions under engineering rules: Anomaly Detection (AD) and Consistency Judgment (CJ). Each question carries exactly one subtask label and one difficulty label (easy/medium/hard, defined in App. B.3). (PDF §3.2, App. B.3)
Original Fig. 2b exposes the dataset's shape — and it is intentionally uneven. The pie chart shows subtask shares: Dimension & Annotation 30.9%, Anomaly Detection 26.0%, Consistency Judgment 16.3%, Geometric Calculation 12.2%, then a long tail (IL 4.1%, PM 3.6%, SU 2.2%, AR 1.9%, IC 1.5%, TT 1.3%). The bars show difficulty counts: 8,138 easy, 7,118 medium, 5,522 hard (summing to 20,778). The right panel (c) previews the training ladder discussed in §§04–05. Keep the skew in mind when reading headline scores: the two judging subtasks plus DA account for nearly three quarters of all questions. (PDF p. 5, Fig. 2b; §3.4)
Total and Avg are two different readings of the same table. The paper defines Avg. explicitly as the mean over all ten subtask scores (a macro average); Total is reported as the headline number and reads as question-level accuracy over the whole test set, which weights the frequent subtasks (DA, AD, CJ) much more heavily (reader inference — the text does not define Total's aggregation explicitly; a weighted estimate from the subtask distribution lands near, though not exactly on, the reported value, since the test split's composition differs slightly from the full dataset). The two can move differently: in Table 3(A), targeted RL lifts Total by +2.90 over full-data RL but Avg. by +4.14, because it improves the rare reasoning subtasks that Avg. counts equally. (PDF §5.3, Table 3)
How answers are scored (App. C.4). The evaluator extracts the content of the model's <answer> tag when present (else text after </think>, else the full response). Three LLM judges — GPT-OSS-120B, DeepSeek-V3.2, and Kimi-k2, temperature 0.1 — independently score semantic consistency with the ground-truth answer as binary 0/1; the final score is the most frequent vote among valid judges. Crucially, judges see only the question, the ground truth, and the model answer — not the drawing — so judging is post-hoc answer verification, not a second attempt at the visual problem. Unparseable or failed judge calls fall back conservatively to 0. (PDF App. C.4, Fig. 18)
Adjacent benchmarks are complementary but scoped to slices; MechVQA's claim is unified coverage of recognition, reasoning, and judging on real part and assembly drawings.
Each existing set stresses one facet of engineering-document understanding.
One benchmark, three capability axes, ten subtasks, three difficulty levels.
MechVL is deliberately simple in architecture — Qwen3-VL-Instruct-4B with the vision side frozen — and puts all the leverage in post-training. The ladder has three rungs: SFT for grounding and schema, full-data DAPO for reliability, then targeted DAPO that resamples weak subtasks. (PDF §4)
SFT creates the reference policy; both RL stages use the same DAPO objective and composite reward — only the sampling distribution changes.
Full-parameter SFT on the LLM module only — vision encoder and projection layers stay frozen — on the MechVQA train split; each instance is (optional drawing, question, target response with rationale + concise final answer), trained with standard causal LM loss (Eq. 1).
For each prompt (drawing + question with a verifiable answer), sample G = 10 candidate responses from the old policy.
Each response gets R = λacc·racc + λfmt·rfmt + λqual·rqual (Eq. 6): semantic accuracy against ground truth via LLM judge [0,1], binary format compliance {0,1}, and LLM-judged quality over Logic/Professionalism/Conciseness [0,1] (Eq. 7); weights 0.60/0.10/0.30 by ablation (Table 7).
Compute group-relative advantage (Eq. 2), token-level importance ratios (Eq. 3), and a PPO-style surrogate with decoupled clipping [0.20, 0.28] (Eq. 4); Dynamic Sampling discards all-correct or all-wrong groups (Eq. 5); overlong reward shaping handles truncation; no KL penalty to the SFT reference.
Run the same objective and reward on a resampled subset with a higher share of underperforming subtasks.
For a prompt with verifiable answer , sample a group from the old policy; each response gets scalar reward . The group-relative advantage, shared by every token of , is: (PDF §4.2, Eq. 2)
With token-level importance ratios (Eq. 3), DAPO optimizes a clipped surrogate with decoupled lower/upper clip ranges — the “Clip Higher” trick that keeps exploration alive by allowing larger upward moves: (PDF §4.2, Eq. 4)
and Dynamic Sampling keeps only groups that contain both correct and incorrect responses, (Eq. 5), resampling otherwise. The rationale (paper statement): an all-correct or all-wrong group has zero advantage variance and thus carries no learning signal; on long-form multimodal reasoning this filtering, plus overlong reward shaping, replaces the usual KL regularizer, which is omitted. (PDF §4.2)
Teaching calculation (this atlas's example, not a paper experiment): suppose responses earn rewards — mean , std . The advantages are roughly : the two correct answers push the policy up, the wrong one pushes down twice as hard, and the middling one barely matters. The group survives Dynamic Sampling because it mixes correct and incorrect; had all four been correct (an already-solved prompt), the whole group would be discarded as signal-free.
with from an LLM judge assessing semantic equivalence to the ground truth (deliberately not strict string matching — differently phrased correct answers still earn reward), requiring exactly one <think>…</think> span and one <answer>…</answer> span, and (PDF §4.2, Eqs. 6–7)
scoring Logic (coherence), Professionalism (drafting terminology), and Conciseness. The reward-weight ablation (App. C.3, Table 7) fixes and finds the best mix at (Total 84.85), ahead of 0.75/0.15 (83.23) and 0.45/0.45 (83.58) — accuracy dominates, but a substantial quality share is still better than none. Teaching calculation: a response with , , earns .
Training configuration (App. C, Table 6). SFT: 3 epochs, AdamW lr , weight decay 0.01, global batch 64, cosine schedule with 0.1 warmup, DeepSpeed ZeRO-3, images capped at 262,144 pixels, max sequence 4,096. RL (all three algorithms): AdamW lr , global batch 128, linear schedule, group size 10, FSDP with CPU offload; GRPO adds a low-variance KL penalty (, clip 0.20/0.30); GSPO uses sequence-level averaging with a very narrow clip (–); DAPO disables KL, clips at [0.20, 0.28], and enables online filtering. Hardware: 8× NVIDIA H800 (80GB). (PDF App. C.1–C.2, Table 6)
Read everything under this protocol: all models are evaluated on the MechVQA test split with the same three-judge binary protocol of §03, no external tools or retrieval. General-purpose baselines are measured as out-of-domain transfer; MechVL is the in-domain baseline quantifying targeted post-training. Table 2's full matrix has 17 general MLLMs; the table below keeps the strongest and most comparable rows. (PDF §5.1, Table 2)
| Model | DA | IL | SU | GC | AR | PM | AD | CJ | Total |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-4B-Instruct (MechVL's base) | 76.96 | 33.09 | 39.58 | 58.54 | 22.00 | 20.00 | 62.86 | 49.33 | 60.23 |
| Qwen3-VL-32B-Instruct | 86.68 | 56.83 | 62.50 | 76.83 | 62.00 | 48.00 | 75.92 | 69.33 | 76.50 |
| GLM-4.6V (best open-source) | 86.68 | 63.31 | 77.08 | 82.93 | 60.00 | 62.00 | 74.29 | 69.33 | 78.91 |
| GPT-5 | 84.99 | 62.59 | 79.17 | 73.58 | 60.00 | 60.00 | 71.02 | 70.67 | 75.44 |
| Gemini-3-Pro-Preview (best closed-source) | 87.74 | 64.03 | 52.08 | 73.58 | 46.00 | 58.00 | 78.37 | 82.67 | 77.28 |
| Qwen3-VL-Plus | 86.68 | 52.88 | 81.25 | 79.67 | 74.00 | 58.00 | 67.76 | 68.00 | 76.33 |
| Claude-Sonnet-4.5 | 78.65 | 56.12 | 70.83 | 75.20 | 62.00 | 54.00 | 64.90 | 64.00 | 71.20 |
| MechVL-4B-SFT (Ours) | 85.20 | 61.15 | 60.42 | 73.17 | 40.00 | 44.00 | 84.49 | 69.33 | 76.36 |
| MechVL-4B-RL (Ours) | 90.70 | 82.01 | 83.33 | 76.83 | 84.00 | 64.00 | 86.94 | 78.67 | 84.85 |
Selected rows from original Table 2, PDF p. 8 (columns IC and TT omitted — MechVL-RL ties with several models there at 88.37 / 97.73; bold marks unique best in the table). Note where RL does not win: GC ties Qwen3-VL-32B, and CJ stays below Gemini-3-Pro-Preview — the paper claims best only on DA, IL, SU, AR, PM, AD, plus Total.
Table 2: 17 open- and closed-source MLLMs plus MechVL, same test split and three-judge protocol.
MechVL-4B-RL Total 84.85: +5.94 over GLM-4.6V (best open-source, 78.91), +7.57 over Gemini-3-Pro-Preview (best closed-source, 77.28), +8.49 over its own SFT stage (76.36). This is a whole-system comparison — it does not by itself isolate which design choice wins.
PDF §5.2, Table 2Fig. 3: accuracy on easy/medium/hard subsets (subset sizes differ, so the paper cautions rankings here need not match Total).
MechVL-4B-RL: 94/79/75 (easy/medium/hard). Versus SFT, RL barely moves easy (92→94) but lifts medium 70→79 and hard 53→75; hard beats the best closed baseline on that subset (Qwen3-VL-Plus, 66) by 9 points. Gains concentrate where reasoning and consistency demands are highest.
PDF §5.2, Fig. 3Table 3(A): controlled stage ablation — SFT, then +DAPO on full data, then +DAPO targeted — same evaluation.
Total 76.36 → 81.95 → 84.85; Reasoning mean 54.40 → 70.75 → 77.04; Recognition 83.11 → 86.26 → 89.70; Judging 76.91 → 81.62 → 82.81. Stage-wise attribution is possible because only the training stage changes.
PDF §5.3, Table 3(A)Table 3(B): GRPO, GSPO, DAPO under the same initialization and full-data setting.
DAPO best: Total/Avg. 81.95/79.12 vs GRPO 80.47/74.80 and GSPO 78.77/73.73; the largest margin is on Reasoning (70.75 vs 64.49 and 61.29). Within this paper's family of group-based optimizers, DAPO's dynamic sampling and decoupled clipping pay off most on reasoning-heavy subtasks.
PDF §5.3, Table 3(B)Table 3(C) ablates the accuracy term and the quality term in the targeted phase; Fig. 4 tracks response length during training.
Full reward best (84.85); removing quality drops to 83.44; binary accuracy 82.24; token-level F1 worst at 80.33. Fig. 4 explains the ordering: Acc(F1) collapses responses from ~1.1K to <0.8K tokens (terse, weakly grounded), w/o Qual drifts to ~1.3K (verbose without accuracy gains), Full holds a controlled 1.2K–1.25K band with the best final accuracy.
PDF §5.3, Table 3(C), Fig. 4Table 5 (App. B.2): correction rates between model-extracted and expert-corrected metadata on audited groups.
View counts corrected 41.6%, technical requirements 43.7%, section views 37.8%, side/top views 33.0%/31.8% — all systematic patterns; part category changes <1%. The structured metadata feeding template questions and ground-truth answers is materially changed by the human pass.
PDF App. B.2, Table 5Each row is one stage transition; unit: points
| Item | Start (%) | End (%) | Change (percentage points) |
|---|---|---|---|
| Baseline → SFT | 60.23 | 76.36 | +16.1 |
| SFT → + DAPO (full) | 76.36 | 81.95 | +5.6 |
| full RL → + DAPO (targeted) | 81.95 | 84.85 | +2.9 |
Total score; unit: points
| Item | Start (%) | End (%) | Change (percentage points) |
|---|---|---|---|
| vs GLM-4.6V (open-source best) | 78.91 | 84.85 | +5.9 |
| vs Gemini-3-Pro-Preview (closed-source best) | 77.28 | 84.85 | +7.6 |
| vs GPT-5 | 75.44 | 84.85 | +9.4 |
Original Fig. 3 shows why the difficulty axis matters. Each bar is one model's easy/medium/hard accuracy. All models drop from easy to hard — harder questions demand cross-view correspondence, stricter constraint satisfaction, and multi-step reasoning — but the spread widens sharply: MechVL-4B-RL (94/79/75) keeps the flattest profile, while the same-size GPT-4o-mini collapses (64/33/26). Reading tip: compare MechVL-4B-SFT (92/70/53) with MechVL-4B-RL next to it — the RL stage's entire effect is visible as the medium and hard segments, exactly where dense-drawing reliability was the bottleneck. (PDF p. 8, Fig. 3)
Original Fig. 4 is the mechanistic explanation for the reward ablation. It plots response length over RL training steps for four reward designs. Acc(F1) — token-overlap accuracy — collapses from ~1.1K to below 0.8K tokens: overlap feedback rewards terseness over grounding, and it is also the worst final scorer in Table 3(C). w/o Qual climbs to ~1.3K, verbosity without matching accuracy. Full holds a controlled band around 1.2K–1.25K with the best final score — the authors read this as better-calibrated reasoning traces rather than merely longer ones. (PDF p. 9, Fig. 4)
Two source discrepancies to carry while reading the numbers. ① The abstract and §5.2 report MechVL outperforms the strongest closed-source baseline by 7.57 points, but the introduction says it surpasses closed-source baselines “by 6%” — no reported number matches 6%; the 7.57-point gap to Gemini-3-Pro-Preview (77.28) is the one grounded in Table 2. ② Fig. 2c's training curve prints RL(targeted) at 83.44, while Tables 2 and 3(A) report the targeted-RL model at 84.85 (83.44 coincidentally equals Table 3(C)'s w/o-Qual row). This atlas uses the table values and flags both mismatches rather than silently resolving them. (PDF pp. 2, 5, 8–9)
The mechanism and the measurement reinforce each other in a specific way: the benchmark's own failure diagnosis predicts where training should help, and the ablations confirm it there. The paper's diagnosis named dense reading, projection reasoning, and standards judging; the training ladder's controlled ablation (Table 3A) shows SFT alone lifts the base model mostly on reading, while RL — with its semantic-accuracy-plus-quality reward — delivers its largest capability gain on Reasoning (54.40 → 77.04 across both stages) and its largest difficulty gain on the hard subset (53 → 75). The reward-dynamics figure then ties the reward design to the shape of the reasoning traces themselves. Each link is supported by a controlled comparison, not just by the headline ranking. (PDF §§5.2–5.3, Figs. 3–4)
The evaluation chain deserves a sober read (reader inference, not a paper claim). Both ends of this work are LLM-mediated: questions and answers are generated and voted on by strong closed-source models (with expert gates), and final scoring uses three LLM judges that never see the drawing. These are different model sets at each end, and the judges do post-hoc answer verification rather than visual re-solving — a reasonable, scalable protocol — but absolute scores still inherit judge behavior, and the conservative fallback (failed judging scores 0) can punish unusual answer formats. The authors also acknowledge the related contamination caveat: public-source drawings plus web-scale pretraining means absolute leakage control is impossible; their mitigation is drawing-level and cluster-level separation. (PDF §§3.4, 5.1, Apps. A, C.4)
Boundaries the paper states itself (App. A): drawings come from educational/professional public sources, not proprietary industrial archives — legacy blueprints and company-specific drafting practices are out of scope; the benchmark is 2D-drawing understanding, not 3D CAD reconstruction or STEP/IGES generation; construction still depends on OCR and visual clarity; and benchmark validation is explicitly incomplete — no human-expert upper bound or inter-annotator agreement statistics are reported yet, so the remaining gap between MechVL and domain experts is unquantified. Release of original drawings follows the underlying materials' redistribution permissions. (PDF App. A)
If you reproduce or build on this work: dataset — 3,281 drawings, 20,778 pairs, 8:1:1 drawing-group split with CLIP-cluster dedup; model — Qwen3-VL-Instruct-4B, vision frozen, LLM-only full-parameter SFT (3 epochs, lr 1e-5, batch 64) then DAPO (group 10, clip [0.20, 0.28], no KL, dynamic sampling, reward 0.60 acc / 0.10 fmt / 0.30 qual), stage 2 upsamples weak subtasks; evaluation — three LLM judges (GPT-OSS-120B, DeepSeek-V3.2, Kimi-k2, temp 0.1) with binary semantic scoring and majority vote; training framework — LLaMA-Factory (SFT) and EasyR1 (RL) on 8× H800. Code: github.com/xiaofengShi/MechVQA. (PDF §§3–5, Apps. B–C)