Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding
From reading dimensions to reasoning across views.
A benchmark
and specialized models for the language of engineering.
What is the value of the module m in the gear parameter table?
Reference answer m = 3Paper-reported dataset scale and evaluation results. See resources for public release sizes.
01 / THE CHALLENGE
Mechanical drawings combine geometry, dimensions, tolerances, and conventions in a single, information-dense image.
MechVQA tests whether multimodal models can understand that language. It spans real part and assembly drawings, with questions that move from visual recognition to spatial reasoning and standards-aware judgment. MechVL builds on this benchmark with supervised fine-tuning and two-stage self-play reinforcement learning.
02 / THE BENCHMARK
Three capabilities. Ten tasks. Grounded in real engineering drawings.
Identification & Counting
Dimension & Annotation
Text
& Table
Item Localization
Structure Understanding
Geometric Calculation
Assembly
Relationship
Projection & Multi-view
Anomaly Detection
Consistency Judgment
Recognition / Easy
What is the value of the module m in the gear parameter table?
The value of module m in the gear parameter table is 3.
From the public evaluation release. Reference answer, not live model output.
Reasoning / Hard
From the right-side sectional view, deduce the function of the serrations and spring, and explain how they achieve the locking function of the handle.
The serrations provide a mechanical interlocking self-locking surface, the spring provides axial preload to maintain engagement, and compressing the spring disengages the teeth to achieve unlocking.
From the public evaluation release. Reference answer, not live model output.
Judging / Medium
Based on the table data, calculate the difference between s1 and s2 for d=20 and d=60, and determine whether the adhesive pad thickness varies with diameter.
According to the table data in the mechanical drawing: - When d=20, s1=2, s2=1, difference s1‑s2 = 1; - When d=60, s1=2.5, s2=1.5, difference s1‑s2 = 1. The difference for other diameters in the table (e.g., d=30, d=40) is also 1. Since the adhesive pad thickness equals the difference between s1 and s2, and this difference is 1 for all diameters, the adhesive pad thickness does not vary with diameter and remains constant.
From the public evaluation release. Reference answer, not live model output.
03 / RESULTS
MechVL-4B-RL achieves the highest total score among the models evaluated in the paper.
over Gemini-3-Pro-Preview,
the strongest closed-source
baseline in this evaluation.
Results from Table 2, arXiv v1 (May 2026). Accuracy on the paper’s test split; higher is better. MechVL receives domain-specific post-training; general-purpose baselines do not. This is a paper snapshot, not a live leaderboard.
| Model | Total ↑ |
|---|---|
| MechVL-4B-RLOurs · SFT + self-play RL | 84.85 |
| GLM-4.6VOpen-source baseline | 78.91 |
| Gemini-3-Pro-PreviewClosed-source baseline | 77.28 |
| MechVL-4B-SFTOurs · supervised fine-tuning | 76.36 |
| GPT-5Closed-source baseline | 75.44 |
| Qwen3-VL-4B-InstructBase model | 60.23 |
Total is the question-level aggregate, not an unweighted mean of the ten subtask scores. Evaluation protocol ↗
04 / MEET MECHVL
Qwen3-VL-4B-Instruct, adapted through supervised learning and targeted self-play.
Learn drawing-grounded answers from MechVQA. Train the language model while freezing the vision encoder and projector.
76.36 total scoreOptimize on the full training split with rewards for answer accuracy, response format, and explanation quality.
81.95 total scoreResample underperforming subtasks and continue reinforcement learning with the same objective and reward design.
84.85 total scoreTraining-stage results: paper Table 3. Explore the training framework ↗
05 / OPEN RESOURCES
Data, checkpoints, and the tools to reproduce and extend the work.
Research paper, inference scripts, SFT recipe, RL framework, and evaluation pipeline.
1,185 question–answer records and 562 drawing images, packaged in the repository.
12,749 training + 766 validation records; 3,371 images. VQA-only release with checksums and provenance.
4B SFT and RL models. Use ModelScope for full weights; check each Hugging Face model card for mirror availability.
The public releases and the paper’s full dataset have different scopes. Additional internal training data are not included. See the repository release status for details.
06 / CITATION
Please cite MechVQA if you use the benchmark, MechVL models, or released training and evaluation assets.
Download BibTeX ↙@misc{kou2026mechvqabenchmarkingenhancingmultimodal,
title={MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding},
author={Qian Kou and Xiaofeng Shi and Yulin Li and Xiaosong Qiu and Xinyang Wang and Hua Zhou and Cao Dongxing},
year={2026},
eprint={2605.30794},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2605.30794}
}