Skip to main content

Qwen3-VL: Two Sub-Models, One Name Space

LoRA fine-tuning of Qwen3-VL-8B-Instruct, and the measured memory curve that tells you how many frames fit on your card.

Example: 04_video_text/06_qwen3vl

This is the successor to Qwen2.5-VL fine-tuning, and the comparison is the lesson. It is not a version bump.

Scope

Qwen3-VL is video/image + text in → text out. If you want speech out, that is a different family — see Omni Models, which takes video and audio in and speaks back.

1. What actually changed between the generations​

Qwen2.5-VL-3BQwen3-VL-8B
classQwen2_5_VLForConditionalGenerationQwen3VLForConditionalGeneration
parameters3.75 B8.77 B
bf16 weights7.5 GB17.5 GB
vision tower0.67 B (17.8%)0.58 B (6.6%)
DeepStacknoyes — layers 8, 16, 24
MRoPEmropeinterleaved

DeepStack is the architectural one. Visual features are injected at several depths of the language model rather than only at the input embedding: the config carries deepstack_visual_indexes: [8, 16, 24], and the module tree carries a matching visual.deepstack_merger_list with one merger per index.

You can see all of this without a GPU and without downloading weights:

uv run verify_arch.py --compare      # ~8 seconds, meta device

2. The trap: a VL model is two sub-models with independent naming​

peft matches LoRA targets by name. A vision-language model has two sub-models that were named by different people, and the result is that the same target list means different things on different models.

Measured on the meta device:

targetQwen2.5-VLQwen3-VL
q_proj k_proj v_proj o_proj36 lang, 0 vision36 lang, 0 vision
gate_proj up_proj down_proj36 lang + 32 VISION36 lang only

Two separate surprises in that table.

Both vision towers use a fused qkv. So the conventional attention target list matches zero vision modules on either model. Freezing the vision tower is usually the right call — but doing it because a name failed to match is not the same as choosing it.

The MLP names collide on one model and not the other. Qwen2.5-VL's vision MLP reuses the language model's gate_proj/up_proj/down_proj, so adding those three targets unfreezes 32 vision modules you did not ask for. Qwen3-VL's vision MLP is linear_fc1/linear_fc2, so it does not.

Nothing in either config.json says any of this, and peft prints a plausible trainable-parameter count either way. Building the module tree on the meta device is two seconds and turns a guess into a fact.

3. What it costs, measured​

Not estimated — measured on one A40 (47.7 GB) by the lab's own --sweep:

framesvisual tokenspeak VRAM% of card
427622.1 GB46%
854025.3 GB53%
161,06831.6 GB66%
241,59637.9 GB79%

Which is very nearly a straight line:

peak≈18.8 GB+11.97 GB×visual tokens1000\text{peak} \approx 18.8\,\text{GB} + 11.97\,\text{GB} \times \frac{\text{visual tokens}}{1000}

At ~67 visual tokens per 224×224 frame:

cardusable
24 GB~4 frames — not viable
48 GB~33 frames
80 GB~70 frames

The 18.8 GB floor is the thing to remember. It is the bf16 weights plus fragmentation, it does not shrink with sequence length, and it is why this lab wants one large card rather than two small ones. A 24 GB card cannot hold the floor, let alone any frames.

4. Two ways this lab nearly lied, and what it does instead​

Both were real bugs in the first version, and both are the failure mode this course keeps returning to: the run exits 0 and the number is meaningless.

A sweep that measured one input four times. The first version passed frames as videos=, so the processor resampled every clip to its default frame rate and handed the model the same 128-token input at 4, 8, 16 and 24 frames. Peak VRAM was a flat 14.2 GB down the column. It looked exactly like a memory ceiling. The sweep now raises if the sequence length does not vary — a flat column across a 6× change in input is not a result.

A parameter count of zero. Under ZeRO-3, zero.Init partitions every parameter, so p.numel() returns 0 and summing it gives an 8.77 B model a parameter count of zero. The lab found out by dividing by it. It now counts via ds_numel, and the lesson generalises: the moment zero.Init fires, every naive bit of parameter accounting in your script changes meaning.

Related, and the reason the config is loaded before the model: the DeepSpeed config must exist before from_pretrained, or zero.Init never fires and every rank materialises the whole model. Holding HfDeepSpeedConfig in a live variable is what springs it.

5. Running it​

cd 04_video_text/06_qwen3vl
uv sync

uv run verify_arch.py --compare                          # no GPU needed
uv run deepspeed --num_gpus=1 train_qwen3vl.py           # needs 48 GB
uv run deepspeed --num_gpus=1 train_qwen3vl.py --sweep 8,16,32,48
Multi-GPU works, and the number is the lesson

Measured on 2 × A40: 10.1 GB per rank against the 8.8 GB that 17.5 / 2 predicts. The 1.3 GB gap is the overhead that does not shard — activations, gather buffers, fragmentation — which is why you budget per GPU as weights/N + overhead and never in aggregate.

It needed --no-p2p to get there. Two independent A40 boxes hung identically in WorkNCCL(SeqNum=6, OpType=BROADCAST) inside _zero_init_param without it. Reproducing on different hardware rules out a bad host: those boxes advertise peer-to-peer they cannot perform, and NCCL_P2P_DISABLE=1 is the documented fix. It costs throughput, and hardware with working P2P does not need it.

References​