From Tunix (JAX) to Native TorchTPU: Fine-Tuning Gemma 4 E2B-IT into an Insurance Expert with PyTorch on Google Cloud TPUs
In my previous deep dive on Tunix (JAX) (crownpku/tunix-gemma4-tpu), I shared the journey of fine-tuning Google’s Gemma 4 E2B-IT (5.1B total parameters, 2.3B active compute parameters) on the deccan-ai/insuranceQA-v2 dataset using Tunix (JAX/Flax/NNX) on a Google Cloud TPU pod. While we achieved a 51.4% drop in validation loss and transformed the model from a verbose academic generalist into a concise insurance advisor, getting there in JAX required navigating custom 4D attention mask broadcasting, manual NNX graph surgery to split kv_einsum LoRA weights during export, and 16 GB HBM memory walls triggered by Gemma 4’s massive 4.7 GB Per-Layer Embedding (PLE) table.
For the #TorchTPUSprint, I asked a direct engineering question:
What happens when we re-implement the exact same Gemma 4 E2B-IT insurance fine-tuning and evaluation pipeline using Google & Meta’s new Native PyTorch backend—
TorchTPU(torch_tpu+libtpu 0.0.53.1, PjRt API v0.117)—and Hugging FacePEFTon a single Cloud TPUv6e-1(Trillium) chip?
Here is the complete technical walkthrough, side-by-side architecture comparison, real hardware execution-mode benchmarks (887.8 tok/s → 14,291.3 tok/s), 500-step SFT convergence metrics, and the 5 real-world debugging walls we conquered along the way.
- Open-Source Repository (TorchTPU / PyTorch): github.com/crownpku/torchtpu-gemma4-insurance
- Companion Baseline Repository (Tunix / JAX): github.com/crownpku/tunix-gemma4-tpu
1. The Architecture: Inside Gemma 4 E2B-IT’s Safetensors
While Gemma 4 E2B-IT is compact enough for low-latency enterprise co-pilots, its internal architecture is far richer than a standard Llama-style decoder:
- The 4.7 GB Per-Layer Embedding (PLE) Table (
embed_tokens_per_layer): Alongside the standard[262144, 1536]token embedding table, Gemma 4 stores a[262144, 8960]per-layer embedding matrix (35 layers × 256 PLE dim) that injects token-specific gated residuals (per_layer_input_gate+per_layer_projection) at the end of every single decoder block. - Hybrid 5-Layer Attention Pattern (
4 Local Sliding : 1 Global): Layers4, 9, 14, 19, 24, 29, 34use full causal attention withhead_dim = 512,rope_theta = 1,000,000, and proportional RoPE (25%rotated dimensions). All other 28 layers use sliding-window attention (window = 512,head_dim = 256,rope_theta = 10,000). - KV Cache Sharing & Double-Wide MLPs (
Layers 15–34): The final 20 decoder layers reuse the post-norm, post-RoPE Key and Value tensors computed by Layer 13 (for local sliding layers) and Layer 14 (for global layers), while expanding their MLP intermediate dimension from6,144to12,288.
In gemma4_torchtpu.py, we implemented this exact architecture in native PyTorch (torch.nn.Module) matching all 600 model.language_model.* keys in google/gemma-4-E2B-it/model.safetensors:
- Zero-Copy Meta Initialization: We initialize the module structure on
torch.device("meta")before assigning thebfloat16tensors fromsafe_open(), skipping the 1,411 unused vision/audio encoder tensors and avoiding double-allocating 9.5 GB in host RAM. - Native
peftCompatibility: Unlike Tunix—which fuses K and V into a 4Dkv_einsumparameter ([2, K, D, H]) that breaks standard Hugging Face weight exporters—our PyTorch implementation keeps standardnn.Linearlayers (q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj).
2. Tunix (JAX/Flax) vs. TorchTPU (Native PyTorch): Developer Ergonomics
Porting our pipeline from Tunix to TorchTPU highlighted how much boilerplate disappears when Cloud TPUs behave like native PyTorch devices (torch.device("tpu")):
import torch
import torch_tpu # Registers the PrivateUse1 'tpu' backend with PyTorch
from peft import LoraConfig, get_peft_model
from gemma4_torchtpu import Gemma4ForCausalLMTorchTPU
device = torch.device("tpu")
base_model = Gemma4ForCausalLMTorchTPU.from_safetensors(
"models/gemma-4-E2B-it",
device=device,
dtype=torch.bfloat16,
)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.0,
bias="none",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(base_model, lora_config)
# Trainable params: 25,337,856 || All params: 4,672,787,747 || Trainable%: 0.5422%
| Dimension | Tunix_Gemma4_TPU (JAX / Flax / Qwix) |
TorchTPU_sprint (Native PyTorch + torch_tpu + PEFT) |
|---|---|---|
| Hardware Used | Cloud TPU v5litepod-4 (4× 16 GB chips) |
Single Cloud TPU v6e-1 Trillium (1× 32 GB chip) |
| LoRA Adapter Setup | qwix.LoraProvider (q_einsum, kv_einsum) |
Hugging Face peft.LoraConfig (25.34M trainable params / 0.5422%) |
| 500-Step SFT Train Loss | 5.2975 → 2.6825 |
5.3066 → 2.1437 (Step 490) / 2.6498 (Step 500) |
| Validation Loss Drop | 4135.72 → 2008.73 (-51.4%) |
4296.76 → 2046.27 (-52.4%) |
| Held-Out Test Perplexity | 192.41 → 23.15 (-88.0%) |
200.4320 → 21.9884 (-89.03%) |
| LoRA Weight Export | 154 lines of manual NNX key/transpose surgery | peft_model.merge_and_unload() in 11.84s (max_diff = 0.000977) |
3. Benchmarking TorchTPU’s 4 Execution Regimes on Cloud TPU v6e-1
One of TorchTPU’s strongest architectural features is giving developers explicit control over the trade-off between eager debuggability and whole-graph XLA compilation. Using bench_execution_modes.py, we benchmarked all four modes on static [BATCH_SIZE=4, SEQ_LEN=1024] batches (4,096 tokens/step) on a single Trillium v6e-1 chip:
| Execution Mode | Environment / API | Median Step Time (ms) | Mean Step Time (ms) | Throughput (tokens/s) | Speedup vs Strict Eager | Speedup vs Debug Eager | PJRT Cache Hit Rate |
|---|---|---|---|---|---|---|---|
| 1. Debug Eager | TORCH_TPU_LAUNCH_BLOCKING=1 |
4,613.53 ms |
4,613.66 ms |
887.8 tok/s |
0.26x |
1.00x |
99.76% (474 compiles) |
| 2. Strict Eager | Default (-O1 async op-by-op) |
1,191.23 ms |
1,192.19 ms |
3,438.5 tok/s |
1.00x |
3.87x |
99.76% (474 compiles) |
| 3. DeferAndFuse | TORCH_TPU_DEFER_AND_FUSE=1 (-O2) |
568.64 ms |
568.94 ms |
7,203.2 tok/s |
2.10x |
8.11x |
99.13% (395 fused clusters) |
| 4. Compiled | torch.compile(model, backend="tpu") |
286.61 ms |
286.99 ms |
14,291.3 tok/s |
4.16x |
16.10x |
7 whole-graph compiles, 0 dynamic recompiles |

Why Does Each Regime Jump by 2x–4x?
- Debug Eager → Strict Eager (
3.87xspeedup,4,613.5 ms→1,191.2 ms): RemovingTORCH_TPU_LAUNCH_BLOCKING=1allows the host Python thread to enqueue single-op PjRt executions asynchronously ahead of the TPU device. - Strict Eager → DeferAndFuse (
2.10xspeedup,1,191.2 ms→568.6 ms): Because Gemma 4 executes 4 RMSNorms, proportional/sliding RoPE rotations, and PLE gating per layer across 35 layers, Strict Eager dispatches474separate micro-graphs per step. SettingTORCH_TPU_DEFER_AND_FUSE=1buffers pending ops in a background window and fuses them into larger XLA clusters—doubling throughput to7,203.2 tokens/secwith zero code changes! - DeferAndFuse →
torch.compile(backend="tpu")(2.00xadditional speedup,568.6 ms→286.6 ms): Lowering the entire forward/backward pass via TorchDynamo into just7whole-graph OpenXLA executables eliminates per-cluster launch overhead entirely, reaching14,291.3 tokens/sec(16.10xfaster than Debug Eager).
4. 500-Step SFT Convergence & Quantitative Perplexity Verification
Using identical hyperparameters to our Tunix run (500 steps, batch_size = 4, seq_len = 1024, LoRA r = 16, alpha = 32, lr = 2e-5) in DeferAndFuse mode (~523 ms/step), our TorchTPU pipeline reproduced the rapid stylistic and factual convergence on deccan-ai/insuranceQA-v2:
- Step 1 Train Loss:
5.3066(40,973.0 msincluding initial cold Tier-3 PJRT compilation) -
Step 50 Train Loss: 3.2375(516.1 ms/step)Step 49 Validation Loss: 2427.1721(down-43.5%from Step 0’s4296.7616!) -
Step 250 Train Loss: 2.3835(522.4 ms/step)Step 249 Validation Loss: 2118.8318 -
Step 490 Train Loss: 2.1437(530.0 ms/step)Step 500 Validation Loss: 2046.2732(-52.4%total validation loss reduction!) - Full-Run PJRT Compilation Cache Stats:
num_cache_reqs = 2,343,615,num_cache_hits = 2,343,213 (99.9%),num_compilations = 402,num_unbounded_dynamic_hits = 0.

Quantitative Perplexity & Weight Verification (base_perplexity.log & perplexity.log)
Running eval_perplexity.py on data/test.jsonl before and after merging the Step 500 LoRA adapter (Layer 0 q_proj max weight delta: 0.000977) confirmed a massive improvement in domain language modeling:
-
Base google/gemma-4-E2B-it: Cross-Entropy Loss =5.3005Perplexity = 200.4320 -
TorchTPU Fine-Tuned gemma-4-E2B-it-finetuned: Cross-Entropy Loss =3.0905Perplexity = 21.9884(-89.03%reduction in perplexity!)
5. The 5 Real-World Engineering Walls We Hit on TorchTPU (and How We Solved Them)
No real systems project on a brand-new hardware backend succeeds without hitting edge cases. Here are the 5 concrete engineering walls we encountered and solved:
Wall #1: Child Subprocess /dev/vfio/0 Lock Contention in bench_execution_modes.py
- The Symptom: When benchmarking all 4 execution modes from a single parent script, importing
torchandtorch_tpuin the parent coordinator process before spawning child worker processes caused every child worker to crash onPJRT_Client_Createbecause the parent process already held the exclusive VFIO hardware lock on/dev/vfio/0. - The Fix: We kept the parent benchmark coordinator 100% pure Python stdlib (
subprocess,json,os) and deferredimport torchandimport torch_tpustrictly inside the child worker entrypoint (--worker-mode <mode>), allowing each mode (debug_eager,strict_eager,defer_and_fuse,compile) to initialize a clean PjRt client with its ownTORCH_TPU_*environment flags.
Wall #2: Single-Chip 32 GB HBM Exhaustion at [B=4, L=1024] (RESOURCE_EXHAUSTED)
- The Symptom: While Gemma 4 E2B-IT’s weights take ~9.5 GB in
bfloat16(including the 4.7 GB PLE table), running a 35-layer backward pass at[B=4, L=1024]without activation checkpointing—plus materializing a[4, 1024, 262144]float32logit tensor (4.3 GB) for cross-entropy—pushed peak HBM past 32 GB (RESOURCE_EXHAUSTED: the TPU ran out of memory while allocating tensor float32[4, 8, 1024, 1024]). - The Fix: In
gemma4_torchtpu.py, we enabledtorch.utils.checkpoint.checkpoint(..., use_reentrant=False)across the 35 decoder layers during training and implemented per-sequence chunked cross-entropy over the262,144vocabulary (computingF.cross_entropyone[1024, 262144]sequence slice at a time), cutting peak HBM usage well below 24 GB across all four execution modes.
Wall #3: torch.compile(backend="tpu") First-Step Compilation Latency (209.6s)
- The Symptom: In
compilemode (torch.compile(model, backend="tpu")), the initial warmup step traces all 35 gradient-checkpointed decoder layers + PEFT LoRA adapters + backward graph into 7 large OpenXLA executables, taking209.6 secondson cold compile—exceeding a standard 180s subprocess timeout. - The Fix: Enabling
TORCH_TPU_TIER3_COMPILATION_CACHE_ROOT=$HOME/.cache/torchtpu_pjrt_cachepersists compiled PjRt binaries across runs by fingerprint (be58cd7209a89904), and once compiled, step time drops to286.61 ms/step(14,291.3 tok/s) with0dynamic recompilations.
Wall #4: Hugging Face PEFT + Rust safetensors SafetensorError: device tpu is invalid
- The Symptom: In
export_model.py, callingPeftModel.from_pretrained(base_model, args.lora_dir)crashed with:safetensors._safetensors_rust.SafetensorError: device tpu is invalid - The Root Cause: When
torch_tpuis imported, it registerstpuvia PyTorch’sPrivateUse1mechanism, sopeft.utils.infer_device()seestorch.tpu.is_available() == Trueand returns"tpu".PeftModel.load_adapter()then callssafetensors.torch.load_file(filename, device="tpu")—which fails inside Rustsafetensorsbecausesafetensors’s Rust device parser only recognizes built-in upstream PyTorch device strings ("cpu","cuda","mps","xpu"). - The 1-Line Fix: Explicitly pass
torch_device="cpu"toPeftModel.from_pretrained(base_model, args.lora_dir, torch_device="cpu"), which loads the 97 MB LoRA adapter viasafetensorson"cpu"before merging all 600 tensors cleanly in11.84s!
Wall #5: Spot TPU VM Preemptions During Long Unbounded Validation Loops
- The Symptom: Running validation across all 838 validation batches (
3,354examples) every 50 steps added ~20 minutes of validation overhead to a 4.3-minute training run, exposing Spotv6e-1instances to preemption windows. - The Fix: Adding
--max-eval-batches 100(400 fixed-shape validation examples per 50-step checkpoint) kept every validation sweep under 15 seconds while preserving deterministic loss tracking across all 11 checkpoints (4296.76 -> 2046.27).
6. Qualitative Comparison: Base vs. Tunix (JAX) vs. TorchTPU (PyTorch)
Evaluating the merged checkpoint on 20 held-out InsuranceQA-v2 test questions (results/comparison_results.json) revealed three major improvements over the base Gemma 4 E2B-IT model:
1. Eliminating False-Positive Safety Over-Refusals on Legitimate Insurance Questions
- Query 20: “Can You Take Out A Life Insurance Policy On A Stranger?”
- Reference Expert Answer: “Being able to take life insurance out on a stranger puts people at risk for being the target of insurance fraud… Insurance companies have cracked down on this mightily by requiring that there is an insurable interest in order for someone to take out a policy on another…”
- Base Gemma 4 E2B-IT (
85 tokens, 5.21s):“I cannot assist with requests that involve taking actions on someone else’s life insurance policy, especially on a stranger. Life insurance policies are private contracts…” (False-positive safety refusal on a standard insurable-interest FAQ!)
- TorchTPU Fine-Tuned Gemma 4 E2B-IT (
39 tokens, 2.32s):“No , you cannot take out a life insurance policy on a stranger . Life insurance is a contract between the policyholder and the insurance company . The policyholder must be the person who is insured .”
2. Fixing Factual Hallucinations on Healthcare Regulation
- Query 9: “Does Obamacare Offer Medigap Insurance?”
- Reference Expert Answer: “No and luckily it is not needed . Medicare along with Medigap Insurance works great . Obamacare or the Affordable Care Act impacts folks under age 65 and not on Medicare…”
- Base Gemma 4 E2B-IT (
128 tokens, 7.50s):“Yes, Obamacare (the Affordable Care Act or ACA) does not require you to have Medigap insurance, but it allows you to purchase it. Here is a breakdown of the relationship between the ACA marketplace plans and Medigap…” (Factually incorrect!)
- TorchTPU Fine-Tuned Gemma 4 E2B-IT (
26 tokens, 1.57s):“Obamacare does not offer Medigap insurance . Medigap is a separate insurance product that is sold by a third party .” (Factually accurate and 4.8x faster!)
3. Quantitative Underwriting Conciseness (and Less Repetition than Tunix JAX)
- Query 6: “How Much More Is Life Insurance For Smokers?”
- Base Gemma 4 E2B-IT (
128 tokens, 7.55s):“The cost of life insurance for smokers is generally higher than for non-smokers. Here’s a breakdown of why and what factors influence the difference: ### Why It’s More Expensive…”
- Tunix (JAX) Fine-Tuned Gemma 4 E2B-IT:
“Life insurance is more expensive for smokers than non-smokers . The difference is usually 10% to 20 % . The difference is usually 10% to 20 % …”
- TorchTPU Fine-Tuned Gemma 4 E2B-IT (
42 tokens, 2.49s):“Life insurance rates for smokers are generally 20% to 30% higher than non-smokers . The exact percentage will depend on the individual , the type of policy , and the carrier .”
7. Final Takeaways from the #TorchTPUSprint
- PyTorch is Now a First-Class Citizen on Cloud TPUs: With
torch_tpu, porting a complex multimodal-derived architecture like Gemma 4 E2B-IT (with 4.7 GB Per-Layer Embeddings and KV sharing) from JAX/Tunix to Cloud TPUs uses 100% standardtorch.nn.Module,torch.optim.AdamW, and Hugging Facepeftidioms. TORCH_TPU_DEFER_AND_FUSE=1+torch.compile(backend="tpu")Deliver Massive Speedups: On a single Trilliumv6e-1chip,DeferAndFusedoubles training throughput over Strict Eager (3,438.5→7,203.2 tok/s) with zero code changes, andtorch.compile(backend="tpu")doubles it again to14,291.3 tok/s(286.61 ms/step).- Mind Ecosystem Boundaries (
safetensors+PEFT): When usingPrivateUse1backends liketorch_tpuwith Hugging Facepeft, passingtorch_device="cpu"duringPeftModel.from_pretrained()cleanly avoids Rustsafetensors’s device-name validation while keeping weight merging fast (11.84s) and PLE-safe.
Explore the full open-source implementation, logs, and benchmark suite at: https://github.com/crownpku/torchtpu-gemma4-insurance (and the companion JAX/Tunix baseline at https://github.com/crownpku/tunix-gemma4-tpu).