Feature 01 · AI Parser Benchmarking

Which model turns court records into clean JSON?

Given one shared prompt and the same 13 record types from two real PTUN Kupang cases, qwen3-32b produced the cleanest, most consistent, and most complete JSON of the five self-hostable models — the one you can point a stable parser at today.

◆ Recommended for self-hosting

qwen3-32b

It returns valid JSON on every field, keeps the same predictable shape from one case to the next — so downstream code can rely on a single schema — and captures the large tables in full where the lighter models dropped rows. Of the five, it needs the least normalization work before the output feeds a database.

One honest constraint. To self-host affordably you would quantize the model. The quantized build (qwen3-32b-awq) handles 24 of 26 fields cleanly but truncates the single largest field — Biaya Perkara, roughly 23,000 characters of source HTML — because that one field's output overruns the generation window under quantization. This is an infrastructure limit on field size, not a limit on the model's parsing. Chunking that one oversized field, or giving the model more GPU headroom to run at full precision, resolves it.

The five models, in plain terms

How each model behaved

One verdict per model, ordered by how ready the output is for production. No scores — this is about what a backend engineer would actually have to deal with.

qwen3-32b

Recommended Valid on all 26 Most consistent Full-fidelity tables

The most directly usable. Valid JSON on every field, and it keeps the same flat, predictable shape from one case to the next, so one schema covers both. It also captured the largest tables in full where lighter models gave up. Evidence — Data Umum returns an identical set of top-level keys for both cases, and Biaya Perkara (the largest field) came back complete with about 76 rows.

mistral-small-3.2-24b

Valid on all 26 Faithful Structure drifts

Strong and complete — valid on every field and faithful on the big tables — but its structure drifts between cases. You would write normalization code before trusting a single schema. Evidence — it wrapped Data Umum in an infoPerkara object for one case and left it flat for the other, and switched key casing on Banding from data_pemohon_banding to dataPemohonBanding.

qwen3.5-9b

Valid on all 26 Lossy on large tables

Always valid and easy on the eyes, but it stays tidy partly by leaving data out. Fine for small key-value fields; lossy on the big ones. Evidence — on Biaya Perkara it returned about 30 rows where the stronger models returned roughly 76.

qwen3-32b-awq

Quantized build Clean on 24 of 26 Truncates largest field

Structurally close to full-precision qwen3-32b on the 24 fields it completed, but the quantized build ran out of generation room on the single largest field and returned truncated JSON there. An infrastructure limit on that one field, not a parsing weakness. Evidence — Biaya Perkara was cut off mid-array in both cases; every other field parsed cleanly.

gemma-3-27b-it

HTML artifacts Inconsistent wrapping Most cleanup

Carries HTML scaffolding into the output — container names become top-level keys — wraps inconsistently between cases, and over-splits some tables. The most cleanup work before it is production-ready. Evidence — the same Data Umum data appears under a tabs1 wrapper, and Kasasi was wrapped as detilkasasi in one case but returned flat in the other.

The difference, in JSON

Same field, same source — different output

All excerpts are real output from the run, trimmed to a few lines. The field is Data Umum (general case info) throughout.

1. Clean vs. HTML scaffolding. The recommended model returns the real fields at the top level. gemma buries them under the HTML container's id.

qwen3-32bdirectly usable

{
  "Nomor Perkara": "10/G/2018/PTUN.KPG",
  "Klasifikasi Perkara": "Pertanahan",
  "Penggugat": [
    { "Nama": "HENDRIKUS CHANDRA" }
  ],
  "Kuasa Hukum Tergugat": null
}

gemma-3-27b-itneeds cleanup

{
  "tabs1": {          // HTML id, leaked as a key
    "Nomor Perkara": "10/G/2018/PTUN.KPG",
    "Penggugat": [
      { "Nama": "HENDRIKUS CHANDRA" }
    ]
  }
}

2. Stable vs. drifting shape. A parser needs the same structure every time. mistral wraps the object in one case and drops the wrapper in the next — so the same field needs two code paths.

mistral-small · case 10/2018wrapper present

{
  "infoPerkara": {
    "Nomor Perkara": "10/G/2018/PTUN.KPG",
    "Klasifikasi Perkara": "Pertanahan"
  }
}

mistral-small · case 14/2024wrapper gone

{
  "Nomor Perkara": "14/G/2024/PTUN.KPG",
  "Klasifikasi Perkara": "Pertanahan"
}
// same field, same prompt — different top-level shape

Method, briefly

What we tested and how we judged it

One shared extraction prompt was applied to 13 record types — from simple key-value blocks like Data Umum to large nested tables like Biaya Perkara and Kasasi — across two real cases from the PTUN Kupang SIPP portal. Every model saw the exact same prompt and the exact same source HTML for each field. We then read the raw JSON each model returned and judged it on four things:

Valid, every time — does it parse as JSON without repair?
Stable structure — same keys and same shape from one case to the next, so one schema fits all?
Easy to parse against — no stray wrapper objects or HTML artifacts a backend engineer has to strip out?
Faithful to the source — every label→value pair and every table row captured, nothing dropped or invented?

Models

Record types

Real cases

Shared prompt