Imagine you're building a pub trivia game, but instead of writing questions yourself, you hire a very smart friend (GPT-4o) to read the rulebook and generate question-answer pairs, then you run nine automated sanity checks, get two other smart friends to review, and finally you personally spot-check 200 pairs. That's the construction pipeline here — and the quality of the game depends entirely on how good that pipeline is. The committed claim: IndicIPR-QA is the first extractive QA benchmark that provides exact answer spans from statutory text across all seven Indian Intellectual Property Rights Acts. This is a dataset contribution, not an algorithmic one. The seven Acts cover patents, trademarks, copyrights, designs, geographical indications, semiconductor IC layout, and plant varieties — a breadth no prior Indian legal QA dataset has attempted. The corpus yields 4,462 question-answer pairs drawn from 1,116 statutory contexts, with section-level partitioning to prevent train-test leakage. The ladder is straightforward but revealing. BM25, a pure lexical retrieval baseline, achieves a token-level F1 of 20.21. Fine-tuned InLegalBERT — a BERT variant pre-trained on Indian legal corpora — reaches 85.02 F1 under a unified sliding-window inference protocol. The 65-point gap is the paper's central empirical finding: legal QA over statutory text requires semantic understanding, not just keyword matching. Performance varies meaningfully across the seven Acts, suggesting structural differences in legal drafting style affect extractability. Architecturally, this is a standard extractive QA setup in the SQuAD mold: given a context passage and a question, predict start and end token positions of the answer span. The sliding-window protocol handles long statutory sections that exceed BERT's 512-token limit. The generation side uses GPT-4o as a question-answer generator, which is increasingly common in benchmark construction but creates a circularity concern: LLM-generated QA pairs may encode LLM-friendly patterns that inflate performance for transformer-based models. Integrity is mixed. The nine automatic quality checks (checking answer span presence, deduplication, length bounds) are sensible engineering. The multi-model LLM evaluation adds a layer. The 200-pair human audit is the load-bearing validation — but 200 out of 4,462 is 4.5%, and we don't know the inter-annotator agreement or the credentials of the auditors beyond being human. Section-level partitioning is a genuine strength that many legal QA datasets skip. No code release is mentioned. The benchmarks were not pre-registered. The milestone question for legal QA benchmarks is adoption: does anyone else use this dataset? The immediate next number is whether cross-domain transfer works — can a model trained on IndicIPR-QA generalize to other Indian statutory domains (tax, criminal, environmental law)? If the F1 holds above 70 on out-of-domain statutes, this becomes a foundation for Indian legal NLP. If it collapses, the benchmark is measuring Act-specific memorization. The obvious experiment not run: testing GPT-4o or other large language models directly on the benchmark in a zero-shot or few-shot extractive setting. The authors used GPT-4o to generate the dataset but did not evaluate it as a QA system. The likely reason is (c) — saving it for the next paper, since the comparison would reframe this from a dataset paper into a methods shootout and potentially show that the generator already knows the answers, which would undermine the benchmark's value.
