3rd February, 2026

Task–Specificity Score: Measuring How Much Instructions Really Matter for Supervision

EMNLP 2026

Pritam Kadasi, Abhishek Upperwal, Mayank Singh

📄 Paper: arXiv:2602.03103

Introduction and Motivation

Instruction tuning is now the default way to train and adapt large language models. But many instruction–input–output examples are only weakly specified: for a given input, the same output would still make sense under several different instructions. A factual answer might satisfy both "answer the question" and "summarize the passage"; a short label might fit several classification prompts.

Such examples are easy for models to learn, but they teach little about how instructions shape behavior. Training heavily on them can waste budget and blur task boundaries.

Most data-selection methods focus on output quality or model fit, such as perplexity filtering or instruction-following difficulty (IFD). We ask a different question:

Does the instruction uniquely determine the target output? In other words, would this output still make sense if the instruction were changed?

Method

Task–Specificity Score (TSS)

For each training example (instruction I, input X, output Y), we:

1. Ask a model to generate K plausible alternative instructions for the same input X, without showing it the output Y.
2. Score how likely Y is under the true instruction, using a frozen language model.
3. Score how likely Y is under each alternative instruction.
4. Compare the two:

TSS(I, X, Y) = log p(Y | I, X) − log( average over k of p(Y | I_k, X) )

A high TSS means the output is much more likely under the true instruction than under the alternatives: the instruction really matters. A low or negative TSS means the output fits many other tasks just as well, so the supervision is weakly specified.

Seen through information theory, TSS is a Monte Carlo estimate of the conditional mutual information between the instruction and the output given the input, which is otherwise intractable for large language models.

TSS++: quality-aware contrastive specificity

Plain TSS has two practical weaknesses on structured instruction datasets: easy alternatives can inflate the score, and specificity alone can underweight output quality, where perplexity filters are strong. TSS++ addresses both:

• Hard alternatives: it keeps only the alternative instructions that are most confusable with the true one.
• Contrastive scoring: an InfoNCE-style score based on length-normalized likelihoods, so the result depends on how the true instruction compares with those hard alternatives.
• Quality term: it adds a small weight on the output's likelihood under the true instruction.

Using the score

• Filtering: keep the top fraction of examples by score, for example the top 5% under a tight budget.
• Weighting: keep everything, but give each example a weight that grows with its score. These variants are marked (E) below.

Experimental Setup

Results

Small budgets: 5% of the data

At 5% retention, the best method beats a random subset in 8 of 9 settings, with an average gain of +0.703 SUM. TSS++ wins most often (6 of 9), and plain TSS wins 2 more.

15% of the data

At 15% retention, the best method improves on random in 6 of 9 settings, with an average gain of +1.253 SUM. The largest gain is +2.881 SUM, for Qwen on NI-20 with weighted TSS. As the budget relaxes, quality signals start to catch up, but instruction specificity stays a strong and complementary signal.

Less data can beat all the data

In 8 of 9 cases, the best result overall comes from training on less than the full dataset. Llama on Alpaca peaks at 147.304 SUM with weighted TSS++ at 45% of the data, +7.916 above training on everything.

Budget sweeps

Across the full range of budgets, an instruction-specificity method (TSS or TSS++) forms the top curve in each of these three representative settings.

SUM score as a function of the fraction of data kept, for each selection method. The dashed line is training on the full dataset.

Which method wins at each budget

Instruction-specificity methods win much of the low-to-mid budget range, while random and quality-based methods appear more often as more data is kept. Specificity matters most when the budget is tight; quality and coverage matter more as the budget grows.

Gains are broad, not from one benchmark

For Llama on Alpaca at 45% of the data, the winning TSS++ configuration improves all four benchmarks at once.

Conclusion

The Task–Specificity Score measures whether an output is uniquely tied to its instruction, compared with other plausible instructions for the same input. TSS++ makes this more robust with hard alternatives and a quality term. Across three instruction datasets and three open models, prioritizing high-specificity examples improves results under tight budgets and complements quality-based filters.

The broader point: data curation for instruction tuning should ask not only whether outputs are easy or likely, but whether the instruction actually constrains them.

Limitations

Scores depend on the scoring model and on the quality of the alternative instructions. Scoring needs extra forward passes for every alternative, which adds cost on very large datasets. The experiments cover three datasets, small open models and four benchmarks, so results may not transfer to every setting. TSS does not by itself guarantee factuality or safety.

Read the full paper on arXiv: Task–Specificity Score: Measuring How Much Instructions Really Matter for Supervision