Changing Annotation Guidelines Mid-Project

The annotation team is partway through a dataset when the requirement changes. Yesterday, a label was acceptable. Today, the same label may be too broad, incomplete, or drawn to the wrong boundary. Updating the instructions is necessary. It does not settle what happens to the work already delivered.

The practical question is not just how quickly annotators can learn the new rule. It is whether old labels, work in progress, and future labels will still mean the same thing when they reach training and evaluation. A small edit can create a large review queue without adding a single source file.

This guide treats the change as an operational decision, not a documented Northern Base project. The vehicle example below is hypothetical. No project volumes, prices, savings, or performance improvements are assumed.

An edited document does not update historical labels

A guideline describes the intended decision; an annotation records a decision already made. Changing one does not automatically change the other. The guide to writing annotation guidelines covers preparation before scale. Once work exists, the additional task is to establish compatibility between releases.

Separate completed batches, active assignments, and work not yet started. Record which rule version each group is using. Where a change affects active work, pause or isolate that queue until a cutover is agreed. Unaffected work can continue if its boundaries are genuinely clear.

This distinction also exists in tools. Labelbox's ontology documentation warns that previous annotations can remain after a schema edit without meeting the revised structure. Tool configuration is therefore not proof that historical data has been migrated.

Classify the change by meaning, not document length

Low impact: a wording correction preserves the decision boundary. Check representative disputed examples before calling it cosmetic. A clarification that changes how reviewers interpret an existing sentence is a semantic change, even if the editor calls it a typo.

Medium impact: an added attribute or inclusion rule affects a defined subset. Existing records may remain useful, but some are incomplete under the new requirement. Determine whether the missing information can be recovered from retained evidence.

High impact: a class split, ontology restructure, or geometry change alters the target substantially. A merge can be mechanically straightforward yet still change model outputs and evaluation meaning. These are planning categories, not measured effort bands; impact depends on coverage and downstream use.

Start with an affected-data inventory

Write down the old decision and the new decision side by side, with examples that produce different answers. Then identify candidate records by label, source, batch, guideline version, and annotation status. Searching only the changed class name can miss objects previously excluded by policy.

Count affected records separately from affected annotations. One image may contain several objects; one record may require multiple edits. Distinguish an inventory count from a review estimate. Until examples have been inspected, neither proves how much manual work is required.

Include train, validation, and test releases, exports already delivered, tool templates, and any model-output mappings. An annotation audit can help locate inconsistent historical decisions, but a clean audit under the old rule does not establish compliance with the new one.

Hypothetical example: splitting a vehicle class

Suppose the original classes are CAR, TRUCK, BUS. The revised classes are CAR, LIGHT COMMERCIAL VEHICLE, HEAVY TRUCK, BUS. The request appears to add one class. In practice, it asks for a distinction the original labels did not record.

An old TRUCK identifier alone cannot tell you which new class applies. Reviewers need the source image or trustworthy metadata that captures the approved distinction. If the distinction depends on a property not observable in the image, visual review cannot manufacture it. The owner must supply evidence, define an uncertainty rule, or accept exclusion.

Do not assume that only TRUCK records are affected. If vans previously counted as CAR but now belong to LIGHT COMMERCIAL VEHICLE, the candidate set extends into CAR. BUS may be unchanged, but that too follows from the definitions, not the spelling of the label.

Preserve the original release. Create a candidate revision with stable record IDs, original class, proposed class, rule version, and disposition. A reviewed example can move forward; an ambiguous one needs a recorded decision. Keep that distinction visible instead of assigning every old TRUCK label a default new class.

Migration is safe only when the missing decision is recoverable

Deterministic migration means the available information uniquely determines the new annotation under an approved rule. It is not a model prediction that seems plausible. A pure rename may qualify. A many-to-one merge may qualify if boundaries, attributes, and relationships remain compatible.

A split usually requires information beyond the old class. Reliable retained attributes might supply it; otherwise review the original source. Model suggestions can prioritize or assist that review, but should not silently become verified ground truth.

Test any mapping on representative cases before applying it broadly. Preserve the input release and mapping revision, check counts and identifiers, and make reruns predictable. Repeated execution must not duplicate annotations or transform already-migrated values again.

For changed image annotation geometry, check whether the original pixels and coordinates still exist. A tighter-box rule cannot necessarily be implemented by shrinking every box by a constant amount. Missing source evidence may make reannotation impossible rather than merely expensive.

The work extends beyond the changed labels

Build the estimate from activities: impact analysis, tool configuration, migration implementation, manual reannotation, review, recalibration, QA updates, export regeneration, evaluation-set work, and documentation. Some costs occur once; others depend on the affected subset or unresolved cases.

Separate correcting work that violated the original agreement from producing work under a new agreement. They can look identical in an editing queue while having different approval and commercial implications. Agree on that distinction before counting all edits as defects or all edits as additional scope.

The annotation cost-planning guide explains broader pricing considerations. For this change, record assumptions rather than inventing a rate: source availability, mapping confidence, review depth, and export dependencies. The number of new labels is not the total amount of work.

Recalibrate before restarting the affected queue

A revised document can be read without the revised boundary being understood. Give annotators changed examples and counterexamples, including cases where the old decision remains valid. Ask them to apply the new rule to a small calibration batch independently.

Review disagreements against the intended distinction. Is the rule unclear, is evidence missing, or is the interface still presenting an old option? Resolve the cause before treating disagreement as a training failure. Update the examples and repeat the relevant checks when necessary.

Record who is cleared for the changed task and which revision they used. A reviewer calibrated on the old classes is not automatically calibrated on the new split. Unaffected tasks need not wait if the workflow prevents accidental mixing.

Previous QA remains evidence for the previous rule

A batch that passed yesterday was assessed against yesterday's criteria. Preserve that result; do not erase it or reinterpret it as approval of new semantics. Assess what must be rechecked using the revised checklist and reference examples.

Check migrated labels as well as manually edited ones. Inspect class counts, required fields, relationships, and examples near the new boundary. Unexpected distribution changes may reveal a bad mapping, though a valid split can also change class proportions intentionally.

The annotation QA guide covers general review design. Here the release decision is narrower: have all candidate records been accounted for, do accepted annotations follow the new rule, and are exceptions explicit? A technically valid export is not sufficient evidence of semantic correctness.

Version the decision trail as well as the dataset

Keep Guideline v1 associated with Dataset v1. Associate Guideline v2 with the accepted revised dataset, along with the cutover date and change approval. A wording-only correction may require a guideline revision without regenerating identical labels; record that compatibility explicitly.

The release manifest should identify source records, schema IDs, guideline revision, migration revision, manual-review status, exclusions, QA approval, and split membership. Record the exact dataset release used by each model run. Version names alone are insufficient if files behind them can change silently.

DVC's data and model versioning documentation illustrates keeping data and code revisions recoverable together. The operational requirement is reproducibility, not a particular product. Preserve old releases subject to applicable retention and access restrictions.

Keep label semantics consistent across evaluation splits

If training predicts two new vehicle classes while validation still expects TRUCK, an apparent error may be a label-space mismatch. Update evaluation definitions deliberately. Do not compare the resulting score directly with an old score as though only the model changed.

Preserve train, validation, and test membership where possible. Relabeling is not permission to move difficult test records into training or tune new rules against model-specific test failures. Keep held-out review controlled by an evaluation owner and retain provenance for related records.

A documented projection back to a shared coarse taxonomy may permit a limited historical comparison. It does not measure the new fine-grained distinction. Report which label space each result uses. If no defensible comparison exists, establish a new benchmark and state the discontinuity.

A decision table for the change request

These are planning examples, not automatic approvals. A new dataset release is warranted when accepted contents or semantics change; an unchanged export can instead reference a compatible guideline revision.

Before approving the change

• Name the decision owner and freeze the old rule and dataset references.
• Identify completed, active, and future work affected by the request.
• Approve mapping, review, uncertainty, and exclusion decisions.
• Account for recalibration, tool configuration, QA, and evaluation work.
• Confirm split integrity, a recoverable release, and a named restart approver.

A guideline change is complete when its consequences are accounted for, not when its document is circulated. The useful deliverable is a dataset whose accepted labels can be explained under a known rule, with unresolved work visible rather than hidden in a default value.

Questions about changing annotation requirements