Md Shazid Islam
WACV 2027

LINGUAL

Language-INtegrated GUidance in Active Learning for Medical Image Segmentation

Md Shazid Islam1, Shreyangshu Bera1, Sudipta Paul2*, Amit Roy-Chowdhury1

1University of California, Riverside   2Samsung Research America

Animated overview of LINGUAL: polygon drawing versus a short language instruction, then the EXPAND and SHRINK operations on an MRI scan

Drawing an exact boundary around every uncertain region is slow. LINGUAL lets the annotator state what is wrong instead, such as “Expand to bottom-right”. An LLM turns that sentence into a program, and the program fixes the mask with no further human input. Accuracy stays close to manual drawing, and estimated annotation time drops by about 25%.

The problem

Active learning (AL) asks a human to label only the most useful regions of an image. Those regions still need dense pixel labels. An annotator usually places polygon vertices one by one along the boundary, and medical boundaries are often irregular and complex.

Existing options each fall short:

  • Polygon delineation is precise, but the cost grows with the region. A larger region has a longer boundary, so it needs more vertices.
  • Superpixel AL needs only one class label per superpixel, so nothing is drawn. But superpixels can cross true object boundaries. A superpixel that covers both organ and background still gets one label, which adds label noise.
Conventional AL refines regions by polygon delineation. LINGUAL refines the same regions with language feedback. The same instruction works for a small and a large region.
Drawing versus language. (a) Conventional AL traces each region by hand. (b) LINGUAL reaches the same correction from instructions like “Shrink to Left”. (c) A larger region (blue) needs many more polygon vertices than a smaller one (red), but both can use the same instruction.

This is the key idea. “Expand to bottom-right” takes the same effort for a small region and a large one, within a reasonable size limit. So human effort stays nearly constant as regions grow.

How LINGUAL works

LINGUAL runs inside active domain adaptation. A model pretrained on a source domain is adapted to a target domain over several rounds, under a fixed annotation budget per image. This setting fits well because EXPAND and SHRINK refine boundaries that the source model already predicts. A model trained from scratch has only noisy early predictions to refine.

LINGUAL workflow: the model predicts, the acquisition function samples regions, the annotator gives language feedback, the refinement block produces refined regions, and the loss updates the model
The workflow. (a) The model is updated during training, then frozen for testing. (b) The acquisition function picks regions, and each one gets language feedback and refinement. (c) Inside the refinement block, the Program Generator turns the feedback into a program and the Program Executor runs it.

1Acquisition: pick the uncertain regions

An acquisition function ranks candidate regions by uncertainty and keeps as many as the budget allows. LINGUAL does not depend on one choice. The paper tests it with Entropy, RIPU, ODES, and HALO.

2Language feedback: state the correction

The annotator looks at a region and names the operation and its direction. No pixel interaction is needed. One region can take several operations at once, for example “Expand at the top-right, remove the fragments at the bottom, and smooth the boundary.”

3Program generation: turn words into code

GPT-3.5 receives hand-written examples of instructions paired with programs. It then writes a program for the new instruction, using in-context learning. For example:

Instruction: Shrink to the Right. Smooth the boundary at Right

DIR0  = DIRECTION(keyword='shrink')
MASK0 = SHRINK(mask=MASK, direction=DIR0)
DIR1  = DIRECTION(keyword='smooth')
MASK1 = SMOOTH(mask=MASK0, direction=DIR1)
FINAL_RESULT = RESULT(var=MASK1)
Program Generator: in-context examples and a target instruction go to GPT-3.5, which outputs a program
Program Generator. In-context examples and the target instruction form the prompt. GPT-3.5 returns the program.

4Program execution: fix the mask

Seven correction modules cover the errors that appear inside a region. Two utility modules, DIRECTION and RESULT, support them.

Error in the regionModuleExample instruction
Under-segmentationEXPANDExpand to right
Over-segmentationSHRINKShrink at top-left
Spurious fragmentsREMOVERemove the small fragments
Internal holesFILLFill small gaps at the bottom
Jagged edgesSMOOTHSmooth the boundary at right
Whole region is objectFOREGROUNDEntire region foreground
Whole region is not objectBACKGROUNDEntire region background
Program Executor: each program line is shown with its output on a breast ultrasound image, ending in the refined segmentation
Program Executor. Each program line runs in turn on the ultrasound image, from the initial segmentation (Pinit) to the refined one (YAL).

5EXPAND and SHRINK: find where to stop

An instruction like “Expand to right” gives a direction but no distance. LINGUAL finds the distance with dual-sided boundary probing:

  • Sample seeds. Pick seed points just inside and just outside the predicted boundary, along the requested direction.
  • Grow regions. GraCo, a granularity-controllable segmentation model, groups similar pixels around each seed. This gives an inside region Ain and an outside region Aout. The algorithm can use any point-to-region function in place of GraCo.
  • Read the overlap. If both seed sets lie in the same organ, the boundary is still misplaced, so Ain and Aout overlap a lot. If they lie on opposite sides of the true boundary, they barely overlap.
  • Update the mask. EXPAND adds Ain ∩ Aout to the mask. SHRINK removes it. Then the steps repeat.
  • Stop at the best iteration. The overlap score is ηt = |Ain ∩ Aout| / |Ain ∪ Aout|. LINGUAL keeps the iteration with the lowest ηt. In the figure below, that is iteration 4 for EXPAND and iteration 2 for SHRINK.

GraCo runs at a low granularity (g = 0.1), so each iteration makes a small change. It works like a small step size in optimization and lowers the risk of over-correction. If no correction improves on the input, LINGUAL keeps the original mask.

Step by step EXPAND and SHRINK on an abdominal MRI: region selection, boundary seeds, inside and outside regions, iterations, overlap plot, and final mask
EXPAND and SHRINK, step by step. (a) Initial prediction. (b) Selected region, with true positives in orange, false positives in blue, and false negatives in green. (c) Boundary and direction. (d, e) Inside seeds in blue, outside seeds in yellow. (f) Ain in pink, Aout in violet, their overlap in brown. (g, h) Updates over iterations. (i) ηt per iteration. (j) Final mask at the lowest ηt.

6Update: adapt the model

The refined regions replace the sampled regions in the prediction map. The model is then updated with a supervised loss between its initial prediction and this refined map, plus any unsupervised loss, and the next round begins. After the last round, the adapted model is frozen for inference.

How feedback is generated in the experiments

To keep runs reproducible on public datasets, the feedback is generated from ground truth, which is standard practice in AL evaluation. It follows three stages:

  • Find the operation. Compare the prediction and ground truth inside the region. If false negatives dominate, choose EXPAND. If false positives dominate, choose SHRINK.
  • Find the direction. Map the vector from the boundary centroid to the error centroid onto one of eight directions.
  • Write the sentence. An LLM turns a keyword pair like “Expand top-left” into a natural instruction.

Ground truth is read only inside selected regions, and only to decide the operation and direction. The Program Generator and Executor never see it. The baselines use the dense pixel labels of the same regions, so LINGUAL works under much weaker supervision.

Results

LINGUAL was tested on three domain shifts across three imaging types:

  • MRI: CHAOS T1, in-phase to out-of-phase (liver, both kidneys, spleen).
  • Ultrasound: breast tumors, BUSI to UDIAT.
  • X-ray: lungs, Montgomery to Shenzhen.

The segmentation model is DeepLabv3+. Patch-based methods use a 5% area budget, and superpixel methods use a 25-superpixel budget.

0.53%average Dice gap to polygon delineation, with no pixel labels
1.38%average Dice gap to pixel-level ground truth in the same regions
+1.88%average Dice gain over superpixel AL
25.27%average reduction in estimated annotation time

Against patch-based AL

MethodIP→OOPBUSI→UDIATMont.→Shen.
Source model53.866.482.6
Random selection65.3±4.267.1±5.287.2±2.6
Fully supervised87.2±0.481.1±0.395.2±1.2
Entropy (GT ROI)85.5±0.579.2±0.994.9±0.4
Entropy (Poly)84.3±0.678.2±1.194.3±0.7
Entropy + LINGUAL83.2±0.677.9±0.993.5±0.5
RIPU (GT ROI)85.9±0.578.5±1.494.8±0.4
RIPU (Poly)84.9±0.877.4±1.994.3±0.3
RIPU + LINGUAL84.1±0.577.7±1.393.7±0.3
ODES (GT ROI)86.4±0.578.5±1.194.6±0.3
ODES (Poly)85.6±0.778.2±0.893.8±0.4
ODES + LINGUAL84.2±0.878.1±0.893.6±0.2
HALO (GT ROI)86.3±0.479.7±0.495.1±0.2
HALO (Poly)85.4±0.778.6±0.894.3±0.3
HALO + LINGUAL84.3±0.678.5±0.794.1±0.2
RIPU ROI + MedSAM81.9±1.176.9±1.591.8±0.6

Dice scores (%). IP→OOP is averaged over the four abdominal structures. GT ROI uses ground truth directly inside each region. Poly simulates polygon delineation from ground truth. Each LINGUAL row shares the regions of its baseline. The paper also reports HD95 and ASSD.

Both GT ROI and Poly get pixel-level supervision, while LINGUAL gets only language feedback. Even so, the gap is small. On the same RIPU regions, LINGUAL also beats MedSAM on Dice in all three shifts.

Against superpixel-based AL

MethodIP→OOPBUSI→UDIATMont.→Shen.
ASAL79.4±0.776.6±1.292.5±0.7
ASAL + LINGUAL80.5±0.977.4±0.593.6±0.5
MulActSeg78.1±1.374.1±1.591.3±1.2
MulActSeg + LINGUAL81.1±0.676.3±0.793.3±0.9
RSAL75.9±0.775.2±0.692.3±0.7
RSAL + LINGUAL79.2±0.777.5±0.793.4±0.5

Dice scores (%), 25-superpixel budget.

Adding LINGUAL improves all three methods on all three shifts. Its iterative corrections move toward the true boundary, instead of staying inside a fixed superpixel that may have leaked across it.

Estimated annotation time

Time is estimated from published benchmarks over each full dataset:

  • Polygon delineation: vertices are simulated with the Douglas-Peucker algorithm, at 5.55 seconds per vertex.
  • LINGUAL: about 4 seconds per region to decide the correction, plus speaking time at 90 words per minute, plus the automatic program generation and execution time.
ShiftPolygonLINGUAL humanLINGUAL autoLINGUAL totalSaved
BUSI→UDIAT13.797.901.789.6829.80%
IP→OOP42.9029.983.4933.4721.98%
Mont.→Shen.34.9023.902.6126.5124.04%

All values are estimates, in hours. Human is the time to decide and speak. Auto is program generation and execution. Saved compares the Polygon and LINGUAL totals.

Larger regions, same effort

Doubling the region size while keeping the number of regions raises Dice. Manual effort stays comparable, because it depends mainly on the number of regions.

CaseRegion areaNumber of regionsManual effortBUSI→UDIATIP→OOP
Baseline1×1×1×72.6265.30
Same total area2×0.5×0.5×69.2962.34
Same region count2×1×1×74.3170.81

Dice scores (%). The baseline uses 2 regions per round and 3 active rounds.

Program Generator robustness

The Program Generator was tested on 688 paraphrased instructions, across all seven operations and eight directions. They include formal and informal wording, alternative vocabulary, and combined operations. Unparseable or incorrect instructions fall back to the original prediction.

OperationSamplesParsed (%)Correct (%)
EXPAND12899.297.7
SHRINK128100.091.4
SMOOTH80100.097.5
FILL80100.098.75
REMOVE80100.098.75
FOREGROUND16100.0100.0
BACKGROUND16100.0100.0
Multiple operations160100.091.9
Overall68899.995.5

Budget and granularity

Two line charts on IP to OOP. Dice rises quickly with budget and plateaus after about 5 percent. Dice peaks at granularity 0.10.
Ablations on IP→OOP. (a) Dice rises quickly at low budgets, then plateaus. (b) Dice peaks at granularity 0.1. Too low, and Ain and Aout barely overlap, so refinement stops. Too high, and the regions spill past the true boundary.

Limitations

  • LINGUAL refines the source model’s predictions, so its efficiency depends on their quality. It still helps from poor starts: Dice rises from 53.8 to 84.1 on IP→OOP with RIPU.
  • Corrections can be weaker in regions with mixed false positives and false negatives, branching, or disconnected structures.
  • The results concern annotation effort and accuracy. Clinical acceptability for a specific task still needs validation with clinicians.

Code

The code is open source at github.com/ShazidAraf/LINGUAL. It needs Python 3.10, PyTorch 2.0.1, CUDA 11.8, one GPU, and an OpenAI API key for program generation.

git clone https://github.com/ShazidAraf/LINGUAL.git
cd LINGUAL
conda env create -f environment.yml
conda activate lingual
export OPENAI_API_KEY=<your-key>
python main.py --cfg configs/UDIAT.txt --method RIPU

Download the GraCo weights, add a source model checkpoint, and set your paths in the config file. The README has the details. Configs are included for all three domain shifts.

Citation

@inproceedings{islam2027lingual,
  title     = {LINGUAL: Language-INtegrated GUidance in Active Learning
               for Medical Image Segmentation},
  author    = {Islam, Md Shazid and Bera, Shreyangshu and Paul, Sudipta and
               Roy-Chowdhury, Amit},
  booktitle = {Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

* Sudipta Paul contributed in his personal capacity. The views expressed are his own and do not necessarily represent those of Samsung Electronics.