Training Segmentation Models: SAM, U-Net, SegFormer

With over 7 years of experience and 50+ computer vision projects completed, we specialize in image segmentation model training. Choosing an architecture like U-Net, SegFormer, or SAM is always a trade-off between accuracy, speed, and annotation volume. Clients often bring a dataset of 200 CT scans expecting mIoU >0.9, but in reality U-Net works while SegFormer lacks data. Or vice versa: an industrial pipeline with 5000 images where SegFormer gives +8% mIoU but latency is critical. We have accumulated experience training segmentation models for medical (organ, vessel, tumor segmentation) and industrial tasks (conveyor defects, cracks, quality control). Over years of work, we have completed over 50 computer vision projects, and in each case the architecture choice was determined by three factors: data volume, required contour accuracy, and allowable inference latency. Average inference cost savings after optimization reach up to 60% of operational costs — for example, saving $10,000 annually on cloud inference for a mid-sized deployment.

U-Net and SegFormer are the most popular architectures, each with its strengths. The former is robust to small data, the latter is a leader in mIoU on large datasets. SAM2 adds interactivity and fine-tuning with minimal annotation.

Which Architecture to Choose: U-Net, SegFormer, or SAM?

U-Net and Its Variants — Medical and Industrial

U-Net remains the standard for medical segmentation not because of quality (SegFormer is better), but due to robustness with small datasets (100–500 images) and interpretability. We recommend U-Net++ for tasks with fine structures (vessels, cracks).

import torch
import torch.nn as nn
import segmentation_models_pytorch as smp

def build_unet_model(
    architecture: str = 'Unet',
    encoder: str = 'efficientnet-b4',
    encoder_weights: str = 'imagenet',
    num_classes: int = 1,
    in_channels: int = 3
) -> nn.Module:
    model = getattr(smp, architecture)(
        encoder_name=encoder,
        encoder_weights=encoder_weights,
        in_channels=in_channels,
        classes=num_classes,
        activation=None
    )
    return model

class CombinedLoss(nn.Module):
    def __init__(self, dice_weight: float = 0.5, bce_weight: float = 0.5):
        super().__init__()
        self.dice_weight = dice_weight
        self.bce_weight  = bce_weight
        self.bce = nn.BCEWithLogitsLoss()
        self.dice = smp.losses.DiceLoss(mode='binary', from_logits=True)

    def forward(self, preds: torch.Tensor,
                targets: torch.Tensor) -> torch.Tensor:
        return (self.bce_weight  * self.bce(preds, targets) +
                self.dice_weight * self.dice(preds, targets))

SegFormer — Semantic Segmentation with Focus on Accuracy

SegFormer-B4 outperforms U-Net on most semantic segmentation tasks when the dataset >1000 images. On ADE20K it achieves mIoU 50.3%, which is 1.2x better than U-Net with EfficientNetB4 (42.1%). For tasks with high contour detail requirements, we use SegFormer-B5 with polynomial learning rate decay.

from transformers import SegformerForSemanticSegmentation, SegformerConfig
import torch
import torch.nn.functional as F

def train_segformer(
    num_labels: int,
    id2label: dict,
    label2id: dict,
    pretrained_model: str = 'nvidia/mit-b4',
    learning_rate: float = 6e-5,
    num_epochs: int = 50
) -> SegformerForSemanticSegmentation:

    model = SegformerForSemanticSegmentation.from_pretrained(
        pretrained_model,
        num_labels=num_labels,
        id2label=id2label,
        label2id=label2id,
        ignore_mismatched_sizes=True
    )

    optimizer = torch.optim.AdamW(
        model.parameters(),
        lr=learning_rate,
        weight_decay=0.01
    )

    scheduler = torch.optim.lr_scheduler.PolynomialLR(
        optimizer,
        total_iters=num_epochs,
        power=0.9
    )

    return model, optimizer, scheduler

def segformer_inference(
    model: SegformerForSemanticSegmentation,
    pixel_values: torch.Tensor,
    target_size: tuple = None
) -> torch.Tensor:
    outputs = model(pixel_values=pixel_values)
    logits = outputs.logits

    if target_size is not None:
        logits = F.interpolate(
            logits,
            size=target_size,
            mode='bilinear',
            align_corners=False
        )
    return logits

SAM2 Fine-Tuning for Custom Domains

SAM2 out of the box does not know specific classes (microscopy, industrial defects). Fine-tuning only the decoder mask head is an efficient approach. With a frozen encoder, only ~4M parameters out of 224M are trained, allowing the model to be trained on a single GPU in one day. This requires 5x less data than training U-Net from scratch.

from sam2.build_sam import build_sam2
import torch

def finetune_sam2_decoder(
    checkpoint_path: str,
    num_epochs: int = 30,
    learning_rate: float = 1e-4,
    freeze_image_encoder: bool = True,
    freeze_prompt_encoder: bool = True
) -> torch.nn.Module:

    sam2 = build_sam2(
        'sam2_hiera_large.yaml',
        checkpoint_path,
        device='cuda'
    )

    for param in sam2.image_encoder.parameters():
        param.requires_grad = not freeze_image_encoder
    for param in sam2.prompt_encoder.parameters():
        param.requires_grad = not freeze_prompt_encoder

    for param in sam2.mask_decoder.parameters():
        param.requires_grad = True

    trainable_params = sum(
        p.numel() for p in sam2.parameters() if p.requires_grad
    )
    print(f'Trainable parameters: {trainable_params:,}')

    optimizer = torch.optim.AdamW(
        filter(lambda p: p.requires_grad, sam2.parameters()),
        lr=learning_rate,
        weight_decay=1e-4
    )

    return sam2, optimizer

Why SAM Fine-Tuning Is More Efficient Than Training from Scratch

The pretrained SAM2 image encoder contains generalized shape and boundary features that transfer to a new domain. This reduces data requirements by 5–10x compared to training U-Net from scratch. Additionally, SAM supports interactive segmentation — you can fine-tune the model for specific prompts (points, boxes).

Work Process: Step by Step

1. Data analysis and prototyping. We assess annotation quality, class distribution, possible artifacts. Choose architecture and augmentation format. Deliverable: technical specification with metric forecast.
2. Training pipeline setup. Assemble preprocessing, augmentation (Albumentations, torchvision), logging in MLflow. For medical tasks, we use combined loss (Dice + Focal).
3. Training and experiments. Run grid search on learning rate, weight decay, batch size. Save best checkpoints. For SegFormer, use polynomial LR scheduler; for U-Net, ReduceLROnPlateau.
4. Inference optimization. Export to ONNX, INT8 quantization, profile on target GPU. Achieve p99 latency below the client's threshold.
5. Integration and documentation. Deploy model in Docker container, prepare REST/gRPC API. Write model card: architecture description, test metrics, limitations.

Segmentation Metrics

Metrics definitions from Wikipedia

Architecture Comparison

What's Included

• Analysis and prototype: data audit, architecture selection, achievable accuracy estimate.
• Pipeline development: preprocessing, augmentation, training with MLflow logging.
• Inference optimization: export to ONNX/TensorRT, INT8 quantization, latency reduction to p99.
• API integration: REST/gRPC endpoint, Docker containerization, Kubernetes deployment.
• Documentation: model card, reproducibility instructions, metric descriptions.
• Team training: workshop on model usage and support.

Timeframes and Pricing

Get a consultation on architecture selection — it's free. If you have a dataset, order a data audit: we'll estimate achievable accuracy and pick the architecture.

Image segmentation