.. _seo, ec_manm_segmentation:
Segmentação Semântica e o *Dataset*
===================================
no discussãoobr os problemas de detecção de objetos nas seções
anteriores, usamos apenas caixas delimitadoras retangulares para
rotulares ola pre objets em imagens. Nesta seção, veremos a segmentação
semâo semntica, que tenta, tet segmentar imagens res m dierentes
categorias semânticas. Essas regiões semânticas rotulam e prevêem
objetos no nível do pixel. :numref:`fig_segmentation` ostra uma
imagemeamente segmentada, com áreas marcadas como “cachorro” , “gato” e
“fundo”. Como você pode ver, em comparação com a detecção de objetos, a
segmentação semântica rotula áreas com bordas em nível de pixel, para
uma precisão significativamente maior.
.. _fig_segmentation:
.. figure:: ../img/segmentation.svg
Imagem segmentada semanticamente, com áreas rotuladas
“cachorroSemantiatiamente mio”, “gcato” e “plano de f”oundo".
Segmentação de Imagem e Segmentação detion and Instâancia
---------------------------------------------------------
No campo da visão computacional, existem dois métodos importantes
relacionados à segmentaçãe Segmenntica: segmentação detion: imagens e
segmentação detion and instâancias. Aqui, vamos distinguir esses
conceitos da segmentação semântica da seguinte forma:
- A segmenta imagmgmen divide umas an imagem em várias regiões into
sconstituente mét regioodo geramente usa as correlações entrel pixels
ain an imagem. Durante oing treainamento, os rótulos não são
necessários para pixels de imagem. No entanto, durante a previsão,
esse método não pode garantir que as regiões segmentadas ain semntic
imagem em .10, amage segmentação da imagem pode dividir o cão em duas
regionsmaone cobverindo a boca do cão e os olhos onde o preto é a
corth and eyes where black is the proeminente e a outra color and the
other cobverindo o resto do cão onde o amarelo é a corg the rest of
the dog where yellow is the proeminente.
- A segmentação da instância também é chamada de detecção e segmentação
simultâneas. Este método tenta identificar as regiões de nível de
pixel de cada color.
- Instance segmentation is also called simultaneous detection and
segmentation. This method attempts to identify the pixel-level
regions of each object instâancia de objeto em umae in an prominent
corin prominent instnc imagem.mIn cont semnticraste com a segmentação
semântica, a segmentação de instância não apenas sgmentation
distinguhes semâanticas, ms o differetes instane objeto. Se umaes. I
imagem contém dois cães, a segmentação de instância distingue quais
pixels pertencem a cada cachorro.
O Conjunto de Dados de Segmentação Semântica Pascal VOC2012
-----------------------------------------------------------
No campo de segmentaçãoains two dogswill distinguish which pixels belong
to which dog.
The Pascal VOC2012 Semantic Segmentation Dataset
------------------------------------------------
In the semâantica, um conjunto de dados importante é segmentation field,
one important dataset is `Pascal
VOC2012 `__. Para
entender melhor este conjunto de dados, devemos primeirootkage our
móoduloe necessário para oeded for t experimento.
.. raw:: html
.. raw:: html
.. code:: python
%matplotlib inline
import os
from mxnet import gluon, image, np, npx
from d2l import mxnet as d2l
npx.set_np()
.. raw:: html
.. raw:: html
.. code:: python
%matplotlib inline
import os
import torch
import torchvision
from d2l import torch as d2l
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
d2l.DATA_HUB['voc2012'] = (d2l.DATA_URL + 'VOCtrainval_11-May-2012.tar',
'4e443f8a2eca6b1dac8a6c57641b67dd40621a49')
voc_dir = d2l.download_extract('voc2012', 'VOCdevkit/VOC2012')
.. parsed-literal::
:class: output
Downloading ../data/VOCtrainval_11-May-2012.tar from http://d2l-data.s3-accelerate.amazonaws.com/VOCtrainval_11-May-2012.tar...
.. raw:: html
.. raw:: html
.. code:: python
#@save
d2l.DATA_HUB['voc2012'] = (d2l.DATA_URL + 'VOCtrainval_11-May-2012.tar',
'4e443f8a2eca6b1dac8a6c57641b67dd40621a49')
voc_dir = d2l.download_extract('voc2012', 'VOCdevkit/VOC2012')
.. parsed-literal::
:class: output
Downloading ../data/VOCtrainval_11-May-2012.tar from http://d2l-data.s3-accelerate.amazonaws.com/VOCtrainval_11-May-2012.tar...
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
def read_voc_images(voc_dir, is_train=True):
"""Read all VOC feature and label images."""
txt_fname = os.path.join(voc_dir, 'ImageSets', 'Segmentation',
'train.txt' if is_train else 'val.txt')
with open(txt_fname, 'r') as f:
images = f.read().split()
features, labels = [], []
for i, fname in enumerate(images):
features.append(image.imread(os.path.join(
voc_dir, 'JPEGImages', f'{fname}.jpg')))
labels.append(image.imread(os.path.join(
voc_dir, 'SegmentationClass', f'{fname}.png')))
return features, labels
train_features, train_labels = read_voc_images(voc_dir, True)
.. raw:: html
.. raw:: html
.. code:: python
#@save
def read_voc_images(voc_dir, is_train=True):
"""Read all VOC feature and label images."""
txt_fname = os.path.join(voc_dir, 'ImageSets', 'Segmentation',
'train.txt' if is_train else 'val.txt')
mode = torchvision.io.image.ImageReadMode.RGB
with open(txt_fname, 'r') as f:
images = f.read().split()
features, labels = [], []
for i, fname in enumerate(images):
features.append(torchvision.io.read_image(os.path.join(
voc_dir, 'JPEGImages', f'{fname}.jpg')))
labels.append(torchvision.io.read_image(os.path.join(
voc_dir, 'SegmentationClass' ,f'{fname}.png'), mode))
return features, labels
train_features, train_labels = read_voc_images(voc_dir, True)
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
n = 5
imgs = train_features[0:n] + train_labels[0:n]
d2l.show_images(imgs, 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_30_0.png
.. raw:: html
.. raw:: html
.. code:: python
n = 5
imgs = train_features[0:n] + train_labels[0:n]
imgs = [img.permute(1,2,0) for img in imgs]
d2l.show_images(imgs, 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_33_0.png
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
VOC_COLORMAP = [[0, 0, 0], [128, 0, 0], [0, 128, 0], [128, 128, 0],
[0, 0, 128], [128, 0, 128], [0, 128, 128], [128, 128, 128],
[64, 0, 0], [192, 0, 0], [64, 128, 0], [192, 128, 0],
[64, 0, 128], [192, 0, 128], [64, 128, 128], [192, 128, 128],
[0, 64, 0], [128, 64, 0], [0, 192, 0], [128, 192, 0],
[0, 64, 128]]
#@save
VOC_CLASSES = ['background', 'aeroplane', 'bicycle', 'bird', 'boat',
'bottle', 'bus', 'car', 'cat', 'chair', 'cow',
'diningtable', 'dog', 'horse', 'motorbike', 'person',
'potted plant', 'sheep', 'sofa', 'train', 'tv/monitor']
.. raw:: html
.. raw:: html
.. code:: python
#@save
VOC_COLORMAP = [[0, 0, 0], [128, 0, 0], [0, 128, 0], [128, 128, 0],
[0, 0, 128], [128, 0, 128], [0, 128, 128], [128, 128, 128],
[64, 0, 0], [192, 0, 0], [64, 128, 0], [192, 128, 0],
[64, 0, 128], [192, 0, 128], [64, 128, 128], [192, 128, 128],
[0, 64, 0], [128, 64, 0], [0, 192, 0], [128, 192, 0],
[0, 64, 128]]
#@save
VOC_CLASSES = ['background', 'aeroplane', 'bicycle', 'bird', 'boat',
'bottle', 'bus', 'car', 'cat', 'chair', 'cow',
'diningtable', 'dog', 'horse', 'motorbike', 'person',
'potted plant', 'sheep', 'sofa', 'train', 'tv/monitor']
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
def build_colormap2label():
"""Build an RGB color to label mapping for segmentation."""
colormap2label = np.zeros(256 ** 3)
for i, colormap in enumerate(VOC_COLORMAP):
colormap2label[(colormap[0]*256 + colormap[1])*256 + colormap[2]] = i
return colormap2label
#@save
def voc_label_indices(colormap, colormap2label):
"""Map an RGB color to a label."""
colormap = colormap.astype(np.int32)
idx = ((colormap[:, :, 0] * 256 + colormap[:, :, 1]) * 256
+ colormap[:, :, 2])
return colormap2label[idx]
.. raw:: html
.. raw:: html
.. code:: python
#@save
def build_colormap2label():
"""Build an RGB color to label mapping for segmentation."""
colormap2label = torch.zeros(256 ** 3, dtype=torch.long)
for i, colormap in enumerate(VOC_COLORMAP):
colormap2label[(colormap[0]*256 + colormap[1])*256 + colormap[2]] = i
return colormap2label
#@save
def voc_label_indices(colormap, colormap2label):
"""Map an RGB color to a label."""
colormap = colormap.permute(1,2,0).numpy().astype('int32')
idx = ((colormap[:, :, 0] * 256 + colormap[:, :, 1]) * 256
+ colormap[:, :, 2])
return colormap2label[idx]
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
y = voc_label_indices(train_labels[0], build_colormap2label())
y[105:115, 130:140], VOC_CLASSES[1]
.. parsed-literal::
:class: output
(array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[0., 0., 0., 0., 0., 0., 0., 1., 1., 1.],
[0., 0., 0., 0., 0., 0., 1., 1., 1., 1.],
[0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],
[0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],
[0., 0., 0., 0., 1., 1., 1., 1., 1., 1.],
[0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],
[0., 0., 0., 0., 0., 1., 1., 1., 1., 1.],
[0., 0., 0., 0., 0., 0., 1., 1., 1., 1.],
[0., 0., 0., 0., 0., 0., 0., 0., 1., 1.]]),
'aeroplane')
.. raw:: html
.. raw:: html
.. code:: python
y = voc_label_indices(train_labels[0], build_colormap2label())
y[105:115, 130:140], VOC_CLASSES[1]
.. parsed-literal::
:class: output
(tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 0, 0, 1, 1]]),
'aeroplane')
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
def voc_rand_crop(feature, label, height, width):
"""Randomly crop for both feature and label images."""
feature, rect = image.random_crop(feature, (width, height))
label = image.fixed_crop(label, *rect)
return feature, label
imgs = []
for _ in range(n):
imgs += voc_rand_crop(train_features[0], train_labels[0], 200, 300)
d2l.show_images(imgs[::2] + imgs[1::2], 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_66_0.png
.. raw:: html
.. raw:: html
.. code:: python
#@save
def voc_rand_crop(feature, label, height, width):
"""Randomly crop for both feature and label images."""
rect = torchvision.transforms.RandomCrop.get_params(feature,
(height, width))
feature = torchvision.transforms.functional.crop(feature, *rect)
label = torchvision.transforms.functional.crop(label, *rect)
return feature, label
imgs = []
for _ in range(n):
imgs += voc_rand_crop(train_features[0], train_labels[0], 200, 300)
imgs = [img.permute(1,2,0) for img in imgs]
d2l.show_images(imgs[::2] + imgs[1::2], 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_69_0.png
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
class VOCSegDataset(gluon.data.Dataset):
"""A customized dataset to load VOC dataset."""
def __init__(self, is_train, crop_size, voc_dir):
self.rgb_mean = np.array([0.485, 0.456, 0.406])
self.rgb_std = np.array([0.229, 0.224, 0.225])
self.crop_size = crop_size
features, labels = read_voc_images(voc_dir, is_train=is_train)
self.features = [self.normalize_image(feature)
for feature in self.filter(features)]
self.labels = self.filter(labels)
self.colormap2label = build_colormap2label()
print('read ' + str(len(self.features)) + ' examples')
def normalize_image(self, img):
return (img.astype('float32') / 255 - self.rgb_mean) / self.rgb_std
def filter(self, imgs):
return [img for img in imgs if (
img.shape[0] >= self.crop_size[0] and
img.shape[1] >= self.crop_size[1])]
def __getitem__(self, idx):
feature, label = voc_rand_crop(self.features[idx], self.labels[idx],
*self.crop_size)
return (feature.transpose(2, 0, 1),
voc_label_indices(label, self.colormap2label))
def __len__(self):
return len(self.features)
.. raw:: html
.. raw:: html
.. code:: python
#@save
class VOCSegDataset(torch.utils.data.Dataset):
"""A customized dataset to load VOC dataset."""
def __init__(self, is_train, crop_size, voc_dir):
self.transform = torchvision.transforms.Normalize(
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
self.crop_size = crop_size
features, labels = read_voc_images(voc_dir, is_train=is_train)
self.features = [self.normalize_image(feature)
for feature in self.filter(features)]
self.labels = self.filter(labels)
self.colormap2label = build_colormap2label()
print('read ' + str(len(self.features)) + ' examples')
def normalize_image(self, img):
return self.transform(img.float())
def filter(self, imgs):
return [img for img in imgs if (
img.shape[1] >= self.crop_size[0] and
img.shape[2] >= self.crop_size[1])]
def __getitem__(self, idx):
feature, label = voc_rand_crop(self.features[idx], self.labels[idx],
*self.crop_size)
return (feature, voc_label_indices(label, self.colormap2label))
def __len__(self):
return len(self.features)
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
crop_size = (320, 480)
voc_train = VOCSegDataset(True, crop_size, voc_dir)
voc_test = VOCSegDataset(False, crop_size, voc_dir)
.. parsed-literal::
:class: output
read 1114 examples
read 1078 examples
.. raw:: html
.. raw:: html
.. code:: python
crop_size = (320, 480)
voc_train = VOCSegDataset(True, crop_size, voc_dir)
voc_test = VOCSegDataset(False, crop_size, voc_dir)
.. parsed-literal::
:class: output
read 1114 examples
read 1078 examples
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
batch_size = 64
train_iter = gluon.data.DataLoader(voc_train, batch_size, shuffle=True,
last_batch='discard',
num_workers=d2l.get_dataloader_workers())
for X, Y in train_iter:
print(X.shape)
print(Y.shape)
break
.. parsed-literal::
:class: output
(64, 3, 320, 480)
(64, 320, 480)
.. raw:: html
.. raw:: html
.. code:: python
batch_size = 64
train_iter = torch.utils.data.DataLoader(voc_train, batch_size, shuffle=True,
drop_last=True,
num_workers=d2l.get_dataloader_workers())
for X, Y in train_iter:
print(X.shape)
print(Y.shape)
break
.. parsed-literal::
:class: output
torch.Size([64, 3, 320, 480])
torch.Size([64, 320, 480])
.. raw:: html
.. raw:: html
.. raw:: html
.. code:: python
#@save
def load_data_voc(batch_size, crop_size):
"""Download and load the VOC2012 semantic dataset."""
voc_dir = d2l.download_extract('voc2012', os.path.join(
'VOCdevkit', 'VOC2012'))
num_workers = d2l.get_dataloader_workers()
train_iter = gluon.data.DataLoader(
VOCSegDataset(True, crop_size, voc_dir), batch_size,
shuffle=True, last_batch='discard', num_workers=num_workers)
test_iter = gluon.data.DataLoader(
VOCSegDataset(False, crop_size, voc_dir), batch_size,
last_batch='discard', num_workers=num_workers)
return train_iter, test_iter
.. raw:: html
.. raw:: html
.. code:: python
#@save
def load_data_voc(batch_size, crop_size):
"""Download and load the VOC2012 semantic dataset."""
voc_dir = d2l.download_extract('voc2012', os.path.join(
'VOCdevkit', 'VOC2012'))
num_workers = d2l.get_dataloader_workers()
train_iter = torch.utils.data.DataLoader(
VOCSegDataset(True, crop_size, voc_dir), batch_size,
shuffle=True, drop_last=True, num_workers=num_workers)
test_iter = torch.utils.data.DataLoader(
VOCSegDataset(False, crop_size, voc_dir), batch_size,
drop_last=True, num_workers=num_workers)
return train_iter, test_iter
.. raw:: html
.. raw:: html
.. raw:: html
`Discussõeions `__
.. raw:: html
.. raw:: html
`Discussõeions `__
.. raw:: html
.. raw:: html