kwimage.structs.mask module

Data structure for Binary Masks

Structure for efficient encoding of per-annotation segmentation masks Based on efficient cython/C code in the cocoapi [CocoStuffPyx] [CocoStuffC] [CocoStuffH] [CocoStuffPy].

References

[CocoStuffPyx]

https://github.com/nightrome/cocostuffapi/blob/master/PythonAPI/pycocotools/_mask.pyx

[CocoStuffC]

https://github.com/nightrome/cocostuffapi/blob/master/common/maskApi.c

[CocoStuffH]

https://github.com/nightrome/cocostuffapi/blob/master/common/maskApi.h

[CocoStuffPy]

https://github.com/nightrome/cocostuffapi/blob/master/PythonAPI/pycocotools/mask.py

Goals:

The goal of this file is to create a datastructure that lets the developer seemlessly convert between: (1) raw binary uint8 masks (2) memory-efficient compressed run-length-encodings of binary segmentation masks. (3) convex polygons (4) convex hull polygons (5) bounding box

It is not there yet, and the API is subject to change in order to better accomplish these goals.

Todo

• [ ] * Create two different classes: MultiLabelMask and BinaryMask

  both can inherit from Mask.

Note

IN THIS FILE ONLY: size corresponds to a h/w tuple to be compatible with the coco semantics. Everywhere else in this repo, size uses opencv semantics which are w/h.

class kwimage.structs.mask.Mask(data=None, format=None)

Bases: NiceRepr, _MaskConversionMixin, _MaskConstructorMixin, _MaskTransformMixin, _MaskDrawMixin

Manages a single segmentation mask and can convert to and from multiple formats including:

• bytes_rle - byte encoded run length encoding

• array_rle - raw run length encoding

• c_mask - c-style binary mask

• f_mask - fortran-style binary mask

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> # a ms-coco style compressed bytes rle segmentation
>>> segmentation = {'size': [5, 9], 'counts': ';?1B10O30O4'}
>>> mask = Mask(segmentation, 'bytes_rle')
>>> # convert to binary numpy representation
>>> binary_mask = mask.to_c_mask().data
>>> print(ub.urepr(binary_mask.tolist(), nl=1, nobr=1))
[0, 0, 0, 1, 1, 1, 1, 1, 0],
[0, 0, 1, 1, 1, 0, 0, 0, 0],
[0, 0, 1, 1, 1, 1, 1, 1, 0],
[0, 0, 1, 1, 1, 0, 1, 1, 0],
[0, 0, 1, 1, 1, 0, 1, 1, 0],

property dtype

classmethod random(rng=None, shape=(32, 32))

Create a random binary mask object

Parameters:

• rng (int | RandomState | None) – the random seed

• shape (Tuple[int, int]) – the height / width of the returned mask

Returns:

the random mask

Return type:

Mask

Example

>>> import kwimage
>>> mask = kwimage.Mask.random()
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> mask.draw()
>>> kwplot.show_if_requested()

classmethod demo()

Demo mask with holes and disjoint shapes

Returns:

the demo mask

Return type:

Mask

classmethod from_text(text, zero_chr='.', shape=None, has_border=False)

Construct a mask from a text art representation

Parameters:

• text (str) – the text representing a mask

• zero_chr (str) – the character that represents a zero

• shape (None | Tuple[int, int]) – if specified force a specific height / width, otherwise the character extent determines this.

• has_border (bool) – if True, assume the characters at the edge are representing a border and remove them.

Example

>>> import kwimage
>>> import ubelt as ub
>>> text = ub.indent(ub.codeblock(
>>>     '''
>>>     ooo
>>>     ooo
>>>     ooooo
>>>         o
>>>     '''))
>>> mask = kwimage.Mask.from_text(text, zero_chr=' ')
>>> print(mask.data)
[[0 0 0 0 1 1 1 0 0]
 [0 0 0 0 1 1 1 0 0]
 [0 0 0 0 1 1 1 1 1]
 [0 0 0 0 0 0 0 0 1]]

Example

>>> import kwimage
>>> import ubelt as ub
>>> text = ub.codeblock(
>>>     '''
>>>     +------------+
>>>     |            |
>>>     |    ooo     |
>>>     |    ooo     |
>>>     |    ooooo   |
>>>     |        o   |
>>>     |            |
>>>     +------------+
>>>     ''')
>>> mask = kwimage.Mask.from_text(text, has_border=True, zero_chr=' ')
>>> print(mask.data)
[[0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 1 1 1 0 0 0 0 0]
 [0 0 0 0 1 1 1 0 0 0 0 0]
 [0 0 0 0 1 1 1 1 1 0 0 0]
 [0 0 0 0 0 0 0 0 1 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]]

copy()

Performs a deep copy of the mask data

Returns:

the copied mask

Return type:

Mask

Example

>>> self = Mask.random(shape=(8, 8), rng=0)
>>> other = self.copy()
>>> assert other.data is not self.data

union(*others)

This can be used as a staticmethod or an instancemethod

Parameters:

*others – multiple input masks to union

Returns:

the unioned mask

Return type:

Mask

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> from kwimage.structs.mask import *  # NOQA
>>> masks = [Mask.random(shape=(8, 8), rng=i) for i in range(2)]
>>> mask = Mask.union(*masks)
>>> print(mask.area)
>>> masks = [m.to_c_mask() for m in masks]
>>> mask = Mask.union(*masks)
>>> print(mask.area)

>>> masks = [m.to_bytes_rle() for m in masks]
>>> mask = Mask.union(*masks)
>>> print(mask.area)

intersection(*others)

This can be used as a staticmethod or an instancemethod

Parameters:

*others – multiple input masks to intersect

Returns:

the intersection of the masks

Return type:

Mask

Example

>>> n = 3
>>> masks = [Mask.random(shape=(8, 8), rng=i) for i in range(n)]
>>> items = masks
>>> mask = Mask.intersection(*masks)
>>> areas = [item.area for item in items]
>>> print('areas = {!r}'.format(areas))
>>> print(mask.area)
>>> print(Mask.intersection(*masks).area / Mask.union(*masks).area)

property shape

property area

Returns the number of non-zero pixels

Returns:

the number of non-zero pixels

Return type:

int

Example

>>> self = Mask.demo()
>>> self.area
150

get_patch()

Extract the patch with non-zero data

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> from kwimage.structs.mask import *  # NOQA
>>> self = Mask.random(shape=(8, 8), rng=0)
>>> self.get_patch()

get_xywh()

Gets the bounding xywh box coordinates of this mask

Returns:

x, y, w, h: Note we dont use a Boxes object because

a general singular version does not yet exist.

Return type:

ndarray

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> self = Mask.random(shape=(8, 8), rng=0)
>>> self.get_xywh().tolist()
>>> self = Mask.random(rng=0).translate((10, 10))
>>> self.get_xywh().tolist()

Example

>>> # test empty case
>>> import kwimage
>>> self = kwimage.Mask(np.empty((0, 0), dtype=np.uint8), format='c_mask')
>>> assert self.get_xywh().tolist() == [0, 0, 0, 0]

bounding_box()

Returns an axis-aligned bounding box for this mask

Returns:

kwimage.Boxes

get_polygon()

DEPRECATED: USE to_multi_polygon

Returns a list of (x,y)-coordinate lists. The length of the list is equal to the number of disjoint regions in the mask.

Returns:

polygon around each connected component of the

mask. Each ndarray is an Nx2 array of xy points.

Return type:

List[ndarray]

Note

The returned polygon may not surround points that are only one pixel thick.

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> from kwimage.structs.mask import *  # NOQA
>>> self = Mask.random(shape=(8, 8), rng=0)
>>> polygons = self.get_polygon()
>>> print('polygons = ' + ub.urepr(polygons))
>>> polygons = self.get_polygon()
>>> self = self.to_bytes_rle()
>>> other = Mask.from_polygons(polygons, self.shape)
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> image = np.ones(self.shape)
>>> image = self.draw_on(image, color='blue')
>>> image = other.draw_on(image, color='red')
>>> kwplot.imshow(image)

to_mask(dims=None, pixels_are='points')

Converts to a mask object (which does nothing because this already is mask object!)

Returns:

kwimage.Mask

to_boxes()

Returns the bounding box of the mask.

Returns:

kwimage.Boxes

to_multi_polygon(pixels_are='points')

Returns a MultiPolygon object fit around this raster including disjoint pieces and holes.

Parameters:

pixel_are (str) – Can either be “points” or “areas”.

If pixels are “points”, the we treat each pixel (i, j) as a single infinitely small point at (i, j). As such, some polygons may have zero area.

If pixels are “areas”, then each pixel (i, j) represents a square with coordinates ([i - 0.5, j - 0.5], [i + 0.5, j - 0.5], [i + 0.5, j + 0.5], and [i - 0.5, j + 0.5]). Must have rasterio installed to use this method.

Returns:

vectorized representation

Return type:

kwimage.MultiPolygon

Note

The OpenCV (and thus this function) coordinate system places coordinates at the center of pixels, and the polygon is traced tightly around these coordinates. A single pixel is not considered to have any width, so polygon edges will directly trace through the centers of pixels, and in the case where an object is only 1 pixel thick, this will produce a polygon that is not a valid shapely polygon.

Todo

• [x] add a flag where polygons consider pixels to have width and the resulting polygon is traced around the pixel edges, not the pixel centers.

• [ ] Polygons and Masks should keep track of what “pixels_are”

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> from kwimage.structs.mask import *  # NOQA
>>> self = Mask.demo()
>>> self = self.scale(5)
>>> multi_poly = self.to_multi_polygon()
>>> # xdoctest: +REQUIRES(module:kwplot)
>>> # xdoctest: +REQUIRES(--show)
>>> self.draw(color='red')
>>> multi_poly.scale(1.1).draw(color='blue')

>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> image = np.ones(self.shape)
>>> image = self.draw_on(image, color='blue')
>>> #image = other.draw_on(image, color='red')
>>> kwplot.imshow(image)
>>> multi_poly.draw()

Example

>>> # Test empty cases
>>> import kwimage
>>> mask0 = kwimage.Mask(np.zeros((0, 0), dtype=np.uint8), format='c_mask')
>>> mask1 = kwimage.Mask(np.zeros((1, 1), dtype=np.uint8), format='c_mask')
>>> mask2 = kwimage.Mask(np.zeros((2, 2), dtype=np.uint8), format='c_mask')
>>> mask3 = kwimage.Mask(np.zeros((3, 3), dtype=np.uint8), format='c_mask')
>>> pixels_are = 'points'
>>> poly0 = mask0.to_multi_polygon(pixels_are=pixels_are)
>>> poly1 = mask1.to_multi_polygon(pixels_are=pixels_are)
>>> poly2 = mask2.to_multi_polygon(pixels_are=pixels_are)
>>> poly3 = mask3.to_multi_polygon(pixels_are=pixels_are)
>>> assert len(poly0) == 0
>>> assert len(poly1) == 0
>>> assert len(poly2) == 0
>>> assert len(poly3) == 0
>>> # xdoctest: +REQUIRES(module:rasterio)
>>> pixels_are = 'areas'
>>> poly0 = mask0.to_multi_polygon(pixels_are=pixels_are)
>>> poly1 = mask1.to_multi_polygon(pixels_are=pixels_are)
>>> poly2 = mask2.to_multi_polygon(pixels_are=pixels_are)
>>> poly3 = mask3.to_multi_polygon(pixels_are=pixels_are)
>>> assert len(poly0) == 0
>>> assert len(poly1) == 0
>>> assert len(poly2) == 0
>>> assert len(poly3) == 0

Example

>>> # Test full ones cases
>>> import kwimage
>>> mask1 = kwimage.Mask(np.ones((1, 1), dtype=np.uint8), format='c_mask')
>>> mask2 = kwimage.Mask(np.ones((2, 2), dtype=np.uint8), format='c_mask')
>>> mask3 = kwimage.Mask(np.ones((3, 3), dtype=np.uint8), format='c_mask')
>>> pixels_are = 'points'
>>> poly1 = mask1.to_multi_polygon(pixels_are=pixels_are)
>>> poly2 = mask2.to_multi_polygon(pixels_are=pixels_are)
>>> poly3 = mask3.to_multi_polygon(pixels_are=pixels_are)
>>> assert np.all(poly1.to_mask(mask1.shape).data == 1)
>>> assert np.all(poly2.to_mask(mask2.shape).data == 1)
>>> assert np.all(poly3.to_mask(mask3.shape).data == 1)
>>> # xdoctest: +REQUIRES(module:rasterio)
>>> pixels_are = 'areas'
>>> poly1 = mask1.to_multi_polygon(pixels_are=pixels_are)
>>> poly2 = mask2.to_multi_polygon(pixels_are=pixels_are)
>>> poly3 = mask3.to_multi_polygon(pixels_are=pixels_are)
>>> assert np.all(poly1.to_mask(mask1.shape).data == 1)
>>> assert np.all(poly2.to_mask(mask2.shape).data == 1)
>>> assert np.all(poly3.to_mask(mask3.shape).data == 1)

Example

>>> # Corner case, only two pixels are on
>>> import kwimage
>>> self = kwimage.Mask(np.zeros((768, 768), dtype=np.uint8), format='c_mask')
>>> x_coords = np.array([621, 752])
>>> y_coords = np.array([366, 292])
>>> self.data[y_coords, x_coords] = 1
>>> poly = self.to_multi_polygon()

Example

>>> # xdoctest: +REQUIRES(module:rasterio)
>>> import kwimage
>>> dims = (10, 10)
>>> data = np.zeros(dims, dtype=np.uint8)
>>> data[0, 3:5] = 1
>>> data[9, 1:3] = 1
>>> data[3:5, 0:2] = 1
>>> data[1, 1] = 1
>>> # 1 pixel L shape
>>> data[3, 5] = 1
>>> data[4, 5] = 1
>>> data[4, 6] = 1
>>> data[1, 5] = 1
>>> data[2, 6] = 1
>>> data[3, 7] = 1
>>> data[6, 1] = 1
>>> data[7, 1] = 1
>>> data[7, 2] = 1
>>> data[6:10, 5] = 1
>>> data[6:10, 8] = 1
>>> data[9, 5:9] = 1
>>> data[6, 5:9] = 1
>>> #data = kwimage.imresize(data, scale=2.0, interpolation='nearest')
>>> self = kwimage.Mask.coerce(data)
>>> #self = self.translate((0, 0), output_dims=(10, 9))
>>> self = self.translate((0, 1), output_dims=(11, 11))
>>> dims = self.shape[0:2]
>>> multi_poly1 = self.to_multi_polygon(pixels_are='points')
>>> multi_poly2 = self.to_multi_polygon(pixels_are='areas')
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> pretty_data = kwplot.make_heatmask(self.data/1.0, cmap='magma')[..., 0:3]
>>> def _pixel_grid_lines(self, ax):
>>>     h, w = self.data.shape[0:2]
>>>     ybasis = np.arange(0, h) + 0.5
>>>     xbasis = np.arange(0, w) + 0.5
>>>     xmin = 0 - 0.5
>>>     xmax = w - 0.5
>>>     ymin = 0 - 0.5
>>>     ymax = h - 0.5
>>>     ax.hlines(y=ybasis, xmin=xmin, xmax=xmax, color="gainsboro")
>>>     ax.vlines(x=xbasis, ymin=ymin, ymax=ymax, color="gainsboro")
>>> def _setup_grid(self, pnum):
>>>     ax = kwplot.imshow(pretty_data, show_ticks=True, pnum=pnum)[1]
>>>     # The gray ticks show the center of the pixels
>>>     ax.grid(color='dimgray', linewidth=0.5)
>>>     ax.set_xticks(np.arange(self.data.shape[1]))
>>>     ax.set_yticks(np.arange(self.data.shape[0]))
>>>     # Also draw black lines around the edges of the pixels
>>>     _pixel_grid_lines(self, ax=ax)
>>>     return ax
>>> # Overlay the extracted polygons
>>> ax = _setup_grid(self, pnum=(2, 3, 1))
>>> ax.set_title('input binary mask data')
>>> ax = _setup_grid(self, pnum=(2, 3, 2))
>>> multi_poly1.draw(linewidth=5, alpha=0.5, radius=0.2, ax=ax, fill=False, vertex=0.2)
>>> ax.set_title('opencv "point" polygons')
>>> ax = _setup_grid(self, pnum=(2, 3, 3))
>>> multi_poly2.draw(linewidth=5, alpha=0.5, radius=0.2, color='limegreen', ax=ax, fill=False, vertex=0.2)
>>> ax.set_title('raterio "area" polygons')
>>> ax.figure.suptitle(ub.codeblock(
>>>     '''
>>>     Gray lines are coordinates and pass through pixel centers (integer coords)
>>>     White lines trace pixel boundaries (fractional coords)
>>>     '''))
>>> raster1 = multi_poly1.to_mask(dims, pixels_are='points')
>>> raster2 = multi_poly2.to_mask(dims, pixels_are='areas')
>>> kwplot.imshow(raster1.draw_on(), pnum=(2, 3, 5), title='rasterized')
>>> kwplot.imshow(raster2.draw_on(), pnum=(2, 3, 6), title='rasterized')

get_convex_hull()

Returns a list of xy points around the convex hull of this mask

Note

The returned polygon may not surround points that are only one pixel thick.

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> self = Mask.random(shape=(8, 8), rng=0)
>>> polygons = self.get_convex_hull()
>>> print('polygons = ' + ub.urepr(polygons))
>>> other = Mask.from_polygons(polygons, self.shape)

iou(other)

The area of intersection over the area of union

Todo

• [ ] Write plural Masks version of this class, which should be able to perform this operation more efficiently.

CommandLine

xdoctest -m kwimage.structs.mask Mask.iou

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> self = Mask.demo()
>>> other = self.translate(1)
>>> iou = self.iou(other)
>>> print('iou = {:.4f}'.format(iou))
iou = 0.0830
>>> iou2 = self.intersection(other).area / self.union(other).area
>>> print('iou2 = {:.4f}'.format(iou2))

classmethod coerce(data, dims=None)

Attempts to auto-inspect the format of the data and conver to Mask

Parameters:

• data (Any) – the data to coerce

• dims (Tuple) – required for certain formats like polygons height / width of the source image

Returns:

the constructed mask object

Return type:

Mask

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> segmentation = {'size': [5, 9], 'counts': ';?1B10O30O4'}
>>> polygon = [
>>>     [np.array([[3, 0],[2, 1],[2, 4],[4, 4],[4, 3],[7, 0]])],
>>>     [np.array([[2, 1],[2, 2],[4, 2],[4, 1]])],
>>> ]
>>> dims = (9, 5)
>>> mask = (np.random.rand(32, 32) > .5).astype(np.uint8)
>>> Mask.coerce(polygon, dims).to_bytes_rle()
>>> Mask.coerce(segmentation).to_bytes_rle()
>>> Mask.coerce(mask).to_bytes_rle()

_to_coco()

use to_coco instead

to_coco(style='orig')

Convert the Mask to a COCO json representation based on the current format.

A COCO mask is formatted as a run-length-encoding (RLE), of which there are two variants: (1) a array RLE, which is slightly more readable and extensible, and (2) a bytes RLE, which is slightly more concise. The returned format will depend on the current format of the Mask object. If it is in “bytes_rle” format, it will be returned in that format, otherwise it will be converted to the “array_rle” format and returned as such.

Parameters:

style (str) – Does nothing for this particular method, exists for API compatibility and if alternate encoding styles are implemented in the future.

Returns:

either a bytes-rle or array-rle encoding, depending

on the current mask format. The keys in this dictionary are as follows:

counts (List[int] | str): the array or bytes rle encoding

size (Tuple[int]): the height and width of the encoded mask

see note.

shape (Tuple[int]): only present in array-rle mode. This

is also the height/width of the underlying encoded array. This exists for semantic consistency with other kwimage conventions, and is not part of the original coco spec.

order (str): only present in array-rle mode.

Either C or F, indicating if counts is aranged in row-major or column-major order. For COCO-compatibility this is always returned in F (column-major) order.

binary (bool): only present in array-rle mode.

For COCO-compatibility this is always returned as False, indicating the mask only contains binary 0 or 1 values.

Return type:

dict

Note

The output dictionary will contain a key named “size”, this is the only location in kwimage where “size” refers to a tuple in (height/width) order, in order to be backwards compatible with the original coco spec. In all other locations in kwimage a “size” will refer to a (width/height) ordered tuple.

SeeAlso:

func:

kwimage.im_runlen.encode_run_length - backend function that does array-style run length encoding.

Example

>>> # xdoctest: +REQUIRES(--mask)
>>> from kwimage.structs.mask import *  # NOQA
>>> self = Mask.demo()
>>> coco_data1 = self.toformat('array_rle').to_coco()
>>> coco_data2 = self.toformat('bytes_rle').to_coco()
>>> print('coco_data1 = {}'.format(ub.urepr(coco_data1, nl=1)))
>>> print('coco_data2 = {}'.format(ub.urepr(coco_data2, nl=1)))
coco_data1 = {
    'binary': True,
    'counts': [47, 5, 3, 1, 14, ... 1, 4, 19, 141],
    'order': 'F',
    'shape': (23, 32),
    'size': (23, 32),
}
coco_data2 = {
    'counts': '_153L;4EL...ON3060L0N060L0Nb0Y4',
    'size': [23, 32],
}

class kwimage.structs.mask.MaskList(data, meta=None)

Bases: ObjectList

Store and manipulate multiple masks, usually within the same image

to_polygon_list()

Converts all mask objects to multi-polygon objects

Returns:

kwimage.PolygonList

to_segmentation_list()

Converts all items to segmentation objects

Returns:

kwimage.SegmentationList

to_mask_list(dims=None, pixels_are='points')

returns this object

Returns:

kwimage.MaskList