multiprocessing_ndfilter

grizli.utils.multiprocessing_ndfilter(data, filter_func, filter_args=(), size=None, footprint=None, cutout_size=256, n_proc=4, timeout=90, mask=None, verbose=True, **kwargs)[source]

Cut up a large array and send slices to scipy.ndimage filters

Parameters:
data : array-like

Main image array

filter_func : function

Filtering function, e.g., scipy.ndimage.median_filter

filter_args : tuple

Arguments to pass to filter_func

size, footprint : int, array-like

Filter size or footprint, see, e.g., scipy.ndimage.median_filter

cutout_size : int

Size of subimage cutouts

n_proc : int

Number of multiprocessing processes to use

timeout : float

multiprocessing timeout (seconds)

mask : array-like

Array multiplied to data that can zero-out regions to ignore

verbose : bool

Print status messages

kwargs : dict

Keyword arguments passed through to filter_func

Returns:
filtered : array-like

Filtered version of data

Examples

>>> import time
>>> import numpy as np
>>> import scipy.ndimage as nd
>>> from grizli.utils import multiprocessing_ndfilter
>>> rnd = np.random.normal(size=(512,512))
>>> t0 = time.time()
>>> f_serial = nd.median_filter(rnd, size=10)
>>> t1 = time.time()
>>> f_mp = multiprocessing_ndfilter(rnd, nd.median_filter, size=10,
>>>                                 cutout_size=256, n_proc=4)
>>> t2 = time.time()
>>> np.allclose(f_serial, f_mp)
True
>>> print(f'  serial: {(t1-t0)*1000:.1f} ms')
>>> print(f'parallel: {(t2-t1)*1000:.1f} ms')
  serial: 573.9 ms
parallel: 214.8 ms