Created: 2011-07-10 18:38
Updated: 2018-08-23 08:14
License: mit

Build Status


This software is free to use and modify, and licensed under MIT License (see LICENSE file).


Parallel tools (named in manner to itertools & functools) is a set of commonly used list traversal functions, which is working in parallel (fault-tolerant) in synchronous or asynchronous manner.

Implementation is based on python threading module, so be aware of GIL.

Currently implemented functions are (both sync & async):

  • filter - filters the list by predicate you provide;
  • map - applies a function to each element of the list.

Important: Due to nature of parallel processing the order of results isn't guranteed. Although, function is returns a list because the objects you want to process might not be hashable, hence you can't use a set.


This module is useful if you do I/O-heavy task, e.g. collecting a RSS-feeds or determining if site is alive or not.


Synchronous with default parameters:

import urllib
import paralleltools

feeds = ['',

comics =, feeds)


import Image
import logging
import paralleltools

images = ['cat1.jpg', 'cat2.jpg', 'cat3.jpg', ..., 'catN.jpg']

def rotate(img):
	return img

def processed(img):"Yay! %s is processed" % img)

paralleltools.async_map(rotate, images, threads=20, callback=processed)"Cats being processed")


Synchronous with default parameters:

import ping
import paralleltools

sites = ['',

def alive(site):
	return ping(site) > 100

result = paralleltools.filter(alive, sites)


import lxml
import paralleltools

docs = ['wikileaks_doc1.xml', 'wikileaks_doc2.xml', 'wikileaks_doc3.xml']

def valid(doc):
		return True
	except lxml.etree.XMLSyntaxError:
		return False

def upload_document(doc):
	# conspiracy

paralleltools.async_filter(valid, docs, callback=upload_document)
find_more_documents()  # while these are processed


Methods available:

  • map
  • async_map
  • filter
  • async_filter


  • function
  • iterable
  • threads (default = 5)
  • result_callback (sync) or callback (async) - note, that callback is called on each result

You can create your own workers by extending AbstractWorker in module. Or altering supervisor behaviour in

Cookies help us deliver our services. By using our services, you agree to our use of cookies Learn more