A simple and efficient tool to parallelize Pandas operations on all available CPUs.
pandarallel is a simple and efficient tool to parallelize Pandas operations on all
With a one line code change, it allows any Pandas user to take advandage of his
multi-core computer, while
pandas uses only one core.
pandarallel also offers nice progress bars (available on Notebook and terminal) to
get an rough idea of the remaining amount of computation to be done.
pandarallel currently implements the following
|Without parallelization||With parallelization|
On Linux & macOS, no special requirement.
On Windows, because of the multiprocessing system (spawn), the function you send to
pandarallel must be self contained, and should not depend on external resources.
✅ Valid on Mac and Linux - ❌ Forbidden On Windows
import math def func(x): # Here, `math` is defined outside `func`. `func` is not self contained. return math.sin(x.a**2) + math.sin(x.b**2)
✅ Valid everywhere
def func(x): # Here, `math` is defined inside `func`. `func` is self contained. import math return math.sin(x.a**2) + math.sin(x.b**2)
Parallelization has a cost (instantiating new processes, sending data via shared memory, ...), so parallelization is efficient only if the amount of computation to parallelize is high enough. For very little amount of data, using parallelization is not always worth it.
Displaying progress bars has a cost and may slighly increase computation time.
An example of each available
pandas API is available:
- For Mac & Linux
- For Windows
For some examples, here is the comparative benchmark with and without using Pandaral·lel.
Computer used for this benchmark:
- OS: Linux Ubuntu 16.04
- Hardware: Intel Core i7 @ 3.40 GHz - 4 cores
For those given examples, parallel operations run approximately 4x faster than the standard operations (except for
series.map which runs only 3.2x faster).
When should I user
pandasis a fast, powerful, flexible and easy to use open source data analysis and manipulation tool,built on top of the Python programming language.
pandas drawback is the fact it uses only one core of your computer, even if
multiple cores are available.
pandarallel gets around this limitation by using all cores of your computer.
But, in return,
pandarallel need twice the memory that standard
would normally use.
pandarallel should NOT be used if your data cannot fit into memory with
pandas itself. In such a case,
spark (and its
will be suitable.
The main drawback of
spark is that
spark APIs are less convenient to user than
pandas APIs (even if this is going better) and also you need a JVM (Java Virtual
Machine) on your computer.
spark you can:
- Handle data much bigger than your memory
- Using a
sparkcluster, distribute your computation over multiple nodes.