[New WfMS] Pipefunc

[JaGeo]

Hi @jan-janssen ,
Just found this repo: https://github.com/pipefunc/pipefunc

The similarities between jobflow's, Aiida's and pyiron's developments are striking:

https://github.com/pipefunc/pipefunc

[jan-janssen]

Yes, I feel there is a raise of more light-weight workflow solutions. As part of the Exascale Computing Project (ECP) we also played around with a similar idea, which lead to the development of https://github.com/pyiron/executorlib . pipefunc as well as executorlib do not provide any specific data storage functionality, but instead primarily focus on distributing Python functions and executing those. The limitation I see with pipefunc is that they do not support MPI parallel Python functions, which was one of the features we needed in the context of the ECP. The other focus for executorlib was staying as close to the Python Standard Library as possible, following the Executor pattern https://docs.python.org/3/library/concurrent.futures.html#executor-objects which pipefunc and parsl also use as their backend. Another tool in this direction is https://github.com/facebookincubator/submitit but again it does not provide advanced features like MPI support and so on.