splitjob 2.2

Useful to split up jobs over multiple CPU cores or even multiple computers. Examples: tar -cf - /bigdirectory | splitjob -j 4 gzip > big.tar.gz splitjob "ssh h1 bzip2" "ssh h2 bzip2" < f > f.bz2

Tags c command-line distributed-computing
License GNU GPL
State development

Recent Releases

2.209 Mar 2019 22:36 major feature: Added experimental support for parallel decompression
2.111 Nov 2017 22:50 major bugfix: Bugfix: Fixed copy-paste error in code which caused writing outside allocated buffer when output data from called program was bigger than input data. This could happen also at compression if data is already compressed. In theory bugs like this could cause more or less random behavior. In practice this bug has caused corrupted backup archives. Any users of version 2.0 should upgrade to version 2.1 to avoid this bug!
2.015 Oct 2017 20:33 major feature: Now accepts signal SIGUSR2 to increase number of parallel running jobs and SIGUSR1 to decrease number of parallel running jobs.
1.209 Oct 2017 21:11 minor feature: Might now be able to recover if sub process fails even if some data has been read out from the sub process.
1.131 Jan 2015 18:51 minor feature: Saving RAM by freeing up unused RAM in child processes.
1.014 Dec 2014 14:57 documentation: No changes since previos beta version which have now been tested for some months without any problems found.
0.9.2beta25 Aug 2014 11:36 minor bugfix: Avoiding short reads from stdin which could cause non optimal compression performance when read block sizes got smaller than intended.