Parallel shell processing with xargs

Today I learned how to parallel run a slow command on my shell. We can use xargs combined with the flags -n and -P flags. Let's see how this works:

find . -type f | xargs -n1 -P8 slow_command

• slow_command your slow command that receives a file as the first arg
• -n to specify how many arguments are passed to the slow_command
• -P how many parallel workers xargs will spawn to run the slow_command

Check this out watch -d -n 0.1 "seq 10 | xargs -n2 -P8 echo":

On this example xargs are spawning up to 8 workers to run the echo command and for each echo execution xargs will pass 2 arguments. The arguments are produced by a seq 10 and as multiple executions of echo runs in parallel we can highlight the output changes with watch.

Tweet