我有一堆用xargs并行读取的bzipped文件,用jq进行一些光处理,然后将输出重定向到一个文件,如下所示:
# Number of workers is one less than the number of cores
NUM_WORKERS=$(($(nproc) - 1))
NUM_WORKERS=$((NUM_WORKERS > 0 ? NUM_WORKERS : 1))
# List all bzipped files and process them with xargs in parallel
ls *.jsonl.bz2 | \
xargs -n 1 -P ${NU