我有要调用的shell命令列表。最多可有四个进程同时运行。
我的基本想法是将命令发送到shell,直到4个命令处于活动状态。然后,脚本通过查找一个常见的字符串,例如“”,不断检查所有进程的进程计数。
一旦进程计数降到4以下,下一个命令就会发送到shell,直到所有命令都完成为止。
有什么方法可以用shell脚本来完成这个任务吗?我想这会涉及到某种无穷无尽的循环,中断条件以及检查活动进程的方法。不幸的是,我并不擅长shell脚本,所以也许有人能引导我走向正确的方向?
nohup scrapy crawl urlMonitor -a slice=0 &
nohup scrapy crawl urlMonitor -a slice=1 &
nohup scrapy crawl urlMonitor -a slice=2 &
nohup scrapy crawl urlMonitor -a slice=3 &
nohup scrapy crawl urlMonitor -a slice=4 &
nohup scrapy crawl urlMonitor -a slice=5 &
nohup scrapy crawl urlMonitor -a slice=6 &
nohup scrapy crawl urlMonitor -a slice=7 &
nohup scrapy crawl urlMonitor -a slice=8 &
nohup scrapy crawl urlMonitor -a slice=9 &
nohup scrapy crawl urlMonitor -a slice=10 &
nohup scrapy crawl urlMonitor -a slice=11 &
nohup scrapy crawl urlMonitor -a slice=12 &
nohup scrapy crawl urlMonitor -a slice=13 &
nohup scrapy crawl urlMonitor -a slice=14 &
nohup scrapy crawl urlMonitor -a slice=15 &
nohup scrapy crawl urlMonitor -a slice=16 &
nohup scrapy crawl urlMonitor -a slice=17 &
nohup scrapy crawl urlMonitor -a slice=18 &
nohup scrapy crawl urlMonitor -a slice=19 &
nohup scrapy crawl urlMonitor -a slice=20 &
nohup scrapy crawl urlMonitor -a slice=21 &
nohup scrapy crawl urlMonitor -a slice=22 &
nohup scrapy crawl urlMonitor -a slice=23 &
nohup scrapy crawl urlMonitor -a slice=24 &
nohup scrapy crawl urlMonitor -a slice=25 &
nohup scrapy crawl urlMonitor -a slice=26 &
nohup scrapy crawl urlMonitor -a slice=27 &
nohup scrapy crawl urlMonitor -a slice=28 &
nohup scrapy crawl urlMonitor -a slice=29 &
nohup scrapy crawl urlMonitor -a slice=30 &
nohup scrapy crawl urlMonitor -a slice=31 &
nohup scrapy crawl urlMonitor -a slice=32 &
nohup scrapy crawl urlMonitor -a slice=33 &
nohup scrapy crawl urlMonitor -a slice=34 &
nohup scrapy crawl urlMonitor -a slice=35 &
nohup scrapy crawl urlMonitor -a slice=36 &
nohup scrapy crawl urlMonitor -a slice=37 &
nohup scrapy crawl urlMonitor -a slice=38 &发布于 2014-12-15 18:04:14
以下是一种通用方法,在启动任何其他作业之前,总是要确保少于4个作业(但是,如果一行同时启动多个作业,则可能同时有4个作业):
#!/bin/bash
max_nb_jobs=4
commands_file=$1
while IFS= read -r line; do
while :; do
mapfile -t jobs < <(jobs -pr)
((${#jobs[@]}<max_nb_jobs)) && break
wait -n
done
eval "$line"
done < "$commands_file"
wait将此脚本与您的文件一起用作第一个参数。
它怎麽工作?对于line读取的每一行,我们首先通过计算正在运行的作业数量(从jobs -pr获得)来确保运行的max_nb_jobs数少于max_nb_jobs。如果超过max_nb_jobs,我们将等待下一个作业终止(wait -n),并再次计数正在运行的作业数量。如果max_nb_jobs的运行量小于eval,我们将运行行line。
更新
下面是一个不使用wait -n的类似脚本。它似乎做得很好(用Bash 4.2在Debian上测试):
#!/bin/bash
set -m
max_nb_jobs=4
file_list=$1
sleep_jobs() {
# This function sleeps until there are less than $1 jobs running
# Make sure that you have set -m before using this function!
local n=$1 jobs
while mapfile -t jobs < <(jobs -pr) && ((${#jobs[@]}>=n)); do
coproc read
trap "echo >&${COPROC[1]}; trap '' SIGCHLD" SIGCHLD
wait $COPROC_PID
done
}
while IFS= read -r line; do
sleep_jobs $max_nb_jobs
eval "$line"
done < "$file_list"
wait发布于 2014-12-15 18:06:44
如果您希望一次连续运行4次,请尝试如下所示:
max_procs=4
active_procs=0
for proc_num in {0..38}; do
nohup your_cmd_here &
# If we have more than max procs running, wait for one to finish
if ((active_procs++ >= max_procs)); then
wait -n
((active_procs--))
fi
done
# Wait for all remaining procs to finish
wait这是对sputnick的回答的一种变化,它与max_procs同时运行。一旦完成,它就会开始下一个。wait -n命令等待下一个进程完成,而不是等待它们全部完成。
发布于 2014-12-15 17:48:49
尝试这样做:
for i in {0..38}; do
nohup scrapy crawl urlMonitor -a slice=$i & _pid=$!
((++i%4==0)) && wait $_pid
donehelp wait
wait: wait [-n] [id ...]
Wait for job completion and return exit status.
Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status. If ID is not
given, waits for all currently active child processes, and the return
status is zero. If ID is a a job specification, waits for all processes
in that job's pipeline.
If the -n option is supplied, waits for the next job to terminate and
returns its exit status.
Exit Status:
Returns the status of the last ID; fails if ID is invalid or an invalid
option is given.https://stackoverflow.com/questions/27489762
复制相似问题