我有超过10个任务执行,系统限制了最多可以同时运行4个任务。
我的任务可以像:
myprog任务名称
我如何编写一个bash shell脚本来运行这些任务。最重要的是,当一个任务完成时,脚本可以立即启动另一个任务,使运行的任务计数始终保持4。
我在考虑编写自己的流程池的同时,特别喜欢Brandon Horsley的解决方案,尽管我无法获取正确的信号,但我从Apache获得灵感,并决定尝试使用fifo作为前叉模型我的工作排队
# \brief the worker function that is called when we fork off worker processes # \param[in] id the worker ID # \param[in] job_queue the fifo to read jobs from # \param[in] result_log the temporary log file to write exit codes to function _job_pool_worker() { local id=$1 local job_queue=$2 local result_log=$3 local line= exec 7<> ${job_queue} while [[ "${line}" != "${job_pool_end_of_jobs}" && -e "${job_queue}" ]]; do # workers block on the exclusive lock to read the job queue flock --exclusive 7 read line <${job_queue} flock --unlock 7 # the worker should exit if it sees the end-of-job marker or run the # job otherwise and save its exit code to the result log. if [[ "${line}" == "${job_pool_end_of_jobs}" ]]; then # write it one more time for the next sibling so that everyone # will know we are exiting. echo "${line}" >&7 else _job_pool_echo "### _job_pool_worker-${id}: ${line}" # run the job { ${line} ; } # now check the exit code and prepend "ERROR" to the result log entry # which we will use to count errors and then strip out later. local result=$? local status= if [[ "${result}" != "0" ]]; then status=ERROR fi # now write the error to the log,making sure multiple processes # don't trample over each other. exec 8<> ${result_log} flock --exclusive 8 echo "${status}job_pool: exited ${result}: ${line}" >> ${result_log} flock --unlock 8 exec 8>&- _job_pool_echo "### _job_pool_worker-${id}: exited ${result}: ${line}" fi done exec 7>&- }
你可以在Github get a copy of my solution。这是一个使用我的实现的示例程序。
#!/bin/bash . job_pool.sh function foobar() { # do something true } # initialize the job pool to allow 3 parallel jobs and echo commands job_pool_init 3 0 # run jobs job_pool_run sleep 1 job_pool_run sleep 2 job_pool_run sleep 3 job_pool_run foobar job_pool_run foobar job_pool_run /bin/false # wait until all jobs complete before continuing job_pool_wait # more jobs job_pool_run /bin/false job_pool_run sleep 1 job_pool_run sleep 2 job_pool_run foobar # don't forget to shut down the job pool job_pool_shutdown # check the $job_pool_nerrors for the number of jobs that exited non-zero echo "job_pool_nerrors: ${job_pool_nerrors}"
希望这可以帮助!