Simple Python Multiprocessing function doesn't output results
I have this very simple function right here in which I'm trying to run and test on,however,it doesn't output anything and it doesn't have any errors either. I've checked the code multiple times but it doesn't have any errors.
I printed jobs and here's what I got:
<pre class="lang-py prettyprint prettyprinted">
<span style="color:#303336;">[<<span style="color:#2b91af;">Process<span style="color:#303336;">(<span style="color:#2b91af;">Process<span style="color:#303336;">-<span style="color:#7d2727;">12<span style="color:#303336;">,<span style="color:#303336;"> stopped<span style="color:#303336;">[<span style="color:#7d2727;">1<span style="color:#303336;">])>,<span style="color:#303336;">
<span style="color:#303336;"><<span style="color:#2b91af;">Process<span style="color:#303336;">(<span style="color:#2b91af;">Process<span style="color:#303336;">-<span style="color:#7d2727;">13<span style="color:#303336;">,<span style="color:#303336;">
<span style="color:#303336;"><<span style="color:#2b91af;">Process<span style="color:#303336;">(<span style="color:#2b91af;">Process<span style="color:#303336;">-<span style="color:#7d2727;">14<span style="color:#303336;">,<span style="color:#303336;">
<span style="color:#303336;"><<span style="color:#2b91af;">Process<span style="color:#303336;">(<span style="color:#2b91af;">Process<span style="color:#303336;">-<span style="color:#7d2727;">15<span style="color:#303336;">,<span style="color:#303336;">
<span style="color:#303336;"><<span style="color:#2b91af;">Process<span style="color:#303336;">(<span style="color:#2b91af;">Process<span style="color:#303336;">-<span style="color:#7d2727;">16<span style="color:#303336;">,<span style="color:#303336;"> stopped<span style="color:#303336;">[<span style="color:#7d2727;">1<span style="color:#303336;">])>]
Here's the code:
<pre class="lang-py prettyprint prettyprinted">
<span style="color:#101094;">import<span style="color:#303336;"> multiprocessing
<span style="color:#101094;">def<span style="color:#303336;"> worker<span style="color:#303336;">(<span style="color:#303336;">num<span style="color:#303336;">):<span style="color:#303336;">
<span style="color:#101094;">print<span style="color:#303336;"> <span style="color:#7d2727;">"worker "<span style="color:#303336;">,<span style="color:#303336;"> num
<span style="color:#101094;">return<span style="color:#303336;">
jobs <span style="color:#303336;">=<span style="color:#303336;"> <span style="color:#303336;">[]<span style="color:#303336;">
<span style="color:#101094;">for<span style="color:#303336;"> i <span style="color:#101094;">in<span style="color:#303336;"> range<span style="color:#303336;">(<span style="color:#7d2727;">5<span style="color:#303336;">):<span style="color:#303336;">
p <span style="color:#303336;">=<span style="color:#303336;"> multiprocessing<span style="color:#303336;">.<span style="color:#2b91af;">Process<span style="color:#303336;">(<span style="color:#303336;">target <span style="color:#303336;">=<span style="color:#303336;"> worker<span style="color:#303336;">,<span style="color:#303336;"> args <span style="color:#303336;">=<span style="color:#303336;"> <span style="color:#303336;">(<span style="color:#303336;">i<span style="color:#303336;">,))<span style="color:#303336;">
jobs<span style="color:#303336;">.<span style="color:#303336;">append<span style="color:#303336;">(<span style="color:#303336;">p<span style="color:#303336;">)<span style="color:#303336;">
p<span style="color:#303336;">.<span style="color:#303336;">start<span style="color:#303336;">()
Here's the result I'm expecting but it's not outputting anything:
<pre class="lang-py prettyprint prettyprinted">
<span style="color:#2b91af;">Worker<span style="color:#303336;">:<span style="color:#303336;"> <span style="color:#7d2727;">0<span style="color:#303336;">
<span style="color:#2b91af;">Worker<span style="color:#303336;">:<span style="color:#303336;"> <span style="color:#7d2727;">1<span style="color:#303336;">
<span style="color:#2b91af;">Worker<span style="color:#303336;">:<span style="color:#303336;"> <span style="color:#7d2727;">2<span style="color:#303336;">
<span style="color:#2b91af;">Worker<span style="color:#303336;">:<span style="color:#303336;"> <span style="color:#7d2727;">3<span style="color:#303336;">
<span style="color:#2b91af;">Worker<span style="color:#303336;">:<span style="color:#303336;"> <span style="color:#7d2727;">4
原因:spyder使用的stdout和windows不支持forking,所以无法打印子进程内容。
stdout
stdout
sys.__stdout__
There are two alternatives:
<ol style="margin-left:30px;">
Not using print
within the child processes,but simply returning the result to the main process. Either by using a (or multiprocessing.Manager().Queue()
since forking is not possible) or more simply by relying on the map
functionality,see example below.
Multiprocessing example with a Pool:
<pre class="lang-py prettyprint prettyprinted">
<span style="color:#101094;">import<span style="color:#303336;"> multiprocessing
<span style="color:#101094;">def<span style="color:#303336;"> worker<span style="color:#303336;">(<span style="color:#303336;">num<span style="color:#303336;">):<span style="color:#303336;">
<span style="color:#7d2727;">"""Returns the string of interest"""<span style="color:#303336;">
<span style="color:#101094;">return<span style="color:#303336;"> <span style="color:#7d2727;">"worker %d"<span style="color:#303336;"> <span style="color:#303336;">%<span style="color:#303336;"> num
<span style="color:#101094;">def<span style="color:#303336;"> main<span style="color:#303336;">():<span style="color:#303336;">
pool <span style="color:#303336;">=<span style="color:#303336;"> multiprocessing<span style="color:#303336;">.<span style="color:#2b91af;">Pool<span style="color:#303336;">(<span style="color:#7d2727;">4<span style="color:#303336;">)<span style="color:#303336;">
results <span style="color:#303336;">=<span style="color:#303336;"> pool<span style="color:#303336;">.<span style="color:#303336;">map<span style="color:#303336;">(<span style="color:#303336;">worker<span style="color:#303336;">,<span style="color:#303336;"> range<span style="color:#303336;">(<span style="color:#7d2727;">10<span style="color:#303336;">))<span style="color:#303336;">
pool</span><span style="color:#303336;">.</span><span style="color:#303336;">close</span><span style="color:#303336;">()</span><span style="color:#303336;">
pool</span><span style="color:#303336;">.</span><span style="color:#303336;">join</span><span style="color:#303336;">()</span><span style="color:#303336;">
</span><span style="color:#101094;">for</span><span style="color:#303336;"> result </span><span style="color:#101094;">in</span><span style="color:#303336;"> results</span><span style="color:#303336;">:</span><span style="color:#303336;">
</span><span style="color:#858c93;"># prints the result string in the main process</span><span style="color:#303336;">
</span><span style="color:#101094;">print</span><span style="color:#303336;">(</span><span style="color:#303336;">result</span><span style="color:#303336;">)</span><span style="color:#303336;">
<span style="color:#101094;">if<span style="color:#303336;"> name <span style="color:#303336;">==<span style="color:#303336;"> <span style="color:#7d2727;">'main'<span style="color:#303336;">:<span style="color:#303336;">
<span style="color:#858c93;"># Better protect your main function when you use multiprocessing<span style="color:#303336;">
main<span style="color:#303336;">()
which prints (in the main process)
<pre class="lang-py prettyprint prettyprinted">
<span style="color:#303336;">worker <span style="color:#7d2727;">0<span style="color:#303336;">
worker <span style="color:#7d2727;">1<span style="color:#303336;">
worker <span style="color:#7d2727;">2<span style="color:#303336;">
worker <span style="color:#7d2727;">3<span style="color:#303336;">
worker <span style="color:#7d2727;">4<span style="color:#303336;">
worker <span style="color:#7d2727;">5<span style="color:#303336;">
worker <span style="color:#7d2727;">6<span style="color:#303336;">
worker <span style="color:#7d2727;">7<span style="color:#303336;">
worker <span style="color:#7d2727;">8<span style="color:#303336;">
worker <span style="color:#7d2727;">9
EDIT: If you are to impatient to wait for the map
function to finish,you can immediately print your results by using imap_unordered
and slightly changing the order of the commands:
<pre class="lang-py prettyprint prettyprinted">
<span style="color:#101094;">def<span style="color:#303336;"> main<span style="color:#303336;">():<span style="color:#303336;">
pool <span style="color:#303336;">=<span style="color:#303336;"> multiprocessing<span style="color:#303336;">.<span style="color:#2b91af;">Pool<span style="color:#303336;">(<span style="color:#7d2727;">4<span style="color:#303336;">)<span style="color:#303336;">
results <span style="color:#303336;">=<span style="color:#303336;"> pool<span style="color:#303336;">.<span style="color:#303336;">imap_unordered<span style="color:#303336;">(<span style="color:#303336;">worker<span style="color:#303336;">,<span style="color:#303336;"> range<span style="color:#303336;">(<span style="color:#7d2727;">10<span style="color:#303336;">))<span style="color:#303336;">
</span><span style="color:#101094;">for</span><span style="color:#303336;"> result </span><span style="color:#101094;">in</span><span style="color:#303336;"> results</span><span style="color:#303336;">:</span><span style="color:#303336;">
</span><span style="color:#858c93;"># prints the result string in the main process as soon as say are ready</span><span style="color:#303336;">
</span><span style="color:#858c93;"># but results are now no longer in order!</span><span style="color:#303336;">
</span><span style="color:#101094;">print</span><span style="color:#303336;">(</span><span style="color:#303336;">result</span><span style="color:#303336;">)</span><span style="color:#303336;">
</span><span style="color:#858c93;"># The pool should join after printing all results</span><span style="color:#303336;">
pool</span><span style="color:#303336;">.</span><span style="color:#303336;">close</span><span style="color:#303336;">()</span><span style="color:#303336;">
pool</span><span style="color:#303336;">.</span><span style="color:#303336;">join</span><span style="color:#303336;">()</span></code></pre>
来源:https://stackoverflow.com/questions/29629103/simple-python-multiprocessing-function-doesnt-output-results/29632397#29632397