Conversation
|
thanks @vlcfaria for this. We dont use the parallel stuff much, which is why it moved to alpha. Can you explain a bit about your use case? |
| # instead of calling fork() and leaving leave the JVM in a weird state | ||
| def _eval_chunk(chunk): | ||
| out = [_evaluate_one_setting(k, v) for k, v in chunk] | ||
| detach() # release JNI refs |
There was a problem hiding this comment.
Flake says we should be decorating this method with @pt.java.required, but I think the actual point is that if the transformer doesnt involve Java (Terrier or Anserini) then jnius may not even be installed. So we need to detect jnuis (try import etc) and act accordingly.
| out = [_evaluate_one_setting(k, v) for k, v in chunk] | ||
| detach() # release JNI refs | ||
| return out | ||
| with ThreadPoolExecutor(max_workers=jobs) as ex: |
There was a problem hiding this comment.
Have you tested that you get a speed up with different Python threads calling Java? I think Terrier's data structures arent thread safe unless they are loaded with "concurrent:" prefix. At least with the forked JVM, if setup correctly, the results would be correct.
Ah, I see the issues now. Usecase is clear. I have some questions that I have placed in the review. |
2 participants
This is a bit of a rough attempt to fix
GridScanon multiprocessing scenarios.My understanding of the bug was that
pyterrier_alpha.parallel._parallel_lambda_joblibuses process-level parallelism, which copies the JVM started flag and the JVM state which can be messy.I swapped into thread-based parallelism, following Retriever.transform which I believe shares the JVM more cleanly while maintaining actual parallelism since they release the GIL. Ive kept the ray backend because it was kind of a grey area.