Home Why Virtual Memory grows during map_async operation with python?
 I am trying to create interatomic distances of N=6000 atoms for 100 frames. Each frame contains atom positions in a periodic boundary box. I am trying to parallelize this process using multiprocessing tools in python. Mainly I use map_async function:  q=Pool(np) for group in groups: first_frames.append(group[0]) start1=timer() #s=q.map(dipoles.new_create_neighborlist,first_frames) s=q.map_async(dipoles.new_create_neighborlist,first_frames) q.close() q.join()  first_frames is a list of objects which contains atom positions etc. Length of the first_frames list is 1000. Number of processors is 16. It starts well and does the computation without problem. However after some time virtual memory used by the processes grows and gives virtual memory error. dipoles.new_create_neighborlist returns a list of atoms with its neighborlists. I tried previously answered methods but did not work. What could I be missing here?