Home Why Virtual Memory grows during map_async operation with python?
Reply: 0

Why Virtual Memory grows during map_async operation with python?

user44472
1#
user44472 Published in September 19, 2018, 11:05 am

I am trying to create interatomic distances of N=6000 atoms for 100 frames. Each frame contains atom positions in a periodic boundary box. I am trying to parallelize this process using multiprocessing tools in python. Mainly I use map_async function:

    q=Pool(np)
    for group in groups:
    first_frames.append(group[0])
    start1=timer()
    #s=q.map(dipoles.new_create_neighborlist,first_frames)
    s=q.map_async(dipoles.new_create_neighborlist,first_frames)
    q.close()
    q.join()

first_frames is a list of objects which contains atom positions etc. Length of the first_frames list is 1000. Number of processors is 16. It starts well and does the computation without problem. However after some time virtual memory used by the processes grows and gives virtual memory error. dipoles.new_create_neighborlist returns a list of atoms with its neighborlists. I tried previously answered methods but did not work. What could I be missing here?

share|improve this question
  • (1) How big are these 1000 objects. (2) Do you actually need them in a list, in order, at the end, or could you just as easily use, say, an iterator in arbitrary order, or a callback on each result in arbitrary order, or something like that? – abarnert Apr 14 at 2:16
  • These objects are positions of atoms. For 6000 atoms it would be 3x6000 floating point numbers. I do not need them in a list, I can sort it later. – dundar yilmaz Apr 15 at 19:56
  • If each one is just 18000 floats, it's probably not the issue. My thought was that you've got 100 result objects, plus 84-100 queued inputs outputs, all alive at the same time. By using imap_unordered and an iterator instead of a list for input, you'd reduce that to worst-case 16 result objects and 2 input objects at peak. But given the size you're talking about, that should be on the order of a few dozen megabytes, nowhere near enough to cause VM pressure unless you're using a 1990s Palm Pilot or something. – abarnert Apr 16 at 17:25

active oldest votes

Your Answer

StackExchange.ifUsing("editor", function () { StackExchange.using("externalEditor", function () { StackExchange.using("snippets", function () { StackExchange.snippets.init(); }); }); }, "code-snippets"); StackExchange.ready(function() { var channelOptions = { tags: "".split(" "), id: "1" }; initTagRenderer("".split(" "), "".split(" "), channelOptions); StackExchange.using("externalEditor", function() { // Have to fire editor after snippets, if snippets enabled if (StackExchange.settings.snippets.snippetsEnabled) { StackExchange.using("snippets", function() { createEditor(); }); } else { createEditor(); } }); function createEditor() { StackExchange.prepareEditor({ heartbeatType: 'answer', convertImagesToLinks: true, noModals: false, showLowRepImageUploadWarning: true, reputationToPostImages: 10, bindNavPrevention: true, postfix: "", onDemand: true, discardSelector: ".discard-answer" ,immediatelyShowMarkdownHelp:true }); } });
 
StackExchange.ready( function () { StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f49827169%2fwhy-virtual-memory-grows-during-map-async-operation-with-python%23new-answer', 'question_page'); } );

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged python dictionary asynchronous or ask your own question.

StackExchange.ready(function(){$.get('/posts/49827169/ivc/5731');});
StackExchange.ready(function () { StackExchange.responsiveness.addSwitcher(); }) (function(i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function() { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m); })(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga'); StackExchange.ready(function () { StackExchange.ga.init({ sendTitles: true, tracker: window.ga, trackingCodes: [ 'UA-108242619-1' ] }); StackExchange.ga.setDimension('dimension2', '|python|dictionary|asynchronous|'); StackExchange.ga.setDimension('dimension3', 'Questions/Show'); StackExchange.ga.trackPageView(); }); /**/ var _qevents = _qevents || [], _comscore = _comscore || []; (function() { var ssl = 'https:' == document.location.protocol, s = document.getElementsByTagName('script')[0], qc = document.createElement('script'); qc.async = true; qc.src = (ssl ? 'https://secure' : 'http://edge') + '.quantserve.com/quant.js'; s.parentNode.insertBefore(qc, s); _qevents.push({ qacct: "p-c1rF4kxgLUzNc" }); /**/ var sc = document.createElement('script'); sc.async = true; sc.src = (ssl ? 'https://sb' : 'http://b') + '.scorecardresearch.com/beacon.js'; s.parentNode.insertBefore(sc, s); _comscore.push({ c1: "2", c2: "17440561" }); })();
You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 1.842173 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO