Home Node Request piping data failure
Reply: 0

Node Request piping data failure

user45767
1#
user45767 Published in September 20, 2018, 4:35 pm

I am trying to download about 800MB per file. This is the code to achieve it:

function makeRequest(url,name,callback){
                name = promObj.Name;
                requestsdone =-1;
                request(url,{auth:auth}, function (err,respnse,body) {
                })
                    .on('error', function (err) {
                        console.log(err)
                    })
                    .on('end', function () {
                        requestsdone++;
                    }).pipe(fs.createWriteStream('./app/data/' + name[requestsdone] + '.zip'));

                callback();

            }

            async.eachLimit(promObj.requestURLS, 2, makeRequest, function(err) {
                if(err) reject(promObj)
                else{
                    resolve(promObj);
                }
            });

I updated Node.js to version 9.5.0 and get this error now:

<--- Last few GCs --->

[12060:000001FED372E610]   487169 ms: Mark-sweep 28.7 (35.4) -> 28.7 (35.4) 
MB, 33.6 / 0.0 ms  allocation failure GC in old space requested
[12060:000001FED372E610]   487205 ms: Mark-sweep 28.7 (35.4) -> 28.7 (35.4) 
MB, 36.2 / 0.0 ms  last resort GC in old space requested
[12060:000001FED372E610]   487240 ms: Mark-sweep 28.7 (35.4) -> 28.7 (35.4) 
MB, 34.7 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001311C6254D9 <JSObject>
1: stringSlice(aka stringSlice) [buffer.js:586] [bytecode=0000031E27B6DEE1 offset=11](this=000003FACAA022D1 <undefined>,buf=000000CDF7528259 <Uint8Array map = 00000392BE4C4D01>,encoding=000003FACAA022D1 <undefined>,start=0,end=801130252)
2: toString [buffer.js:664] [bytecode=0000031E27B6DB29 offset=148](this=000000CDF7528259 <Uint8Array map = 00000392BE4C4D01>,encoding=000003FACAA022D1...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

How could I solve this? Thanks in advance.

PS: This question refers to Requests with request-promise pipe result (Large data) ,which I asked some time ago. I restructured my code and thought that this problem is worth a new, sepperated question because the original topic of the other question was a different one.

share|improve this question
  • Request is the npm request module? I guess. – Michał Kapracki Feb 12 at 9:44
  • @MichałCzapracki yes thats right. – PhilG Feb 12 at 9:47
  • Can you make sure you're not using a body parser of some sort? To my knowledge it shouldn't give you any issues - at least I don't see much wrong. I suspect some wrong outside of the paste... – Michał Kapracki Mar 3 at 23:59

active oldest votes

Your Answer

StackExchange.ifUsing("editor", function () { StackExchange.using("externalEditor", function () { StackExchange.using("snippets", function () { StackExchange.snippets.init(); }); }); }, "code-snippets"); StackExchange.ready(function() { var channelOptions = { tags: "".split(" "), id: "1" }; initTagRenderer("".split(" "), "".split(" "), channelOptions); StackExchange.using("externalEditor", function() { // Have to fire editor after snippets, if snippets enabled if (StackExchange.settings.snippets.snippetsEnabled) { StackExchange.using("snippets", function() { createEditor(); }); } else { createEditor(); } }); function createEditor() { StackExchange.prepareEditor({ heartbeatType: 'answer', convertImagesToLinks: true, noModals: false, showLowRepImageUploadWarning: true, reputationToPostImages: 10, bindNavPrevention: true, postfix: "", onDemand: true, discardSelector: ".discard-answer" ,immediatelyShowMarkdownHelp:true }); } });
 
StackExchange.ready( function () { StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f48743315%2fnode-request-piping-data-failure%23new-answer', 'question_page'); } );

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged javascript node.js asynchronous npm-request or ask your own question.

StackExchange.ready(function(){$.get('/posts/48743315/ivc/af20');});
StackExchange.ready(function () { StackExchange.responsiveness.addSwitcher(); }) (function(i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function() { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m); })(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga'); StackExchange.ready(function () { StackExchange.ga.init({ sendTitles: true, tracker: window.ga, trackingCodes: [ 'UA-108242619-1' ] }); StackExchange.ga.setDimension('dimension2', '|javascript|node.js|asynchronous|npm-request|'); StackExchange.ga.setDimension('dimension3', 'Questions/Show'); StackExchange.ga.trackPageView(); }); /**/ var _qevents = _qevents || [], _comscore = _comscore || []; (function() { var ssl = 'https:' == document.location.protocol, s = document.getElementsByTagName('script')[0], qc = document.createElement('script'); qc.async = true; qc.src = (ssl ? 'https://secure' : 'http://edge') + '.quantserve.com/quant.js'; s.parentNode.insertBefore(qc, s); _qevents.push({ qacct: "p-c1rF4kxgLUzNc" }); /**/ var sc = document.createElement('script'); sc.async = true; sc.src = (ssl ? 'https://sb' : 'http://b') + '.scorecardresearch.com/beacon.js'; s.parentNode.insertBefore(sc, s); _comscore.push({ c1: "2", c2: "17440561" }); })();
You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.407253 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO