Steadily increasing memory usage

[scjody]

After a 2 hour run of all our mocks except gl* and mapbox* using test/image-make_baseline.js (351 mocks, taking about 55 seconds per run), against image-exporter running as an imageserver in a Docker container, memory usage grew steadily:

https://plot.ly/%7EJodyMcintyre/2209/

When the run was stopped, memory usage decreased a bit then leveled out.

Examination of ps results showed that two electron processes (probably the plot image and plot thumbnail processes) were responsible for the memory usage:

A similar issue was observed in a 12 hour run of 3 imageservers in our staging environment:

As a workaround for this issue we could restart Electron after a reasonably large number of requests (e.g. 1,000). I'm going to look into this but a fix for the root cause is needed at some point.

@etpinard @monfera FYI

[scjody]

Note: this issue was first noted when we tried to use the new imageservers for Plotly Cloud prod. The problems encountered are discussed starting here: https://github.com/plotly/streambed/issues/9865#issuecomment-349995119

[scjody]

I've worked around this with #47. The underlying issue should be investigated at some point.

[mag009]

I was able to reproduce the error feeding sequential request of a large json file ( 4.7M ) and every single time at 233 request the process crash. The larger the file is the quicker the process crash.

• it's not a resources issue as no message appear in dmesg nor a segfault.
• I've tested with current version of electron and 2.0.9 same result.

I've tested with the following file : success/ee151de8-d07e-4d19-a2b6-400a2292c609_200.json
from : https://github.com/plotly/streambed/issues/9865#issuecomment-360823004

command used to test:
for i in {0..5000}; do curl -d @ee151de8-d07e-4d19-a2b6-400a2292c609_200.json http://localhost:9091/ ;echo $i;done

capture of orca's container when the 233rd requests happen:

12544ae95ac6 eager_banach 149.46% 2.447GiB / 4GiB 61.18% 0B / 0B 0B / 303kB 153

error message :

<--- Last few GCs --->

[44:0x2542baa0000] 286865 ms: Mark-sweep 2047.7 (2087.0) -> 2047.7 (2087.0) MB, 1668.0 / 0.0 ms allocation failure GC in old space requested
[44:0x2542baa0000] 288532 ms: Mark-sweep 2047.7 (2087.0) -> 2047.7 (2087.0) MB, 1667.2 / 0.0 ms last resort
[44:0x2542baa0000] 290208 ms: Mark-sweep 2047.7 (2087.0) -> 2047.7 (2087.0) MB, 1675.5 / 0.0 ms last resort

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1783e7b2d681
0: builtin exit frame: parse(this=0x1783e7b13ec9 ,0x2a6c162ff499 <Very long string[1514304]>)

2: /* anonymous */ [/var/www/image-exporter/src/app/server/create-server.js:144] [bytecode=0x27ecb9d52d21 offset=51](this=0x631dc09bb61 <JSGlobal Object>,err=0x363dedf82201 <null>,_body=0x2a6c162ff499 <Very long string[1514304]>)
4: onEnd [/var/www/image-exporter/...

[antoinerg]

I would like to know if this issue still affects the latest docker images for Orca. cc @scjody