...
MemoryProfiler uses a lot of memory. Normally a worker exits after consuming 450 Mb of resident memory from a 200-record run. With MemoryProfiler wrapped around Krikri::Mapper.map in Krikri::Mapper::Agent#run, the process ends up using almost 2 Gb for a 5-record run. Therefore, we can only profile 5 records being mapped, which may not be realistic. I hope that it reveals something useful that can be acted upon, nevertheless.
Profiling and logging code
See https://github.com/dpla/KriKri/tree/example-mem-prof-201603 for some of the changes that were made to profile the code.
Specifically, https://github.com/dpla/KriKri/blob/example-mem-prof-201603/lib/krikri/mapper.rb
I have another branch that's similar to the one tagged above, in which RubyProf is wrapped around Krikri::Mapper::Agent#run, instead of MemoryProfiler. I could push that, too, but I imagine it's clear how you could implement it.
Files
System calls issued during a mapper run, focusing on where it stalls out for minutes:
...