Adjust queues on Ingestion 2 workers
The resque workers are very large – starting life at 200+ MB and growing to multiple gigabytes. A decent-sized mapping can take up half of the server's real memory and cause paging.
Do some combination of the following:
Adjust the queues on the production workers so that only one or two mappers run on one box.
Add production worker instances.
Upsize the memory of worker-prod2.
A third worker instance has been added. Let's see how this goes.
worker-prod2 has been upsized, and the worker counts have been adjusted, but I still plan to add another worker instance.