...
The EBS volume that houses Marmotta's triples table is currently of highest concern, because it is the one that gets hit hardest when we encounter periods of bad write performance. It's rated at 12,000 IOPS. Our CloudWatch chart for this volume shows it using a maximum of about 350K read/write operations per minute.
Meanwhile, One might be inclined to think that 12,000 operations per second would provide 720,000 operations per minute, and 350,000 would be within that limit; but our kernel statistics indicate high utilization for this device:
ThereUnfortunately, the title of Amazon CloudWatch's "Consumed Read Write Ops" chart may be causing some confusion, because it implies that it has to do with IOPS being "consumed." Instead, there's some possibility that, because of the way in which AWS's IOPS are calculated (considering not just the number of operations, but the size of each operation, which could vary), we're running up against a storage volume speed limitation again. Throwing faster hardware at this problem might make it go away for a while, but it still seems to us that the patterns documented in the bad insert / update performance page are more fundamental.
...