2016-10-28 Meeting notes

• • Speed: speed is a feature. Predictably say how long some ingest will take.
  • Allowing recovery from failure; pick up where it left off.  Speed affects this; if it's fast enough you don't have to worry about it. Otherwise, make sure there's recovery. Harvesters should allow recovery, where possible. Indexers could also be less speedy than mappings and enrichments, and may deserve recovery features.
  • Adding automation that was originally specified: have a program that shepherds the process all the way through. Scheduling.
  • Eventually, provide a useable mapping DSL
    • Needs real market research
    • This is not a turnkey solution yet. Some things like DSLs will be evaluated later when we can more confident in understanding how big the user base is.
    • Writing mappings ourselves in the third system without a DSL will allow us to understand the problem space better.
  • Ability to debug things, especially mappings

Michael et. al.

Got walkthroughs of the Python, Python + Spark, Java, and Scala prototypes

• https://github.com/dpla/pyfeta
• https://github.com/dpla/jetta
• https://github.com/dpla/scetta

Scheduling / operation chaining / "Plans" in the Prov-O sense

• Need metrics for what qualifies job failure. (Partly thought out)
• Need to get together and assess our experiences running ingests.
• If we automate things, we need to know how to define success.
• Tools exist that can help with this.
• Need to schedule a period after basic manual ingest running is figured out, but need to design for there being a scheduling facility. Per "General consensus" section below, we will design programs for each activity in the ingest process that have their concerns passed to them. They will not know anything about the scheduling system that calls them. They will not be bound to them with database models. They will save manifests that document the results of their operations.