Record Processing

Description:

An execution environment for running harvests, maps, and enrichments across a provider's contributed metadata.

Selection criteria:

• TODO

Nice-to-haves:

• TODO

Notes:

It might not make sense to consider the Record Processing, Mapping DSL, and Queuing System projects separately if they are highly coupled.

Mapping DSL

Description:

A generalized, easy-to-use language for converting documents of arbitrary schemas into DPLA MAP. At first, this will likely be implemented in a general programing language as part of the Record Processing project, with the expectation that we will eventually deliver a language that metadata experts with little-to-no programming experience will be successful using on their own, or with minimal supervision.

It's expected that this project will primarily be custom code with possibly a number of implementations if it needs to work in mutually-incompatible environments. Therefore, framework exploration probably isn't needed.

Selection criteria:

• Simple to use 
• Accessible by non-programmers
• Needs to handle core use cases 
  • JSON
  • XML
  • RDF
  • multi-schema/multi-namespace documents
  • DPLA MAP

Nice-to-haves:

• Able to run in a variety of execution contexts (browser, command line, grid computing frameworks)
• Easily usable by partners in their own environments
• Deep document validity checks (not just well-formedness)
• Use of a declarative language like XPath, JSONPath, or XQuery for specification of field sources/destinations
• Use of a framework for creating custom languages starting from some sort of formal specification ala YACC/LLVM, etc.

Dashboard

Description:

The Dashboard is a web application that will allow DPLA staff and partners and hubs to see the status of ingestion, mapping and enrichment processes on their data. It is now the Tech Team's intent that this application will get information about the status of these ingests through a REST API, which means that the Dashboard will be loosely coupled to the Ingestion stack. This will allow for evolution of the implementation and implementation technology of Ingestion without needing to modify the Dashboard application.

Selection criteria:

• TODO

Nice-to-haves:

• TODO

Notes:

The tech selection process for the Dashboard may very well be similar to that of the QA app, with the caveat that the Dashboard app will be built by a third party (HM).

QA App

Description:

The QA application will allow metadata experts to examine the output of mapping and harvest prior to writing to the production Elasticsearch index.

Selection criteria:

• TODO

Nice-to-haves:

• TODO

Queueing System

Description:

The queuing system controls the runtime execution of activities. Currently, Ingestion 2 uses Resque, which is a Ruby-based environment that uses Redis as a datastore and for transaction logic.

Selection criteria:

• Must allow for a batch of operations to be queued
• Must somehow report statistics about the state of play of a batch for reporting purposes
• Must allow for management of failures
• Must allow for distribution of tasks among multiple workers
  
  

Nice-to-haves:

• Choice of implementation languages for workers
• Retrying capabilities
• Broader utility outside of ingestion use cases

Developers Experience / Interests