See also

• Benchmarks

Ingest Shared Concerns

Generally speaking, the following sections ("Mapping DSL," "Queueing System", "Record Processing") represent intimately linked decisions to be made, in that it is impossible to make incompatible choices across the three areas. However, it's interesting to consider each of these independently to avoid prejudicing the discussion by making an early decision in one area and disregarding the other two.

Mapping DSL

Description:

A generalized, easy-to-use language for converting documents of arbitrary schemas into DPLA MAP. At first, this will likely be implemented in a general programing language as part of the Record Processing project, with the expectation that we will eventually deliver a language that metadata experts with little-to-no programming experience will be successful using on their own, or with minimal supervision.

It's expected that this project will primarily be custom code with possibly a number of implementations if it needs to work in mutually-incompatible environments. Therefore, framework exploration probably isn't needed.

Selection criteria:

• Simple to use 
• Accessible by non-programmers
• Needs to handle core use cases 
  • JSON
  • XML
  • multi-schema/multi-namespace documents
  • DPLA MAP (which is JSON-LD)

Nice-to-haves:

• Able to run in a variety of execution contexts (browser, command line, grid computing frameworks)
  • Support for some kind of 'live-preview' of DSL transformations, allow for rapid prototyping and development of mappings by non-developers 
• Easily usable by partners in their own environments
• Deep document validity checks (not just well-formedness)
• Use of a declarative language like XPath, JSONPath, or XQuery for specification of field sources/destinations
• Use of a framework for creating custom languages starting from some sort of formal specification ala YACC/LLVM, etc.

Job Queueing System

Description:

The queuing system controls the runtime execution of activities. Currently, Ingestion 2 uses Resque, which is a Ruby-based environment that uses Redis as a datastore and for transaction logic.

Selection criteria:

• Must allow for an Activity's Job to be queued
• Must somehow report statistics about the state of play of an Activity for reporting purposes.
• Must allow for management of failures.
• Must allow for distribution of Jobs among multiple workers.
• We need to be able to support this system, so having an ops playbook ahead of time is a good idea.

Nice-to-haves:

• Choice of implementation languages for workers
• Retrying capabilities
• Broader utility outside of ingestion use cases
• Allow for jobs to be scheduled for a specific date/time
• Able to define workflows for different phases of ingestion and explicitly model our workflows. We might want this because the steps for a new provider are different than a normal re-ingest.  

Record Processing

Description:

An execution environment for running harvests, maps, and enrichments across a provider's contributed metadata.

Selection criteria:

• Speed
  • Fast replay of ingestion pipeline; no provenance to support rollback
  • TODO: Establish baseline requirements for ingestion velocity in terms of providers and records 
• Reliability 
• Will need to interact with and share models between queueing system and QA app. 

Nice-to-haves:

• Concurrency
• Scalability 
• Easy to understand or common platform/framework/language within community 

Notes:

It might not make sense to consider the Record Processing, Mapping DSL, and Queuing System projects separately if they are highly coupled.

Webapp Shared Concerns

As each of the following tech selection sections are related to creating webapps, they share concerns. However, in this case, the decisions are not intimately related; we could very easily make separate decisions in each case.

Dashboard

Description:

The Dashboard is a web application that will allow DPLA staff and partners and hubs to see the status of ingestion, mapping and enrichment processes on their data. It is now the Tech Team's intent that this application will get information about the status of these ingests through a REST API, which means that the Dashboard will be loosely coupled to the Ingestion stack. This will allow for evolution of the implementation and implementation technology of Ingestion without needing to modify the Dashboard application.

Selection criteria:

• Easy to code
• Easy to deploy 
• Easy to manage dependencies
• Open source
• Active community of developers
• Well documented
• Hudson Molonglo can do it
• Quality error logging
• Ability to communicate with ingestion systems over API
• Authentication
• Lightweight relational database
• REST
• HTML templates
• JSON
• Quality test runner
• Secure
• Asset precompiling (JavaScript, SASS, etc)

Nice-to-haves:

• Used in the library community
• Language or framework familiar to staff
• Bootstrap compatible
• Lightweight
• ORM
• Lots of high-quality plugins or packages available

Notes:

The tech selection process for the Dashboard may very well be similar to that of the QA app, with the caveat that the Dashboard app will be built by a third party (HM).

Technology Option	Language	Strengths	Weaknesses	Opportunities	Threats
Rails	Ruby	Stable and widely used Good test runner Conventions make things like routing very easy Good HTML templating Good ORM for basic CRUD Bundler for managing dependencies Expressive language	Resource-intensive hosting More features than we really need Many dependencies Mediocre documentation Not as performant or flexible as lighter-weight frameworks	We already use it Popular in library community Development is fast if prepackaged features and existing gems meet our needs	High learning curve
Flask	Python	Simple, extensible core Serves JSON responses relatively fast Good for building APIs Flexible Explicit, expressive language	Less mature Smaller web community Poor documentation Testing okay, not great Have to do more of your own security	Python popular in library community Extensions probably available meet most of our needs	Not MVC
Django	Python	Stable and widely used Good HTML templating Good ORM Pip for managing dependencies Explicit, expressive language	More features than we really need Many dependencies Not as performant or flexible as lighter-weight frameworks Testing okay, not great	Python popular in library community Development is fast if prepackaged features and existing gems meet our needs	High learning curve
Play	Java or Scala	Good testing Good in-browser error handling Flexible	Immature Small dev community Upgrades not backward compatible Async, non-blocking I/O can make it hard to keep code clean SBT has reputation for being difficult build system Hot reload slow with Scala	Extensions probably available meet most of our needs	Not widely used in library community High learning curve No one really knows it

QA App

Description:

The QA application will allow metadata experts to examine the output of mapping and harvest prior to writing to the production Elasticsearch index.

Selection criteria:

• • Easy to code
  • Easy to deploy 
  • Easy to manage dependencies
  • Open source
  • Active community of developers
  • Well documented
  • Hudson Molonglo can do it
  • Quality error logging
  • Ability to communicate with ingestion systems over API
  • Ability to communicate with ElasticSearch
  • Authentication
  • REST
  • HTML templates
  • JSON
  • Quality test runner
  • Secure
  • Asset precompiling (JavaScript, SASS, etc)

Nice-to-haves:

• • Used in the library community
  • Language or framework familiar to staff
  • Bootstrap compatible
  • Lightweight
  • Lots of high-quality plugins or packages available

Developers Experience / Interests