Ingest Shared Concerns
Mapping DSL
Description:
A generalized, easy-to-use language for converting documents of arbitrary schemas into DPLA MAP. At first, this will likely be implemented in a general programing language as part of the Record Processing project, with the expectation that we will eventually deliver a language that metadata experts with little-to-no programming experience will be successful using on their own, or with minimal supervision.
It's expected that this project will primarily be custom code with possibly a number of implementations if it needs to work in mutually-incompatible environments. Therefore, framework exploration probably isn't needed.
Selection criteria:
- Simple to use
- Accessible by non-programmers
- Needs to handle core use cases
- JSON
- XML
- RDF
- multi-schema/multi-namespace documents
- DPLA MAP
Nice-to-haves:
- Able to run in a variety of execution contexts (browser, command line, grid computing frameworks)
- Easily usable by partners in their own environments
- Deep document validity checks (not just well-formedness)
- Use of a declarative language like XPath, JSONPath, or XQuery for specification of field sources/destinations
- Use of a framework for creating custom languages starting from some sort of formal specification ala YACC/LLVM, etc.
| Technology Option | Strengths | Weaknesses | Opportunities | Threats |
|---|---|---|---|---|
| Javascript | JSON is native. DOM manipulations are built-in. Runs in a number of contexts (JVM, Browser, CLI). | Weak typing. Tooling is less mature than other options. Lots of churn in best practices/fashionable libraries. | ||
| Python | Explicit but succinct syntax. Very mature. Certain libs built from C can be very fast. Dynamic typing. Functional. Expressive. | |||
| Scala | Nearly as fast as Java while being less verbose. Great type system. XML parsing / navigation / document creation already available in strongly-typed DSLs. Functional. | Because Haskell-type people are working on it, can be overcomplicated. | ||
| Java | Fast. Strong typing. Sorta functional. | Verbose. Libraries can be overcomplicated or too low-level. Not very expressive. | ||
| Ruby | We "know" it. Very expressive. | Slow. Attempts to write expressive code yield to unintelligible code. |
Queueing System
Description:
The queuing system controls the runtime execution of activities. Currently, Ingestion 2 uses Resque, which is a Ruby-based environment that uses Redis as a datastore and for transaction logic.
Selection criteria:
- Must allow for a batch of operations to be queued
- Must somehow report statistics about the state of play of a batch for reporting purposes
- Must allow for management of failures
- Must allow for distribution of tasks among multiple workers
Nice-to-haves:
- Choice of implementation languages for workers
- Retrying capabilities
- Broader utility outside of ingestion use cases
| Technology Option | Worker Language | Strengths | Weaknesses | Opportunities | Threats |
|---|---|---|---|---|---|
| Airflow | Many | ||||
| RQ | Python | ||||
| Custom | Many | ||||
| Kafka | Many | ||||
| Resque | Ruby |
Record Processing
Description:
An execution environment for running harvests, maps, and enrichments across a provider's contributed metadata.
Selection criteria:
- TODO
Nice-to-haves:
- TODO
Notes:
It might not make sense to consider the Record Processing, Mapping DSL, and Queuing System projects separately if they are highly coupled.
| Technology Option | Language | Strengths | Weaknesses | Opportunities | Threats |
|---|---|---|---|---|---|
Webapp Shared Concerns
Dashboard
Description:
The Dashboard is a web application that will allow DPLA staff and partners and hubs to see the status of ingestion, mapping and enrichment processes on their data. It is now the Tech Team's intent that this application will get information about the status of these ingests through a REST API, which means that the Dashboard will be loosely coupled to the Ingestion stack. This will allow for evolution of the implementation and implementation technology of Ingestion without needing to modify the Dashboard application.
Selection criteria:
- TODO
Nice-to-haves:
- TODO
Notes:
The tech selection process for the Dashboard may very well be similar to that of the QA app, with the caveat that the Dashboard app will be built by a third party (HM).
| Technology Option | Language | Strengths | Weaknesses | Opportunities | Threats |
|---|---|---|---|---|---|
| Rails | Ruby | ||||
| Flask | Python | ||||
| Django | Python | ||||
| Play | Java or Scala |
QA App
Description:
The QA application will allow metadata experts to examine the output of mapping and harvest prior to writing to the production Elasticsearch index.
Selection criteria:
- TODO
Nice-to-haves:
- TODO
| Technology Option | Language | Strengths | Weaknesses | Opportunities | Threats |
|---|---|---|---|---|---|
| Rails | Ruby | ||||
| Flask | Python | ||||
| Django | Python | ||||
| Play | Java or Scala |
Developers Experience / Interests
| Dev | Expert At | Good At | Familiar With | Wants to Learn |
|---|---|---|---|---|
| Audrey | HTML+CSS, Javascript for DOM manipulations, Ruby (in Ruby on Rails context) | Object oriented Javascript, PHP (a little rusty), Ruby, SQL | Python, Java | Python, Scala, Java |
| Mark | Unix, Python(was pretty confident, now a little rusty), Javascript, PHP(formerly, doesn't like), HTML+CSS(a little rusty), Perl(rusty, been a while, is so over that) | Ruby | C, Java | Go, more Python, Scala, Java, Natural Language Processing |
Michael | Java, XML, Solr, Hadoop | Scala, Ruby (mostly not Rails) | Python, Javascript, Perl, C, Objective-C, XSLT, Spark, NLP, Machine Learning, Elasticsearch, Redshift, | Python, more Scala, Spark, |
| Scott | Not claiming "expert" skills in these subject but its what I'm strongest at. Java (<1.7), SQL (MSSSQL), Solr | Ruby (still learning), Python | Elasticsearch, Django (the only web framework I've do work with), C++ (bloodshed days) | Java 1.8, Scala, Spark, Go |