Page Comparison

...

See 

Avro File

...

Schemas

In all schemas below, <DOCUMENTATION> is optional, but should contain some contextual note about the Activity that generated the file; especially for mappings and enrichments, where this could say the name of the harvested-records or mapped-records Avro file from which it is derived. It should contain the software version number or git commit identifier of the agent that wrote it.

la.dp.avro.MAP.3_1.IndexRecord.v1

For enriched records coming out of Ingestion 1 in JSON format, ready to be indexed into Elasticsearch.

{
	  "namespace": <NAMESPACE>
	 "la.dp.avro.MAP.3_1",
  "type": "record",
  	"name": <NAME>,
	"IndexRecord.v1",
  "doc": <DOCUMENTATION>,
  	"fields": [
		
    {"name": "id", "type": "string"},
    {"name": <FIELDNAME>"document", "type": "string"}
  ]
}

la.dp.avro.OriginalRecord.v1

For Original Records,

...

Where:

...

as they have been harvested from our providers.

{
  "namespace": "la.dp.avro",
  "type": "record",
  "name": "OriginalRecord.v1",
  "doc": <DOCUMENTATION>,
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "provider", "type": "string"},
    {"name": "document", "type": "string"},
    {"name": "mimetype": "type": "enum",
     "symbols": ["application/json", "application/xml", "text/turtle"]}
  ]
}

la.dp.avro.MAP

...

.4_0.MAPRecord.v1

For DPLA MAP records in our system – those that have been mapped or enriched, and are represented as MAPv4 RDF.

{
  "namespace": "la.dp.avro.MAP.4_0

...

",
  "type": "record",
  "name": "MAPRecord.v1",
  "doc": <DOCUMENTATION>,
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "document", "type": "string"}
  ]
}

la.dp.avro.MAP.4_0.IndexRecord

For DPLA MAPv4 records that have been mapped or enriched, and are being indexed into Elasticsearch.

This has not been merged with la.dp.avro.MAP.3_1.IndexRecord.v1 because we may want to add information in the future about which Elasticsearch schema it is destined for.

{
  "namespace": "la.dp.avro.MAP.4_0",
  "type": "record",
  "name": "IndexRecord.v1",
  "doc": <DOCUMENTATION>,
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "document", "type": "string"}
  ]
}

S3 Bucket Folder Structure and Contents

<PROVIDER>
    "plan"
        <TIMESTAMP>
			<TIMESTAMP>"-"<SCHEMA NAME>".json"
    <SCHEMA NAME><ACTIVITY>
        <DATE>
            <TIMESTAMP>"-"<PROVIDER>"-"<SCHEMA NAME>".avro"
            <TIMESTAMP>"-"<PROVIDER>"-"<SCHEMA NAME>"-prov.json"

...

• <PROVIDER> is a token like "cdl" or " or mdl".
• <DATE> is a string date representation in the form yyyymmdd, for example, "20170209". This is always the date when the relevant event started.
• <TIMESTAMP> is a string timestamp representation in the form yyyymmdd_hhmmss, for example, "20170209_104428". This is always the time when the relevant event started.
• <ACTIVITY> is one of harvest, mapping, enrichment, or indexing.
• <SCHEMA NAME> corresponds to the "name" property of the Avro file schema documented above .  It's "OriginalRecord" for harvests, "MappedRecord" for mappings, and "EnrichedRecord" for enrichments."plan" is minus the la.dp.avro prefix.

• plan is for manifests of whole ingests, where the term "plan" refers to the PROV-O notion of a Plan, or sequence of activities. The manifest is a set of JSON objects that documents which Avro files are relevant elsewhere in the provider's folder tree. A Plan can include one or more Activities, so a standalone remapping, for instance, could be part of a Plan with just that one activity. The JSON files in the "plan" directory are written only – they are not modified after being written. They function as a journal of all of the activities that have happened in the particular ingest.

• The files ending in "-prov.json" document which Plan the relevant Avro file belongs to, or which Avro file represents the generator Activity from which the relevant one is derived. For example, if you are looking at a particular mapping Avro file, this can direct you to the harvest Avro file whose records were mapped. This may be redundant with the information in a Plan file, but can make it easier to find things.

...

CDL's records from a harvest started on Feb. 9th, 2017:

/cdl/OriginalRecordharvest/20170209/20170209_104428-cdl-OriginalRecord.v1.avro

MDL's records from an enrichment started on Feb. 14th, 2017:

/mdl/enrichment/20170214/20170214_151848-mdl-MAP.4_0.MAPRecord.v1.avro

Plan Files and Provenance Files

...