...
Queueing System
Description:Lorem ipsum
The queuing system controls the runtime execution of activities. Currently, Ingestion 2 uses Resque, which is a Ruby-based environment that uses Redis as a datastore and for transaction logic.
Selection criteria:
...
- Must allow for a batch of operations to be queued
- Must somehow report statistics about the state of play of a batch for reporting purposes
- Must allow for management of failures
- Must allow for distribution of tasks among multiple workers
Nice-to-haves:
...
- Choice of implementation languages for workers
- Retrying capabilities
- Broader utility outside of ingestion use cases
| Technology Option | Worker Language | Strengths | Weaknesses | Opportunities | Threats |
|---|---|---|---|---|---|
| Airflow | Many | ||||
| RQ | Python | ||||
| Custom | Many | ||||
| Kafka | Many | ||||
| Resque | Ruby |
Developers Experience / Interests
...