Queues
Any can run heavy processing asynchronously. The FileSystem Importer, the SyncAPIs, the integrations, the asynchronous method execution requests drop messages on queues and consumers consume them to run them. The queues are carried by ActiveMQ on-premise and by SQS in the cloud.
Configuration
The queues are declared in the queues field of the messaging in the settings.json: the mdb-messaging section for ActiveMQ (on-premise), the aws-messaging section for SQS (cloud).
{
"mdb-messaging": {
"jms_server_address": "tcp://localhost:61616",
"queues": [
{ "name": "input", "queue": "input", "balancing": 1 },
{ "name": "methods", "queue": "methods", "balancing": 1 },
{
"name": "highprioritymethods",
"queue": "methods",
"balancing": 1,
"match": ["MassUpdateMethod", "SynchronizeRoomMethod"]
},
{ "name": "maintenance", "queue": "maintenance_methods", "balancing": 1 },
{ "name": "output", "queue": "output" },
{ "name": "pdf_request", "queue": "pdf_request" }
]
}
}
Each entry describes a queue:
- name: physical name of the queue on the broker. It is also the identifier the consumers are attached to.
- queue: type of the queue, which determines its role (see Queue types).
- balancing: number of consumers that consume the queue in parallel (see Number of consumers). Defaults to 2.
- match: list of methods routed to this queue (see Method-dedicated queues).
Default queues
| name | queue (type) | Role | Consumed at startup |
|---|---|---|---|
input | input | Incoming synchronization messages | Depending on the Synchronization module configuration |
methods | methods | Execution of the standard methods | Yes |
highprioritymethods | methods | Mass methods isolated from the methods queue | Yes |
maintenance | maintenance_methods | Methods run in maintenance mode | In maintenance mode only |
output | output | Outgoing synchronization messages | No, it is consumed by the recipient |
pdf_request | pdf_request | PDF and thumbnail generation requests | Yes |
Queue types
The queue field sets the role of the queue:
- input: incoming synchronization messages to import.
- output: outgoing synchronization messages pushed to a remote repository.
- methods: asynchronous execution of the methods.
- maintenance_methods: methods run when Any is in maintenance mode.
- pdf_request: PDF and thumbnail generation requests.
- bulk_import: bulk import.
Method-dedicated queues
By default, every method goes through the methods type queue. A large method then occupies the consumers and delays the other methods as long as it runs.
The match field routes some methods to a dedicated queue, served by its own consumers. Any provides the highprioritymethods queue to isolate the mass methods:
{
"name": "highprioritymethods",
"queue": "methods",
"balancing": 3,
"match": ["MassUpdateMethod", "SynchronizeRoomMethod"]
}
When a message is dropped, Any looks, among the queues of the targeted type, for the one whose match holds the name of the method. If a queue matches, the message is dropped there, otherwise it goes to the default queue of the type, the one without a match. The consumers of the dedicated queue therefore process these methods independently of the general methods queue.
To dedicate other methods, add an entry of the same methods type with a distinct name and the list of the methods in match. The default queue of the type must stay declared, it is the entry without a match. The name of a method is its class, for instance MassUpdateMethod.
Maintenance queues
The maintenance mode changes the queue that receives the methods. As long as it is active, Any drops the methods on a maintenance_methods type queue instead of the methods queue, and the Messaging Server stops the consumers of the methods queue to start those of the maintenance queue. On leaving the maintenance mode, the operation is reversed.
The maintenance queue provided by default serves this role:
{ "name": "maintenance", "queue": "maintenance_methods", "balancing": 1 }
Without a maintenance_methods type queue declared, no method runs during the maintenance: the messages stay pending until the mode is left. The match field also applies to this type: adding a maintenance_methods entry with a distinct name and a match list isolates some maintenance methods on their own consumers, as the highprioritymethods queue does for the standard methods.
Number of consumers
The balancing field sets the number of consumers attached to a queue, thus the number of messages processed in parallel. Increasing this value speeds up the consumption, at the cost of an increased load on the server, MongoDB and Elasticsearch. By default, Any starts 2 consumers.
{ "name": "methods", "queue": "methods", "balancing": 5 }
The change takes effect on the restart of the Messaging Server or of the application that hosts the consumers.
Multi-server configuration
The Any webapp processes no message of the queue, it only posts messages. The consumption of the messages is done exclusively by the MessagingServer. The MessagingServer instances can be multiplied to increase the number of consumers. Each instance can have a different queue configuration, by adjusting the queues entries and their balancing in each settings.json.
The output queue is never consumed automatically, and the input one is consumed depending on the configuration of the Synchronization module.
The installation of these components is described in Installing the Messaging Server. The maintenance mode and the refresh of the pending messages are covered in Maintenance.