DataQueue

DataQueues are a generic way to store and fetch large amounts of data. The main goal is to store the data as fast as possible, making it ideal for import scenarios where the processing of the imported data occurs at a later stage in the entire import process.

Overview

The DataQueue feature has a single database table where each queue is stored with a configuration (table [DataQueues]). Then, each queue holds its own data in a separate table named [DataQueue_<queue-name>].

The DataQueue feature is generally available for each buzzle ServicePlatform module in the namespace Buzzle.ServicePlatform.Infrastructure.Features.DataQueue.

Initialization

The following prerequisites must be met to use the DataQueue feature:

• Add the abstract migration Buzzle.ServicePlatform.Infrastructure.Features.DataQueue.Migrations.DataQueueInitialMigration to the module project (create a new migration class that derives from the abstract migration class).
• Add the feature in the module services registration using services.AddDataQueueFeature().

Usage

The following section describes how to define a single DataQueue, how to add data to the queue, and how to read data from a queue.

Always keep in mind that the DataQueue feature is designed to write large amounts of data in bulk and read the data later in batches.

Define a Single DataQueue

To define a single DataQueue, register the DataQueue using a unique queue name (unique for the implemented buzzle ServicePlatform module) by adding services.AddDataQueue("<queue-name>") to the module services registration.

Each queue has a configuration that can be optionally overwritten. See the details of the DataQueueConfigurationModel here.

For convenience, there are DataQueueConfigurationModel extensions to configure the queue:

Extension method	Description
SetDataCompression(bool compressData)	Defines if data compression should be enabled

Store Data to DataQueue

To store data to the DataQueue, the following steps are required:

• Gather an instance of IDataQueueItemManagerFactory using dependency injection.
• Call the Create(string dataQueueName) method to instantiate a IDataQueueItemManager.
• Call the method StoreDataQueueData(IEnumerable<DataQueueItemModel> queueItems).

Keep in mind that this method stores large amounts of data using SQL Server's BulkInsert, so you can add thousands of DataQueueItemModel without any issues.

See the details of the DataQueueItemModel properties here.

Receive Data from DataQueue

To receive data from the queue, you should follow this scheme:

1. Gather an instance of IDataQueueItemManagerFactory using dependency injection.
2. Call the Create(string dataQueueName) method to instantiate a IDataQueueItemManager.
3. Call the LoadPriorityMap() method to load an array of DataQueuePriorityEntryModel containing the available priority numbers and a count for each priority.
4. Call either LoadDataQueueItems(int priority) or LoadDataQueueItems(int priority, int count) to load the data for the given priority.
5. After successfully processing each item, call the method DeleteDataQueueItems(IEnumerable<Guid> itemIds) to remove all successfully processed items and repeat steps 4 and 5 until no data is available.

Object References

DataQueueConfigurationModel

Property	Data Type	Default Value	Description
CompressData	bool	false	Defines if the data stored in the DataQueue should be compressed in the database. This is especially useful if the JSON input data is very large (> 20kb) for a single item. The data compression uses the default zip algorithm with a low compression rate for fast processing.

DataQueueItemModel

Property	Data Type	Description
Id	Guid	Main ID of the DataQueue item entry. Must be unique (ideally set to Guid.NewGuid()). If not set, it will be automatically set to Guid.NewGuid().
ExternalId	string	External ID of this single DataQueue entry, can be used to identify the entry by a custom ID.
TypeDiscriminator	string	The type discriminator of the given data. Can be used to identify the data without reading the content.
Priority	int	Used to prioritize all data in the queue for later processing. The data loading back from the queue only uses the priority ID (and an optional count).
Data	JObject	The JSON data representation of the single data item.
Created	DateTime	Creation date of this entry in the database. It is automatically set when adding the items to the queue.

DataQueueModel

Property	Data Type	Description
QueueName	string	The name of the queue
Configuration	DataQueueConfigurationModel	The configuration of the queue

DataQueuePriorityEntryModel

Property	Data Type	Description
Priority	int	The priority ID found in the existing data queue items.
Count	int	The number of items found for this priority.