Message Queues, Messages and Polling

The One Network Platform can receive files from an external system using many different protocols and formats. Possible message sources include:

Processing of messages is done asynchronously. This means clients can drop a large message, get an immediate response when the message is delivered, and then check on the processing status later. To enable asynchronous processing without loss of data in the event of a power outage or other event, Platform persists all inbound Messages in Message Queues. The Messages are stored in the database, and the queue guarantees the order of processing is guaranteed to be first-in-first-out.

In this tutorial section, we will create an Inbox queue to allow a Book CSV file to be dropped and processed. We will setup file polling to assign the file to an interface and queue when it is created.

To begin our exercise, let's assume we have a legacy Enterprise system which contains a catalog of books. It is capable of producing a CSV file containing the details of any book added to that catalog each night, and dropping them into a directory. We will setup polling in Platform to watch for that file to be dropped and add it to a message queue.

Let's first setup the MessageSource. A MessageSource identifies a location from which Messages can be acquired. We will give it the following properties:

Create a new file in the Bookstore_dataset/data directory called "MessageSources.csv" with the following contents:

#* Name,Enterprise Name,* Source Type,* Config
BookstoreFileDrop,,File,{ srcDir: 'C:/PLT/inbox/bookstore' }

Now, submit the file using the interface PLT.MessageSource_IB, version 1.0, and username BookstoreAdmin. (This is an interface which is available as part of the Platform core, thus the "PLT" prefix.) You should get confirmation of successful submission.

Next, let's create a MessageQueue. This is the actual queue into which the Message will be placed after it's picked up from the filesystem. We will give it the following properties:

Create a new file in the Bookstore_dataset/data directory called "MessageQueues.csv" with the following contents:

#* Name,Enterprise Name,Classification,"Exclusive Consumer (0 or 1 (0=false, 1=true))","Paused (0 or 1 (0=false, 1=true))"

Submit this file using the interface PLT.MessageQueue_IB, version 1.0. (Please keep in mind that the MessageQueue has no relationship to the "C:/PLT/inbox/bookstore" message source yet.)

Once this is loaded, we can actually log in and see this MessageQueue. Log in to Command Center as InstanceAdminUser. You will see a menu item called Message Queues under Integraton menu - open that page then provide required filter "Message Created After" date value(Select Today).

You may actually have several queues, but at a minimum you should see the new inbox/Bookstore queue. We can use this page to view pending and processed messages, pause processing, or reprocess old messages.

Finally, we need to setup polling. By creating a MessageSourcePoll, we can tell Platform that "when a file comes into a directory whose name matches a certain pattern, put it in a particular queue and process it with a particular interface." Specifically, we'll set ours up as follows:

Create a new file in the Bookstore_dataset/data directory called "MessageSourcePoll.csv" with the following contents:

#* Name,Enterprise Name,Message Source Name,Message Source Enterprise Name,Group Name,"Precedence (#,###)",Include Expr,Exclude Expr,Inbound Queue Name,Inbound Queue Enterprise Name,Inbound Interface,Inbound Interface Version

Submit this file using the interface PLT.MessageSourcePoll_IB, version 1.0

Now that we have created our MessageSource, MessageSourcePoll and MessageQueue, we're ready to start dropping files. Let's create a new file Book.csv somewhere other than the polled directory with the following contents:

#* Title,Published Date (MM/dd/yyyy),"Rating (#,#.#####)"
The Mythical Man Month,3/01/1975,8

Now copy it to the polled directory. Navigate to C:/PLT/inbox/bookstore in a Windows explorer. (Platform should have created it for you by now!) Copy/paste your Book.csv file into this directory.

Within a few seconds, Platform should pick up the file, after which it should disappear:

Now if we return to the Message Queues page, we should see that 1 file was Processed from the inbox/Bookstore queue, and a new message was created in the outbox/Bookstore/success queue (indicating that processing was 100% successful).

But what if there are processing errors? Try deleting "The Mythical Man Month" from Book.csv and dropping the file again. This time, after the file is picked up, you will see a new queue outbox/Bookstore/error with a new message in it. Click the queue name to see the error message, then click Download to get the details of the error.

This should help you to understand the basics of message queues and file polling. Here are a few additional troubleshooting tips: