Topic Scalability PreviousNext
Data Distribution Service (DDS) Forum > DDS Technical Forum >
Message/Author
Next message Stephan Damen  posted on Tuesday, February 10, 2009 - 12:58 pm
I'm currently investigating which DDS or RTPS implementation is most suitable for a system with a large number of topics.

How scalable are the DDS implementations in relation to topics?

The system contains in the order of a 100000 topics. Registration of all these topics to a manager / infoRepo takes up too much time and resources.

One option would be to merge a large number of these topics together and using the ContentFilteredTopic construction to filter them.
OpenDDS does not have ContentFilteredTopic support yet. Is this implemented in the soon-to-be opensource OpenSplice DDS? And also in the OpenSplice RTPS?

Furthermore, the bandwidth between the nodes in the system is limited. Which DDS or RTPS implementation filters the messages on the DataWriter side to reduce traffic?

Kind regards,

Stephan
Next message Hans van 't Hag  posted on Tuesday, February 10, 2009 - 01:22 pm
Hi Stephan,

When talking about scalability, its important to distinguish between the number of topics (with predefined type), and the amount of published instances cq. samples of these topics (where instances are samples that have different values of the topic's key-attributes). The total set of topics could be viewed as 'the information model' where the optimal granularity of the model is also driving the system scalability (i.e. many small topics with key-attributes to model relationships versus fewer large topics).
OpenSplice DDS has extensive support for the relational information-model of DDS in that it supports both content-filtering and queries (on DCPS-level) as well as allows for automatic reconstruction of multiple topics into (local) objects by means of the Data Local Reconstruction Layer (DLRL) that also allows fine-grained triggers as well as navigation over object-relations.

To further improve scalability, DDS supports the concept of logical 'partitions' (a QoS policy), which in OpenSplice DDS can be dynamically bound to multicast groups so to benefit from modern (IGMP) network technology.

Another scalability aspect is related to the number of applications on a specific node that would be interested in the information. In OpenSplice DDS there is always only 1 copy of the information in any physical node, regardless of the number of applications/readers/writers for that data (we utilize shared memory for that) and related to this architecture choice, we only need to deserialize the information once regardless of the number of subscribers on a node.

Our current OpenSplice DDS version doesn't do writer-side filtering yet (mainly since it almost implies multiple/individual sends whereas our multicast-enabled partitioning and single-copy architecture allows very efficient/fast filtering at the destination).

Hope his helps a little
Next message Stephan Damen  posted on Tuesday, February 10, 2009 - 02:54 pm
Dear Hans,

Thank you for your quick and elaborated reaction.

Are there any rules of thumb regarding the partitioning of 'the information model' as compared to for instance database normalization?

We are trying to get a feel of the capabilities of DDS systems and were wondering if systems with a 100000 topics or more are considered small, normal, or large?

Furthermore, do I understand correctly that content-filtering and queries are only implemented on DCPS-level and not on DSSI (RTPS) level with OpenSplice?

And will shared-memory transport also be available under the community license?

Kind regards,

Stephan
Next message Hans van 't Hag  posted on Tuesday, February 10, 2009 - 03:42 pm
Hi Stephan,

W.r.t. the partitioning of the 'information' model, some rules of thumb could be:

1) to ensure process autonomy, the model should be granular enough to allow each attribute of a topic to be 'sensible' for a publisher/writer (since you can't write a 'half' topic without taking additional measures w.r.t. 'default-values' )

2) to increase performance, it makes sense to separate slow-changing and fast-changing parts in the information into separate topics since samples of a single topic are also distributed 'as a whole'

3) to increase efficiency, it also makes sense to distinguish between reliable and best-effort data (easiest is again to create differen topics for each) since reliable distribution incurs more overhead (its a heavier-weight protocol than for best-effort distribution). Typically for periodic 'measuremen' like data, a best-effort delivery is sufficient since re-sending old data might not make much sense (better wait for the next 'new' data). For one-shot and/or state-related data, reliable delivery is typically required since you can't afford to loose it.

4) again to increase efficiency (and with that performance) it also makes sense to distinguish between volatile and non-volatile data since non-volatile data must be 'maintained' by the middleware (either as 'TRANSIENT' data that is maintained while the system is 'up' or as PERSISTENT data that is maintained on disks) for late-joining subscribers. Please note that unlike most DDS vendors, OpenSplice supports both TRANSIENT as well as PERSISTENT QoS for transparant information-management in a distributed and fault-tolerant way (i.e. without relying on the presence of the original publisher of the information). This can be instrumental in creating system-fault-tolerance by allowing applications to publish their internal state as TRANSIENT data into DDS which can be picked-up by the same applications after a crash (even if they are restarted on another machine) so that they can continue with their business logic in the correct state.

A system with thousands of topics and hundred-thousands of topic-samples I would classify as 'medium'. When you really mean 100.000 different topics, then you need to realize that this implies quite a lot of meta-data (since each topic is related to an instance of a socalled "built-in-topic" that describes that topic). I wouldn't say that its impossible though, you just need enough memory to hold it. Speaking of memory/footprint, its also important to realize that different vendors apply different architectures that impact footprint. In OpenSplice DDS for instance, the shared-memory architecture as explained in my previous response ensures that data is never duplicated for each 'interested party' within a node.

W.r.t. the shared-memory-transport, this is indeed available under the community license and unlike other DDS implementations is not an "either/or" configuration option but is transparently (and allways) used for communication within a physical processing box whereas the communication between different boxes is transparantly taking care of by an elaborate 'network-scheduler' that schedules the transports based upon urgency (the 'latency budget' QoS at which data is published) and 'importance' (the 'transport-priority' QoS at which data is published). We support multiple priority-lanes to allow peer-to-peer priority-preemption w.r.t. delivery of information as well as provide traffic-shaping on each channel to preserve the proper bandwidt on the network.

Regards,
Hans
Next message Ulf Sjöstedt  posted on Thursday, February 19, 2009 - 10:09 am
Hi!

Point 2) above intrigues me somewhat. Would the splitting into fast-changing / slow-changing topics make sense even for data which is logically part of the same information-level entity?

Say you're keeping track of F1 cars in a race (go Hamilton!).

For an individual car certain attributes, such as its current position in the field, would change rarely (typically several seconds between changes). Other attributes, such as its current engine RPM, speed, acceleration, wheel positions etc, might change every few milliseconds.

Would it make sense to split this information into several topics or not?

The case for splitting it is that there are orders of magnitude difference in update rates.

The case against it is that you then introduce a relationship between topics which DCPS has no mechanism for dealing with (no explicitly shared key types).

In this case, you'd presumably use Presentation Group / Coherent.

How expensive would that be?
Next message Emiel Schuurman  posted on Thursday, February 19, 2009 - 03:58 pm
Well imagine the following topic definition:

struct FormulaOneCar
{
short carId; /* just some id */
short position; /* position in field, slow updating*/
string driverName; /* doesnt change */
long engineRPM; /* frequently updated, like every few milliseconds */
long speed; /* speed in km/h, updated every few milliseconds */
long laps; /* number of laps raced */
};
/* We'll define the 'carId' as the key for this topic. */
#pragma keylist FormulaOneCar carId;

Now if you were to distribute that topic, then you would be forced to send over the entire topic every few milliseconds because that is how often the speed and engineRPM attributes are updated. But things like the driverName, position and number of laps changes much less frequently, so you are resending the exact same data multiple times, basically wasting bandwidth and processor power, as you send too much data and have to de-serialize and serialize it every time. In general you would want to avoid that, as in bigger system that will be killing.

So as Hans explained, it would be better to try and split this data up, as it would increase performance if the middleware has to perform less work. So you could define the following topics:

struct FormulaOneCar
{
short carId;
short position;
string driverName;
long laps;
};

#pragma keylist FormulaOneCar carId;

struct FormulaOneCarStats
{
short carId;
long engineRPM;
long speed;
};

#pragma keylist FormulaOneCarStats carId;

With the above definition you have achieved that the frequently updated data has become just a small subset of the entire relevant data, to achieve this we placed the two fast updating attributes into a separate topic (FormulaOneCarStats), and we added the same key (carId) to that topic. The key allows us to relate the two topic together.

The DLRL API as defined in the DDS specification makes use of the above topic definitions even easier. It allows you to define one object, which consists out of multiple topics! Which means that for your application you just see the big FormulaOne car object with ALL data, but the DLRL logic will ensure that in fact only parts of the object are updated with incoming topic data and when you change some data that only the changed topics are written into the system. Basically you'll have the advantages of splitting up your topics in terms less load on the node and less data distributed, without the disadvantages of having write code in your application to look up and match topics together, and the DLRL can perform these tasks very efficiently because it can, behind the scenes, access non public APIs to perform the tasks.

But the DLRL offers alternative ways to model this as well. You could also choose to model two objects (FormulaOneCar and FormulaOneCarStats) and create a relationship between the two objects (based on their shared key 'carId'), then you simply have to navigate from one object to the other (just like in any OO language) to retrieve your data, no difficult mechanisms needed to relate topics etc, this is all done by DLRL in a very efficient manner!

Whichever option is chosen kinda depends on the specific situation, sometimes embedding two topics into one object makes more sense logically, and at other times a relationship between two objects makes more sense logically. But both ways are easy to use and efficient when talking in terms of the DLRL API of DDS.
Next message Ulf Sjöstedt  posted on Friday, February 20, 2009 - 11:14 am
Agreed that DLRL Magically Makes Things Easier (TM). :-)

But DLRL is only optional in the standard. With DCPS, I'd most likely end up having to use a Group Presentation policy as outlined above (since I don't want to push the information-level reassembly problem onto the subscribers).

So when I created or deleted a pair of instances of the two topics, I'd use begin/end_coherent_changes(). When updating only the FormulaOneCarStats topic I wouldn't need to bother.

In the Subscriber, I wouldn't know beforehand whether there was a set of coherent changes coming, so I'd need to do begin/end_access() each time.

#1: Have I understood this correctly?

#2: How costly is that solution?

Also, is there anything going on within the standard to introduce sparse updates in DCPS?.
Next message Hans van 't Hag  posted on Thursday, March 05, 2009 - 08:59 am
Hi Ulf,

A few comments

1) the 'GROUP' presentation QoS policy is part of the same optional 'Object model Profile' that DLRL is also part of.

2) In many cases, a system can be designed such that you don't need to publish related topic as a coherent set. I've been active for many years in developing naval combat systems that are fully based on this technology and where for instance the notion of a 'track' is represented by as many as 52 individual yet related topics and without having to rely on coherent updates. There is alwasy some business-logic that takes care of 'collecting' the information you need and to react on lifecycle changes (like when an observed outside-world 'track' disappears, i.e. gets out of range', its 'lifecycle-topic' goes to 'track-lost' which also invalidates position-udates sent in the past. The DLRL greatly facilitates such business-logic by taking away the burden of the 'collecting' and 'maintaining-relationships' whereas fine-grained triggers on individual attributes as well as the possbility to embed business-logic in the local Object all make life pretty easy.

3) w.r.t. 'sparse updates', yes the good news is that there is a 'work-in-progress' within the OMG that addresses the need for evolutionary information models including the need for 'sparse updates'. Please checkout http://www.omg.org/cgi-bin/doc?mars/2009-02-06 for detailed information on this extension of the DDS specification.
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Post as "Anonymous"
Enable HTML code in message
Automatically activate URLs in message
Action: