As far as I can gather most implementors of Durability use a "service" sitting on the network somewhere to implement DURABLE. This service (I guess) is nothing more than a subscriber on all topics which hardens them to disk and then replays them when it recognises a new consumer has come onto the network.
So (assuming this is roughly correct) , here's a few questions I have please...
1) new and old messages: Isn't there potential for both new and old consumers to get confused - if there are still producers producing on the topic and a new consumer comes onto the network then the durability service will replay old messages - I think I understand that consumers who are already on the network will filter those out and ignore them because they understand they're old messages (or perhaps they understand they've seen thenm before?). However, the new consumer will get both the messaqes from the durability server plus the messages being published from the current publisher. This suggests that the replayed messages must have something special about them in order that the new consumer recognises that they are older than the new messages being published by the current publisher - what is this thing that helps them tell old messages from new ones?
2) Durability service GUID: Does the durability service publish messages as a new publisher or do they re-use the original publisher GUID's ?
3) Liveliness: Given question two. If the durability service publishes as a new publisher - what if the topic is single ownership - will the messages ever be received? I assume not unless the durability service always has a higher ownership strength but then any current publishers will send out newer (and one assumes therefore more important messages) and these will be ignored until the durability service has finished its replay - and the replay may happen time and again, if consumers keep coming onto the network.
4) Durability publisher QOS: If the messages were published on a shared ownership topic and the messages were published by multiple publishers with different QOS does the durability service represent each publisher or does it take e.g. the lowest common denominator QOS and publish using those? AND How does it know that it's the publisher of the messages and not to put them back into the durability service?
5) regex partitions: If the topics were published to a partition on a regex then the durability service must listen on each partition individually - how does it do this - it's not possible, unless manually set up by an administrator. What partition does it then publish the messages to - I assume the absolute partition.
many thanks - I realise there are a lot of questions but as i wrote I thought of more issues :-)
The durability capabilities of the different DDS implementations are very different. Keeping historical data in one location is not very fault-tolerant, but keeping all data everywhere might not be feasible as well. The desired behaviour is driven by performance requirements (boot time, memory usage, network load, CPU load) on the one hand and fault-tolerance requirements (the need for replicates) on the other hand. I know OpenSplice DDS supports any combination of these configurations, making the mechanism quite sophisticated and suitable for all needs. Let me try to answer your questions: 1. That may depend on the implementation, but the new messages should only be delivered to new Subscribers, so the existing ones don't need to filter. 2. The right thing to do is using the original DataWriter info (source timestamp, identity, QoS policies, etc), but the behaviour may depend on the DDS implementation. 3. Not applicable if the original Publisher info is used as stated in 2. 4. The TopicQos is leading in determining how to handle the data in the durability implementation. For ownership goes that RxO requires the setting for the DataWriter and DataReader to be exactly equal so there cannot be one common denominator in all cases. Data not matching the ownership of the TopicQos will not be stored by a durability implementation. One can use two different Topics with the same type (one with exclusive and one with shared ownership Qos policy) and publish the topic that matches the desired ownership to make sure the durability implementation will store both shared and exclusive data. 5. In my opinion using regular expressions for partitions on the publishing side is quite doubtful. This is because no actual partitions are created when using an expression. Consider for instance two subscribers that subscribe to partition 'aa' and 'ab' respectively and a publisher that publishes in partition 'a*'. The resulting behaviour depends on the order in which the system is started. When the subscribers are started first, everything works fine and data is published in both 'aa' and 'ab'. The durability implementation is able to keep the data for late joining Subscribers. However, if the Publisher is started first, nothing is stored because there is no actual partition in the system. Only when starting the subscribers, the partitions are created and communication takes place (and data is stored). This means there is no historical data for late joining subscribers from before the partition was actually created. Publishing in partition 'aa,ab' would solve the issue.