Class OutTopicEventPropertyLimiter

java.lang.Object
org.odpi.openmetadata.frameworkservices.omf.outtopic.OutTopicEventPropertyLimiter

public class OutTopicEventPropertyLimiter extends Object
Limits the size of each property in an out topic event, so that the event fits within the largest message that the event bus accepts (1 MiB by default for Apache Kafka). An event that is too large is dropped by the event bus and every listener misses the change: a Jacquard connector activity report listing 2,454 elements serialised to 1.3 MB and was lost this way.

A property whose serialised form is estimated to be larger than the limit is shortened: an array or map keeps as many of its leading values as fit, a string is truncated, and any other value is left out. The stored element is not changed - a listener that needs the whole value reads the element.

The size is estimated from the values rather than measured by serialising them, which would double the cost of publishing every event. The estimate allows for the class and type names that each value carries in JSON, which make up most of an event's size when a property holds many short values such as GUIDs.
  • Field Details

    • DEFAULT_MAXIMUM_PROPERTY_SIZE

      public static final int DEFAULT_MAXIMUM_PROPERTY_SIZE
      Default limit for one property: an event carries a property in its current and previous properties, so several properties at the limit still fit well within 1 MiB.
      See Also:
  • Constructor Details

    • OutTopicEventPropertyLimiter

      public OutTopicEventPropertyLimiter()
      Constructor using the default limit.
    • OutTopicEventPropertyLimiter

      public OutTopicEventPropertyLimiter(int maximumPropertySize)
      Constructor.
      Parameters:
      maximumPropertySize - largest estimated size of one property, in bytes
  • Method Details

    • limitProperties

      public ElementProperties limitProperties(ElementProperties properties, List<OutTopicEventPropertyLimiter.ShortenedProperty> shortenedProperties)
      Return the properties with any property that is too large shortened. The properties passed in are not changed; if nothing needs shortening, they are returned as they are.
      Parameters:
      properties - properties for the event
      shortenedProperties - list to add a description of each shortened property to
      Returns:
      properties to send