Class OutTopicEventPropertyLimiter
java.lang.Object
org.odpi.openmetadata.frameworkservices.omf.outtopic.OutTopicEventPropertyLimiter
Limits the size of each property in an out topic event, so that the event fits within the largest message that
the event bus accepts (1 MiB by default for Apache Kafka). An event that is too large is dropped by the event bus
and every listener misses the change: a Jacquard connector activity report listing 2,454 elements serialised to
1.3 MB and was lost this way.
A property whose serialised form is estimated to be larger than the limit is shortened: an array or map keeps as many of its leading values as fit, a string is truncated, and any other value is left out. The stored element is not changed - a listener that needs the whole value reads the element.
The size is estimated from the values rather than measured by serialising them, which would double the cost of publishing every event. The estimate allows for the class and type names that each value carries in JSON, which make up most of an event's size when a property holds many short values such as GUIDs.
A property whose serialised form is estimated to be larger than the limit is shortened: an array or map keeps as many of its leading values as fit, a string is truncated, and any other value is left out. The stored element is not changed - a listener that needs the whole value reads the element.
The size is estimated from the values rather than measured by serialising them, which would double the cost of publishing every event. The estimate allows for the class and type names that each value carries in JSON, which make up most of an event's size when a property holds many short values such as GUIDs.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic final recordA property that has been shortened. -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final intDefault limit for one property: an event carries a property in its current and previous properties, so several properties at the limit still fit well within 1 MiB. -
Constructor Summary
ConstructorsConstructorDescriptionConstructor using the default limit.OutTopicEventPropertyLimiter(int maximumPropertySize) Constructor. -
Method Summary
Modifier and TypeMethodDescriptionlimitProperties(ElementProperties properties, List<OutTopicEventPropertyLimiter.ShortenedProperty> shortenedProperties) Return the properties with any property that is too large shortened.
-
Field Details
-
DEFAULT_MAXIMUM_PROPERTY_SIZE
public static final int DEFAULT_MAXIMUM_PROPERTY_SIZEDefault limit for one property: an event carries a property in its current and previous properties, so several properties at the limit still fit well within 1 MiB.- See Also:
-
-
Constructor Details
-
OutTopicEventPropertyLimiter
public OutTopicEventPropertyLimiter()Constructor using the default limit. -
OutTopicEventPropertyLimiter
public OutTopicEventPropertyLimiter(int maximumPropertySize) Constructor.- Parameters:
maximumPropertySize- largest estimated size of one property, in bytes
-
-
Method Details
-
limitProperties
public ElementProperties limitProperties(ElementProperties properties, List<OutTopicEventPropertyLimiter.ShortenedProperty> shortenedProperties) Return the properties with any property that is too large shortened. The properties passed in are not changed; if nothing needs shortening, they are returned as they are.- Parameters:
properties- properties for the eventshortenedProperties- list to add a description of each shortened property to- Returns:
- properties to send
-