Package org.apache.pinot.common.lineage

Class SegmentLineage

  • public class SegmentLineage
extends Object
    Class to represent segment lineage information. Segment lineage keeps the metadata required for supporting m -> n segment replacement. Segment lineage is serialized into a znode and stored in a helix property store (zookeeper). This metadata will be used by brokers to make sure that the routing does not pick the segments with the duplicate data. NOTE: Update for the underlying segment lineage znode needs to happen with read-modify-write block to guarantee the atomic update because this metadata can be modified concurrently (e.g. task scheduler tries to add entries after scheduling new tasks while minion task tries to update the state of the existing entry)

    • Constructor Detail

      • SegmentLineage

        public SegmentLineage​(String tableNameWithType)

    • Method Detail

      • getTableNameWithType

        public String getTableNameWithType()

      • addLineageEntry

        public void addLineageEntry​(String lineageEntryId,
                            LineageEntry lineageEntry)
        Add lineage entry to the segment lineage metadata with the given lineage entry id
        Parameters:
        lineageEntryId - the id for the lineage entry
        lineageEntry - a lineage entry

      • updateLineageEntry

        public void updateLineageEntry​(String lineageEntryId,
                               LineageEntry lineageEntry)
        Update lineage entry to the segment lineage metadata with the given lineage entry id
        Parameters:
        lineageEntryId - the id for the lineage entry to be updated
        lineageEntry - a lineage entry to be updated

      • getLineageEntry

        public LineageEntry getLineageEntry​(String lineageEntryId)
        Retrieve lineage entry
        Parameters:
        lineageEntryId - the id for the lineage entry
        Returns:
        the lineage entry for the given lineage entry id

      • getLineageEntryIds

        public Set<String> getLineageEntryIds()
        Retrieve the lineage ids for all lineage entries
        Returns:
        lineage entry ids

      • deleteLineageEntry

        public void deleteLineageEntry​(String lineageEntryId)
        Delete lineage entry
        Parameters:
        lineageEntryId - the id for the lineage entry

      • fromZNRecord

        public static SegmentLineage fromZNRecord​(org.apache.helix.zookeeper.datamodel.ZNRecord record)
        Convert ZNRecord to segment lineage
        Parameters:
        record - ZNRecord representation of the segment lineage
        Returns:
        the segment lineage object

      • toZNRecord

        public org.apache.helix.zookeeper.datamodel.ZNRecord toZNRecord()
        Convert the segment lineage object to the ZNRecord
        Returns:
        ZNRecord representation of the segment lineage

      • toJsonObject

        public com.fasterxml.jackson.databind.node.ObjectNode toJsonObject()
        Returns a json representation of the segment lineage. Segment lineage entries are sorted in chronological order by default.