logged responses should be made available to an offline log table in Hive
with columns
key_bytes, value_bytes, ts_millis, join_name, schema_hash and ds (date string)
partitioned by join_name and ds
Note the camel case to snake case conversion: Hive doesn't like camel case.
logged responses should be made available to an offline log table in Hive
with columns
key_bytes, value_bytes, ts_millis, join_name, schema_hash and ds (date string)
partitioned by join_name and ds
Note the camel case to snake case conversion: Hive doesn't like camel case.
The key bytes and value bytes will be transformed by chronon to human readable columns for each join.
<team_namespace>.<join_name>_logged
To measure consistency - a Side-by-Side comparison table will be created at
<team_namespace>.<join_name>_comparison
Consistency summary will be available in
<logTable>_consistency_summary
Register external sources from join configurations.
Register external sources from join configurations. This function reads configurations for each join, extracts FactoryConfig information from external sources, and stores this information in the ExternalSourceRegistry.
Sequence of join names to process