Updated view definition for AttributeData delete ETL - #975
Conversation
| -- Jay Allen - Only propagate a delete if the kew no longer exists in the source. EventDataAndName | ||
| -- is not unique when CAMP holds duplicate attribute rows, so cleaning up duplicate | ||
| -- caused ETL to delete the legitimate value from exp.ObjectProperty | ||
| -- |
There was a problem hiding this comment.
the comment says "EventDataAndName is not unique..." but this query keys on PROC_ID, ATTRIB_KEY. Worth fixing so it doesn't mislead the next person reading this file.
There was a problem hiding this comment.
EventDataAndName is equivalent to PROC_ID + ATTRIB_KEY. forgot to include that in parentheses. fixed
| a.audit_date_tm | ||
| FROM audit.audit_coded_proc_attribs AS a | ||
| WHERE a.audit_action = 'D' | ||
| AND NOT EXISTS ( |
There was a problem hiding this comment.
A correlated NOT EXISTS per audit row is fine as long as CODED_PROC_ATTRIBS(PROC_ID, ATTRIB_KEY) is indexed, otherwise this becomes a per-row scan during larger backfills.
There was a problem hiding this comment.
I measured it against the actual db, and on a normal ETL run it does about 1500 quick index lookups rather than scanning the table, which is 4600 extra reads and 0.2 seconds. During a truncate and reset this takes 1.6 seconds in a job that runs for hours, so I don't think it makes that much of a difference. We can think about indexing the key value pair down the road, but for now I don't think it's necessary.
There was a problem hiding this comment.
when you say actual db was it your local dev data or a local copy of VGER animal db?
There was a problem hiding this comment.
I did a fresh restore of the animal db this morning
There was a problem hiding this comment.
lintilla copy dump is the same exact animal db that was on VGER at 6 am this morning
|
The teamcity build is still in progress just fyi |
…teETLOnDuplicateDelete
2 participants
No description provided.