Skip to content

Updated view definition for AttributeData delete ETL #975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jallentxbiomed merged 3 commits into from
Aug 10, 2026
Merged

Updated view definition for AttributeData delete ETL #975

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6d6bb8f
updated view definition for attribute delete ETL
jallentxbiomed Aug 7, 2026
91186ba
updated comment
jallentxbiomed Aug 7, 2026
dfa086e
Merge branch 'release26.3-SNAPSHOT' into 26.3_fb_fixAttributeDataDele…
jallentxbiomed Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions snprc_ehr/resources/source_queries/create_v_delete_snd_attributeData.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ CREATE VIEW [labkey_etl].[v_delete_snd_attributeData] AS
-- Note:
--
-- Changes:
-- Jay Allen - Only propagate a delete if the kew no longer exists in the source. EventDataAndName
-- (PROC_ID + ATTRIB_KEY) is not unique when CAMP holds duplicate attribute rows,
-- so cleaning up duplicate caused ETL to delete the legitimate value from exp.ObjectProperty
--

@RameshRapa RameshRapa Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the comment says "EventDataAndName is not unique..." but this query keys on PROC_ID, ATTRIB_KEY. Worth fixing so it doesn't mislead the next person reading this file.

@jallentxbiomed jallentxbiomed Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EventDataAndName is equivalent to PROC_ID + ATTRIB_KEY. forgot to include that in parentheses. fixed

-- ==========================================================================================

Expand All @@ -29,6 +32,12 @@ SELECT
a.audit_date_tm
FROM audit.audit_coded_proc_attribs AS a
WHERE a.audit_action = 'D'
AND NOT EXISTS (

@RameshRapa RameshRapa Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A correlated NOT EXISTS per audit row is fine as long as CODED_PROC_ATTRIBS(PROC_ID, ATTRIB_KEY) is indexed, otherwise this becomes a per-row scan during larger backfills.

@jallentxbiomed jallentxbiomed Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I measured it against the actual db, and on a normal ETL run it does about 1500 quick index lookups rather than scanning the table, which is 4600 extra reads and 0.2 seconds. During a truncate and reset this takes 1.6 seconds in a job that runs for hours, so I don't think it makes that much of a difference. We can think about indexing the key value pair down the road, but for now I don't think it's necessary.

@RameshRapa RameshRapa Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when you say actual db was it your local dev data or a local copy of VGER animal db?

@jallentxbiomed jallentxbiomed Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a fresh restore of the animal db this morning

@jallentxbiomed jallentxbiomed Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lintilla copy dump is the same exact animal db that was on VGER at 6 am this morning

SELECT 1
FROM dbo.CODED_PROC_ATTRIBS cpa
WHERE cpa.PROC_ID = a.PROC_ID
AND cpa.ATTRIB_KEY = a.ATTRIB_KEY
)
GO

GRANT SELECT on labkey_etl.v_delete_snd_AttributeData to z_labkey
Expand Down