Get up to the Stage¶
Great, now we’ve done some data cleaning and related items in the BASE layer. Now, let’s distance ourselves from the essential (Analytics) Data Engineering implementations and proceed on to some more interesting items– in the Staging layer!
You can find the video summary for this here.
The following are some of the things to consider in this implementation:
Column renaming
Required, if not implemented in BASE layer
JOIN data, if foreign keys exist
Filtering
Custom fields
E.g. transforming an existing column
For example, the following attempts to JOIN other tables through foreign keys in preparation for our data warehouse.
This can be done in the mart layer as well, but I like to do this here
-- Create the stg_customer_income_band table with a CONCATentation, transformation.
WITH source AS (
SELECT * FROM {{ ref('snowflake_demo_income_band') }}
),
final AS (
SELECT
ib_income_band_sk AS income_band_sk,
ib_lower_bound || '-' || ib_upper_bound AS income_band_bound
FROM source
)
SELECT * FROM final
The following is an example of creating a STG layer views for our Customer Catalog Sales:
I created all the staging models through this command.
dbt run --model stage.*
You can find other related dbt run commands here.