Clean the raw data and build a daily mart with code

The src_weather_hourly dataset from lesson 03 preserves the API response. This lesson adds a second Python code resource that gives the columns readable names and summarizes 72 hourly rows into three daily rows.

collect_weather_hourly → src_weather_hourly → prepare_weather_daily
                                              ├→ stg_weather_hourly
                                              └→ mart_weather_daily

Confirm the result datasets

Lesson 03 created these datasets:

• stg_weather_hourly: observed_at, observed_date, temperature_c, humidity_pct
• mart_weather_daily: observed_date, avg_temp_c, max_temp_c, min_temp_c, avg_humidity_pct, sample_count

Use Double for avg_humidity_pct and Bigint for sample_count.

Create the preparation code

1. In the practice collection, choose Add item → Code → Python.
2. Enter prepare_weather_daily for both the name and alias.
3. Add a short description and choose Next.

Replace the editor contents with the following code and choose Create.

def run(src_weather_hourly, options=None, contexts=None):
    import polars as pl

    frame = src_weather_hourly.rename({
        "temperature_2m": "temperature_c",
        "relative_humidity_2m": "humidity_pct",
    })
    daily = (
        frame.group_by("observed_date")
        .agg(
            pl.col("temperature_c").mean().alias("avg_temp_c"),
            pl.col("temperature_c").max().alias("max_temp_c"),
            pl.col("temperature_c").min().alias("min_temp_c"),
            pl.col("humidity_pct").mean().alias("avg_humidity_pct"),
            pl.col("observed_at").count().alias("sample_count"),
        )
        .sort("observed_date")
    )
    return {
        "stg_weather_hourly": frame,
        "mart_weather_daily": daily,
    }

Dataset inputs arrive as Polars DataFrame objects, so use rename() and group_by() rather than pandas copy() and groupby().

Connect the pipeline

1. Open weather_daily_pipeline and expand the Component Library.
2. Drag prepare_weather_daily, stg_weather_hourly, and mart_weather_daily onto the canvas.
3. Connect src_weather_hourly to prepare_weather_daily.
4. Connect the code output to both result datasets.

Set full read and overwrite

Select prepare_weather_daily and open Options.

1. Expand the src_weather_hourly input and choose Read mode → Full.
2. Expand both outputs and choose Write mode → Overwrite.
3. Save the inspector and then save the pipeline.

Run and verify

Choose Run now and confirm that both code nodes succeed. The staged dataset must contain 72 rows with temperature_c and humidity_pct; the mart must contain three dates with sample_count equal to 24.

Self-check

• prepare_weather_daily uses Polars syntax.
• Its input uses Full and both outputs use Overwrite.
• The staged dataset has 72 rows.
• The mart has three rows and every sample_count is 24.

Next lesson

Next, schedule this verified pipeline to run automatically.