The goal is to stand up the first end-to-end Jira → S3 → Snowflake → Power BI flow as a proof of concept — establishing a repeatable, governed pattern for moving operational data onto a central warehouse, with all transformation performed in the data layer rather than in the BI tool. The single-purpose Jira flow proves out the architecture before extending the same pattern to other sources.
Issues are pulled from the Jira REST API using the enhanced /search/jql endpoint with token-based pagination.
A Python extractor fetches issues for a target project and serialises them as newline-delimited JSON. Authenticates to storage via its IAM execution role.
Raw issue files land in a date-partitioned key layout (raw/jira/issues/YYYY/MM/DD/…) — an immutable landing zone for the warehouse.
Auto-ingest is triggered by S3 object-create events, streaming new files into the warehouse without manual loads.
Files land as raw JSON in a Raw database, preserving the source payload exactly as received.
SQL transforms flatten and shape the raw data into clean reporting tables and views (issue summary, open-by-assignee, cycle time).
Dashboards read the Mart layer directly. No transformation happens in the BI tool — it only visualises governed models.
/search/jql endpoint with token pagination, replacing the deprecated offset-based search.