📢Announcing our first Data Factory & Data Integration Contest!

[slindsay]

Medallion to Magic: Using Data Factory & Copilot to Transform Your Data to Be AI Ready

***April 2 update (not April 1 so you know we're serious 😉)*** 

We are extending the contest deadline so you all have more time to plan and to WIN! 

📅 Contest Timeline

• Contest opens: Monday, March 16
• Submissions close: Friday, May 8
• Judging period: May 11 - 14
• Winners announced: Friday, May 15

🎯 The Goal

This contest is about demonstrating a repeatable, end-to-end pattern for building AI-ready analytics in Microsoft Fabric by combining:

• Medallion data pipelines
• Metadata driven design
• Copilot assisted transformations
• Power BI semantic models prepared specifically for AI

The key idea: data and metadata are both first‑class inputs to AI — and your solution should treat them that way.

TL;DR:

We’re asking you to build a clear, end-to-end Fabric solution that uses a Medallion architecture: ingest raw data with pipelines, manage both data and metadata, transform it into Silver and Gold layers in a Fabric Data Warehouse (using a dbt job), and finish with an AI-ready Power BI semantic model. The goal is to show a repeatable pattern where well-designed pipelines and warehouse models lead to trustworthy Copilot answers in Power BI.

🏗️ What You’ll Build (At a High Level)

Teams will build an ELT‑style solution using Fabric where:

1. Raw data is ingested via Fabric Data Factory
2. Pipelines orchestrate data and metadata flows
3. Silver and Gold layers are created in a Fabric Data Warehouse
4. A dbt job is used to transform Silver → Gold
5. The final output is an AI-ready Power BI semantic model

This is not about one clever transformation - it’s about showing a pattern others could reuse.

✅ Architecture Expectations (Important!)

This challenge intentionally standardizes the Medallion architecture to remove ambiguity. The Fabric Data Warehouse is the authoritative storage layer.

• Silver and Gold must live in a Fabric Data Warehouse
• Silver and Gold must be schemas in the same warehouse
• dbt must target the Fabric Data Warehouse for all Silver → Gold transformations
• Lakehouses may be used for ingestion or exploration, but not for Silver or Gold

Data must follow a clear Bronze → Silver → Gold flow:

• Bronze: raw data from Microsoft and/or external sources, ingested using Fabric Data Factory with minimal transformation
• Silver: cleaned and standardized data written to a Silver schema in the warehouse and used as input to dbt
• Gold: curated, analytics‑ready models created via dbt, written to a Gold schema, and designed for Power BI and Copilot

Metadata is a required part of the architecture, not an afterthought.

• A dedicated metadata pipeline must be built in Fabric Data Factory
• Metadata must be captured across Bronze, Silver, and Gold, including tables, columns, data types, and business meaning
• Metadata must be orchestrated through pipelines, not manual documentation
• There must be a clear connection between metadata, downstream analytics, and the Power BI semantic model

All Silver → Gold transformations must be implemented in dbt, with dbt jobs containing the core modeling and business logic to demonstrate a repeatable, production‑ready Medallion pattern.

Copilot usage is required and must be intentional.

• Copilot should be used multiple times during data preparation
• At least two transformations must be generated or meaningfully enhanced by Copilot
• Transformations must go beyond renaming (e.g., business logic, structural changes, data quality improvements)
• We evaluate how Copilot helped, not just whether it appeared

The end‑to‑end flow must culminate in Power BI configured with Prep data for AI, including:

• A simplified schema that guides Copilot
• AI instructions defining business meaning and rules
• At least one verified answer with a human‑approved visual, trigger phrases, and storage in the semantic model

This is where your architecture, metadata, and modeling decisions come together.

🎥 What You Must Submit

Video Walkthrough (Required)

Each team must submit an end‑to‑end video walkthrough showing:

• The Medallion architecture and pipelines in Fabric Data Factory
• The metadata pipeline and how metadata flows with the data
• The dbt job transforming Silver → Gold in the Data Warehouse
• Copilot assisted transformations
• The final AI-ready Power BI experience
• Note: you can paste a video URL directly into the description of your entry

Clear storytelling matters just as much as technical correctness. Submit your entries in the AI-ready Data Gallery.

🏆 How Submissions Will Be Evaluated

Entries will be judged on:

• Clarity and correctness of the Medallion architecture
• Quality and intentionality of the metadata pipeline
• Effective use of Copilot in transformations
• Strength of Power BI AI readiness (schema, instructions, verified answers)
• Overall storytelling and reusability of the pattern

🎁 What You Win

• Bragging rights
• Exclusive swag
• A chance to work directly with the Data Integration product team
  

More importantly, you’ll help define what “AI-ready analytics” actually looks like in Fabric. We can't wait to see what you all come up with!!

Have questions? Drop them here in this thread and we'll answer as soon as we can!

[geekkailash]

I have few questions regarding this, will you answer this

1. What type of data will be provided for the challenge?
2. How many tables are expected at the end of each medallion layer?
3. Where should the completed work be submitted?
4. What is the approximate size of the dataset?

[HarshDubey]

Lets go

[slindsay]

@geekkailash - there's no starter data for this contest, you can source your own! Completed work must be submitted by April 10. I have asked the product team to jump in with answers to questions 2 and 4 🙂 Stay tuned! Thanks for asking, I am sure others have the same questions!

[MichaelaIsaacs]

Hey @geekkailash - thank you for your interest in the contest! To answer your questions, 2 and 4, there is no expectation for the number of tables, and the size of the dataset (but it's always recommended to wow the judges with the capabilitie of Data Factory 😊)!

[venkat-subha]

What about the resources — do we need to use our own, or will Fabric sponsor them?

[slindsay]

@venkat-subha - you will need access to a Fabric tenant (or a Fabric trial) to participate in this contest. We do have other contests that don't require Fabric coming up in the future - stay tuned to aka.ms/communitycontests for more info!

[geekkailash]

Fabric Trial version does not support copilot. 😞

[slindsay]

@geekkailash you are correct, you must have access to Copilot to be eligible to win this contest. You could do the rest of the end-to-end transformation and still submit - I am sure it would be a great learning opportunity! We'll have more contests in the future that don't include Copilot so you can participate!