This post is about module 6 in the Cloud Skills challenge.

Previous posts in this series:

Microsoft Learn Module

Ingest Data with Dataflows Gen2 in Microsoft Fabric

  • This module is about Dataflows Gen2, a cloud-based ETL (Extract, Transform, Load) tool, utilizing Power Query Online as the visual interface to perform the tasks much in the same way users might in Excel or Power BI for other types of modelling.
  • The exercises for this module have you creating a Lakehouse, and then a Dataflow using Power Query Online from a text file. The Lakehouse is the destination for the data, and as part of the exercise users will see how to map to columns in a table, append or replace existing data (when the dataflow is run), and view the data afterwards.

The session I watched on Day 5 was fantastic (no disrespect to whoever recorded the other session on the same topic!). Rishi (Sapra) and Cathrine (Willhemsen) knocked it out of the park. Most of the presenters so far were great, but some pairings worked well together to present an informative and entertaining session. This was one of them. The next one (Day 6) was just as great as you'll read about in my next post.

Key Takeaways

Dataflows Gen2 uses Power Query Online, which is beneficial to users like me who are very familiar with Power Query in Excel or Power BI. I'd heard of Dataflows Gen2 before of course but hadn't had a need or opportunity to try or use them. I like these kinds of skills challenges as I get a chance to try things and get hands-on with something new. I loved getting my hands on the Power Query Online interface, which isn't new but I do most of my work in Excel or Power BI Desktop, so PQO is not something I have seen much of yet.

One key limitation that is good to know is that Row-level security is not supported which, for me in the scenarios I have in mind, would be a deal-breaker. Otherwise, it simplifies data integration and where there isn't a lot of complexity, this would be a great solution to get data ingested.