Blog posts
All Microsoft Fabric icons for diagramming
Do you often have to draw up diagrams of data flows or data architectures and would you like to use Microsoft Fabric icons in your diagrams? Then this post is for you! I always found it hard to find the right icons I needed, so I’ve put them all on this single page for easy access, both as PNG and SVG.
I just took the new Fabric DP-700 Data Engineering Exam: here's what you should know
I took the new DP-700 exam, released just hours ago in beta, and I’m here to share my experience along with some tips to help you prepare.
Connecting Neon with dbt Cloud
If you’re looking to build a simple, serverless data stack, then you might have considered the amazing combo of dbt Cloud with Neon. Neon is a fully managed cloud offering with serverless PostgreSQL databases. They separate storage and compute, which makes it easy to scale and manage your data. If your data is not “big”, then Neon is a great choice. dbt Cloud is a cloud-based service for running dbt. It’s a great way to run dbt without having to manage infrastructure. It features a user-friendly IDE, scheduling, and monitoring.
Fabric: Lakehouse or Data Warehouse?
There are 2 kinds of companies currently active in the Microsoft data space: those who are migrating to Microsoft Fabric, and those who will soon be planning their migration to Microsoft Fabric. 😅 One question that often comes back is Should I focus on the Lakehouse or the Data Warehouse? Let’s answer that in this post. I can already tell you this: you’re asking the wrong question 😉
Is Microsoft Fabric just a rebranding?
It’s a question I see popping up every now and then. Is Microsoft Fabric just a rebranding of existing Azure services like Synapse, Data Factory, Event Hub, Stream Analytics, etc.? Is it something more? Or is it something entirely new? I hate clickbait titles as much as you do. So, before we dive in, let me answer the question right away. No, Fabric is not just a rebranding. I would not even describe Fabric as an evolution (as Microsoft often does), but rather as a revolution! Now, let’s find out why.
My take-aways from Big Data London: Delta Lake & the open lakehouses
Last week I attended Big Data London. Both days were filled with interesting sessions, mostly focussing on one of the vendors also exhibiting at the conference. There are 2 things I am taking away from this conference: Delta Lake has won the data format wars, and your next data platform is either Snowflake, either an open Lakehouse.
Fabric end-to-end use case: Analytics Engineering part 2 - Reports
Welcome to the fifth part of a 5-part series on an end-to-end use case for Microsoft Fabric. This post will focus on the analytics engineering part of the use case. In this series, we will explore how to use Microsoft Fabric to ingest, transform, and analyze data using a real-world use case.
Fabric end-to-end use case: Analytics Engineering part 1 - dbt with the Lakehouse
Welcome to the fourth part of a 5-part series on an end-to-end use case for Microsoft Fabric. This post will focus on the analytics engineering part of the use case. In this series, we will explore how to use Microsoft Fabric to ingest, transform, and analyze data using a real-world use case.
Fabric end-to-end use case: Data Engineering part 2 - Pipelines
Welcome to the third part of a 5-part series on an end-to-end use case for Microsoft Fabric. This post will focus on the data engineering part of the use case. In this series, we will explore how to use Microsoft Fabric to ingest, transform, and analyze data using a real-world use case.
Fabric end-to-end use case: Data Engineering part 1 - Spark and Pandas in Notebooks
Welcome to the second part of a 5-part series on an end-to-end use case for Microsoft Fabric. This post will focus on the data engineering part of the use case. In this series, we will explore how to use Microsoft Fabric to ingest, transform, and analyze data using a real-world use case.
Fabric end-to-end use case: overview & architecture
Welcome to the first part of a 5-part series on an end-to-end use case for Microsoft Fabric. This post will focus on the architecture overview of the use case. In this series, we will explore how to use Microsoft Fabric to ingest, transform, and analyze data using a real-world use case.
Let Fabric teach you how to code with Data Wrangler
I’m going to be honest with you. I’m bad at writing Pandas data transformation code. Throughout the years I mostly focussed on the Spark APIs in Scala and PySpark, SQL, dbt, and some others, but I find the Pandas APIs usually just confusing and hard to read. I don’t like the black box magic and lock-in of low-code solutions either. Did you know that Microsoft Fabric has the perfect middle ground for this? It’s called Data Wrangler. Let’s dive in!
How to use service principal authentication to access Microsoft Fabric's OneLake
Microsoft recently added support to authenticate to OneLake using service principals and managed identities. This allows users to access OneLake from applications without having to use a user account. Let’s see how this works.
A closer look at Microsoft Fabric pricing, billing, and autoscaling
If you’re considering using Microsoft Fabric, you’re probably thinking “How much is this going to cost me?” Continue reading to learn how Microsoft might have just created the most compelling data platform offering available today.
Migrating Azure Synapse Dedicated SQL to Microsoft Fabric
If all those posts about Microsoft Fabric have made you excited, you might want to consider it as your next data platform. Since it is very new, not all features are available yet and most are still in preview. You could already adopt it, but if you want to deploy this to a production scenario, you’ll want to wait a bit longer. In the meantime, you can already start preparing for the migration. Let’s dive into the steps to migrate to Microsoft Fabric. Today: starting from Synapse Dedicated SQL Pools.
Connect to Fabric Lakehouses & Warehouses from Python code
In this post, I will show you how to connect to your Microsoft Fabric Lakehouses and Warehouses from Python.
Preparing a migration to Microsoft Fabric: from Azure Synapse Serverless SQL
If all those posts about Microsoft Fabric have made you curious, you might want to consider it as your next data platform. Since it is very new, not all features are available yet and most are still in preview. You could already adopt it, but if you want to deploy this to a production scenario, you’ll want to wait a bit longer. In the meantime, you can already start preparing for the migration. Let’s dive into the paths to migrate to Microsoft Fabric. Today: starting from Synapse Serverless SQL Pools.
Microsoft Fabric's Auto Discovery: a closer look
In previous posts , I dug deeper into Microsoft Fabric’s SQL-based features and we even explored OneLake using Azure Storage Explorer . In this post, I’ll take a closer look at Fabric’s auto-discovery feature using Shortcuts. Auto-discovery, what’s that? Fabric’s Lakehouses can automatically discover all the datasets already present in your data lake and expose these as tables in Lakehouses (and Warehouses). Cool, right? At the time of writing, there is a single condition: the tables must be stored in the Delta Lake format. Let’s take a closer look.
Exploring OneLake with Microsoft Azure Storage Explorer
Recap: OneLake & Delta Lake One of the coolest things about Microsoft Fabric is that it nicely decouples storage and compute and it is very transparent about the storage: everything ends up in the OneLake. This is a huge advantage over other data platforms since you don’t have to worry about moving data around, it is always available, wherever you need it.
Welcome to the 3rd generation: SQL in Microsoft Fabric
While typing this blog post, I’m flying back from the Data Platform Next Step conference where I gave a talk about using dbt with Microsoft Fabric . DP Next Step was the first conference focussed on Microsoft data services right after the announcement of Microsoft Fabric so a lot of speakers were Microsoft employees and most of the talks had some Fabric content. Fabric Fabric Fabric, what is it all about? In this post I’ll go deeper into what it is, why you should care and focus specifically on the SQL aspect of Fabric.
Filling the gaps in your code with the Terraform azapi provider for Azure
The cloud is just someone else’s computer and to manage that we prefer to use Infrastructure as Code (IaC). dataroots believes that IaC can benefit any team working with cloud resources and most often Terraform is our tool of choice there. As a data & cloud engineer focusing on Microsoft Azure, that is true for me as well.
Some interesting takeaways from this year's Techorama
Last week was a busy week for fans of the Microsoft technology stack like myself. Microsoft hosted its yearly developer conference, Microsoft Build, announcing lots of exciting updates to new and existing Azure services. In the meantime, the Belgian community of Microsoft technology users gathered in Kinepolis Antwerp for this year’s edition of Techorama.
Setting up your machine for local Terraform provider development
If you’ve been developing Terraform providers, you might have upgraded your machine to use the newly released Terraform 0.13. This version introduces a lot of changes related to where providers can be found. Before version 0.13 you would just put the binary of your provider in ~/.terraform.d/plugins and you could start using your provider.
Installing the Azure Event Hubs Python SDK on Raspberry Pi OS 64-bit
Since we’re going through some heat waves in Europe, I thought it might be interesting to start measuring the humidity, temperature and pressure in my apartment. To do so, I decided to use my Raspberry Pi 3 and the Pi Sense HAT running a Python script constantly sending measurements to an Azure Event Hub.
Debugging a Terraform provider
While contributing to a couple of Terraform providers I often found the need to set some breakpoints in the provider code and inspect what was going at runtime. However, debugging Terraform providers is not that easy apparently… I found a way to do this VS Code, but feel free to drop me a line if you have some other tips & tricks and I’ll update this post.
Easy localization in Flutter with continuous integration
Did you know Flutter has built-in support for l10n, also known as localization? I’ve been searching for an easy way to localize my app, but most of the blog posts I’ve found rephrased the official documentation on internationalization . I live in Belgium where we have 3 official locales: Flemish (nl_BE), Belgian French (fr_BE) and German (de_BE).
Create App Previews with the iOS Simulator
I don’t always have a ton of iOS devices lying around when I submit iOS apps to iTunes App Store Connect. So I take my screenshots on the simulators. At the time of writing Apple requires screenshots made with an iPhone 8 Plus and an iPad Pro. You can optionally add screenshots from an iPhone XS Max.
Tips for developing Android JobScheduler Jobs
If your app has to sync some data in the background, you’ll definitely want to use the Android JobScheduler API to schedule a background job. There are a few helpful libraries that make this task easier, but you can get away with using the JobScheduler API itself. I work in Xamarin, so my samples are in C#.
Optimize memory usage in Xamarin apps
This post has been translated to Russian by Denis Gordin . You can read the Russian version on the Russian website TechMedia . Thanks, Denis! Xamarin is amazing in how it allows .NET developers to write apps for Android, iOS, MacOS… in C#. But that amazing capability comes with a prize and even the most simple apps can suffer from high memory usage.
A few common issues with .csproj files in Xamarin apps
Usually you don’t need to manually edit .csproj files for your apps and most developers don’t even know what’s going on inside this file. However, sometimes you might run into issues related to this file where Visual/Xamarin Studio can’t help you. The case of the missing references At my current client our solution consists of more than 10 projects:
Diagnosing memory issues with the Xamarin profiler
The Xamarin profiler is a must-have tool for every Xamarin developer. The profiler comes with a Xamarin business or enterprise license and is available as a standalone installer at xamarin.com/profiler . To get started, make sure a debug version of your app is installed on a device or a simulator (works both for Android and iOS).
Creating a Xamarin.iOS binding project for dummies
What you need Experience with Xamarin.iOS Xamarin Studio for Mac An empty binding project (just create a new project in Xamarin Studio) A very short intro to Objective-C for C# developers Oh god, Obj-C, the most incomprehensible programming language in the app dev world. You simply can’t create an iOS binding project without some very basic knowledge of Obj-C.
Fix common binding errors with MVVM Light on Xamarin
There isn’t much documentation available for MVVM Light when it comes to Xamarin.Android and Xamarin.iOS. There are several overloads for the SetBinding method and using the wrong overload causes TargetInvocationException or TargetException like this one . It’s also possible that your bindings don’t update anymore after you set one binding using an incorrect syntax.
Setting up a Raspberry Pi as a git server for Windows clients
Sometimes you’re not allowed or you don’t want to publish source code on GitHub, Bitbucket etc. In this post, I’ll explain how you can set up a Raspberry Pi (or any other Linux server) as a git repository server and how you can configure Windows clients for that server. We’ll automate everything from authentication to creating and deleting repositories.
Dependency injection with Autofac and MVVM Light in Xamarin
You gotta have MVVM A developer and his tools are inseparable. We all like SOLID and every (.NET) developer has his or her favourite dependency injection tool. There is a lot to choose from. I like Autofac because of the way it handles modules, the lifetime of a type and how it registers types.
Run Docker on Hyper-V with Docker Machine
Docker is awesome, right? And thanks to boot2docker Windows users were no longer left out of the fun. Still, setting everything up could be a PITA and you had to install Oracle VirtualBox to use it as Docker containers were actually run inside of a VM (which was the purpose of boot2docker).
How to fix common Hyper-V errors on Windows 10
This short post is more of a note to myself for when I run into the same problem. I just wasted an hour on it so I guess that justifies it. So, what happened? I wanted to check out the new Docker Machine with Hyper-V support, but the Hyper-V Manager told me it could not connect to my local computer.
Docker image for Hugo builds (with Wercker)
This blog is built with Hugo , which allows me to use some great CI tools like Wercker . The first part of writing a wercker.yml is picking a build container. A build container on Wercker is the environment in which your build or deploy steps run. Wercker used to build its own containers for that, but they’ve moved away from that approach to Docker containers.
Experiences with Windows 10 on a new Dell XPS 13 (2015)
Ordering a new workhorse Windows 10 RTM has been out for about a month now after little less than a year of public Insider preview builds. The motherboard of my old, trusty, Asus laptop was slowly dying and the Dell XPS 13’s thinness and battery life looked very attractive. I decided to go for it and order a Dell XPS 13 (2015), without the touch screen.
Wercker step to minify static resources
I recently blogged a lot about static websites and continuous integration . There was still one step missing in my continuous integration cycle: minification. Wercker is a great CI tool and it allows you to create custom build steps that can be reused in several projects. So I created a build step to minify HTML, CSS and JS files.
Wercker step to validate Hugo themes
So last week, I created a material design theme for Hugo , a static site generator that I’m quite fond of](/tags/hugo/). As I discovered Wercker , an awesome CI tool, I went looking for a way to automatically validate themes. There wasn’t any, so I simply wrote a build step for Wercker .
Continuous integration with Hugo and Wercker
Why? Who doesn’t love GitHub Pages ? It’s the easiest way to create a simple website about a repository and you can even use Jekyll to start blogging. As I recently switched from Jekyll to Hugo , I needed a new way to enable continuous integration for my blog. New to CI?
Material-lite theme for Hugo
Last week, I wrote a post about how awesome Hugo is and why I switched from Jekyll to Hugo. Then, a few days later, Google released an awesome template called Material Design Lite . It’s basically Google’s famous Material Design in HTML, CSS and JavaScript instead of in Polymer .
An introduction to Hugo, a static site generator
A good blogging platform What makes a good blogging platform really good? Well, that depends on the blogger. Programmers would need different features than make-up bloggers. As a programmer, I need to be able to easily integrate pieces of code with syntax highlighting in my posts and I’d prefer writing them in my favourite text/code editor instead of in a WYSIWYG editor.
Geocoding addresses in Talend Open Studio for Data Integration
Last week I had to geocode the addresses in a dataset using Talend. Geocoding is adding coordinates to addresses. This way you can easily visualize your data on a map. I experienced problems when I tried to download some plugins from Talend Exchange but I found this blog post about geocoding without the use of a plugin.
Natively boot Windows with a virtual HDD
A few days ago Microsoft released a new build for Windows 10 and I wanted to give it a try. However, I didn’t want to resize my partitions or overwrite my Windows 8.1 partition. Well, then install it in a VM! Then I wouldn’t have access to all of my RAM and CPU, so that wasn’t a solution either.
Queue for MessageDialog in Windows RT
When I write Windows Store applications, I use MessageDialog a lot. It’s the easiest way to show a quick informative pop-up message or a question to the user. However, when you tend to use this quite often, you’ll probably run into a problem. The Windows Runtime framework doesn’t allow you to stack MessageDialogs, queue them etc.
Workaround for password confirmation with Laravel & Ardent
While I was working on a website made with Laravel and Ardent I had to validate a password confirmation field for a user registration. This will not work if you follow the standard documentation. This is how your validation rules in your Ardent model should look like (users only have an email address and a password):
Using markdown to create any kind of document
A few weeks ago I had to submit the contents of a GitHub repository to my teacher for grading. I wanted to use my readme.md-file while my teacher asked for a Word or a PDF document. Then I discovered this amazing program which converts Markdown to any desirable format! Almost every programmer knows Markdown, right?
Secure SSH access with 2-step authentication (extended)
In this quick tutorial I’ll show you how to secure SSH access to your Linux server with 2-step authentication. Why did I call this post ’extended’? Because I’ll show you how to add extra rules so you don’t have to use 2-step authentication from certain locations. I’m not going to explain what 2-step authentication is .
Wake up your computer(s) with a simple email using a Raspberry Pi
Introduction I have a Spotify Premium subscription but I also have a few CDs which are not available on streaming services. I don’t like the whole process of syncing files, that’s why I took a Spotify subscription in the first place. Google offers a solution with Play Music : you can upload up to 20,000 songs to their servers for free and stream them to all of your devices.
Connecting Serviio with Telenet's Yelo TV
Introduction Recently Belgian ISP Telenet introduced Yelo TV . It’s the new brand for all of their efforts to innovate with television. Yelo TV includes a series of apps for almost every device (except for Windows Phone) and a website on which you can watch live television, watch recordings or schedule recordings.