I wanted to share my experiences and highlight the things that I found most valuable while creating a Dedicated SQL Pool instance in Azure that is used by one of my clients to query and analyze over 5 billion rows of data. This is not a comprehensive list of the capabilities or best practices for Azure Dedicated SQL Pool. This is more of list of things that got me my best bang for my buck.
So you have a web app up and running in Azure and now you need to do some background work that’s related to your new web site but it definitely doesn’t belong in the web app itself. This could be anything from importing data, to sending out subscription emails, to updating cached data, or running database maintenance.
You may be wondering, with so many viable options at your fingertips in Azure, what is the best and most cost effective way to get work done? You want to strike the balance between having something that’s powerful enough to grow if it needs to grow but you don’t want to incur a bunch of unnecessary costs right out of the gate by choosing too much power!
The task at hand for me was that I needed to import and transform several different Excel formats into an existing SQL user table. The formats were not guaranteed to always be the same and I needed a way to quickly add new formats without writing a bunch of new code each time.
This past month I finally had a business case that called for Azure Event Hubs to be part of my overall solution to a real-world problem. Up until this point I was always able to get by mostly with plain old Table Storage Queues and sometimes Service Bus Topics, but the business case I started working on this past month involved sending many megabytes of information from potentially thousands of devices around the clock.
DocumentDB is a great NoSql database engine for working with big data. One thing that was difficult to implement though was an intelligent way to page through the results. In this article I will show how I used a combination of caching at the client side of my app along with utilizing continuation tokens from the SDK response to build a decent paging mechanism for big data queries using DocumentDB.
It seems to be a common story with almost every software dev I meet; they have a laptop full of side projects that were started many months or years ago but, sadly, never saw the light of day. Many of my projects have met the same fate. Why is this so common?
I think the answer is that it's easy to start something, it's very hard to finish it.
There's a valid psychological reason why most of us are more likely to remember a 10 digit phone number rather than the first 10 digits of the value of Pi. It has to do with how we commit information to our short term memory. We are much better at memorizing lists or numbers in chunks. In UI design it's called chunking and it's the technique of breaking many units of information into smaller, limited units or chunks, so that the information is easier to process and remember.
When you begin the habit of writing code to solve problems that you don't have - that's a real problem. Joel Spolsky famously called these people Architecture Astronauts. They create such ambiguous, high-level abstractions in the code that at a certain point the code becomes unreadable and, for all intents and purposes, unmanageable.
There is a somewhat lesser known space trilogy from CS Lewis and in the first book a human from Earth visits a foreign planet that is mainly inhabited by three different classes of physical creatures: the Sorns, the Hrossa, and the Pfifltriggi. At one point in the story the human asks one of the Sorns (its name is Augray) where he got his impressive oxygen mask from. Augray answers the human somewhat sheepishly...
