A sense of Wonder… reflecting on the #PowerBI amazing journey

Few hours left in the day, been an amazing day for a lot of reasons… better late than never, they usually say right?
:)

So… finally dumping a blog pots that formed on my mind going back several months (cof, years…) now… Anyway, today is even more relevant than then, so…
 
Warning: there won’t be fancy templates, crazy ideas,  strange tools, insights, snippets or new downloads on this post. in fact it will be a completely useless post for anyone reading I guess, I just need to get it out.  Don’t have a clear objective, or clear picture of what will come up. Also I won’t be slightly concerned if it’s too short, or too long, I won’t massively edit it (ok, slightly…). Fellow reader, I’m not writing for you today, I’m sorry.

Still, A-post-must-be-written.

So, how to start? How did we get here so fast? Seems only yesterday we were seeing this new Power BI Dashboards screenshots shown at WPC 2014. ~7-8 months after seeing Power BI GA then already a immensely powerful & stable Enterprise/Self-Service/even Personal BI platform, soon after that our first major project with the technology is online.

I’m not easily surprised, and yet time & time again I was completely baffled with what Power BI teams came up delivering, beyond user & community most requested features & needs, a lot others I honestly didn’t knew I was missing, and yet I can’t live without today.

Such an amazing journey… not in my wildest dreams, honestly I would say we would get here. And boy did we need it. Back in 2013-2014 remember we were getting very uncomfortable, mainly for the medium sized/enterprise BI scenarios, at least for any BI team that only works with Microsoft BI stack: powerful but completely stale OLAP engine, no clear roadmap,  failed products ("Planning" anyone?) , acquisitions that ended up bringing no value in the long term, huge number of disparate tools with poor integration, pieces that didn’t fit properly together aside from very superficial demonstrations. (we deliver solutions we hope can add value, not fancy toys that work great in demos)

The nagging feeling that just selling the latest tools was more important the adding value. I’m still a big believer that revenue is a side effect from value we bring for customers. Add value, then capture some of it so you can add more value, repeat.  You can make the other way work temporarily but it won’t be sustainable in the long term, and even if it were… that’s not the point right?   That was the  feeling honestly, yes, we had some great tools (heck PowerView  on SilverLight was great for its time, we couldn’t have Power BI today without “project crescent” right? :)  ), but still msbi kept lagging dangerously behind new vendors like Tableau, Qlick, etc in several key areas.

Something was missing, purely speculating here. Maybe that came finally with Nadella, also James Phillips strong vision, bringing the needed kind of startup mindset to Power BI vision (reminder, speculating here!).

Don’t have much visibility on Microsoft Power BI internal journey, for years know I’ve been inspired with the lean startup, customer development movement, mindset, and from the outside at least, seems a lean startup perfect case study. Hope one day some of that backstage story can be told, really.

A new or reinforced  why- building a platform that could add immense value through the use of data, none of the typical limiting constraints, an inspiring vision, a fine tuned learning machine, powered by real feedback, real needs, really hearing customers & msbi –still- strong and devoted community. And, not less amazing, brilliant , flawless execution. Incrementally, steadily, the pieces simply kept fitting together perfectly. How can these teams have such agility without never slightly compromising a stable, robust, and coherent platform down the road. I don’t know. It shouldn’t be possible. Am I dreaming?

Mission, vision, execution, brilliantly orchestrated.

To the unsung heroes, the amazing minds than could plan, design, envision such a brilliant architecture where pieces keep fitting magically together release after release… To the teams than then gradually brought it to life. Here’s to you. Know a few, but it’s pretty clear it’s only the tip of the iceberg. :)

I remember that right about Power BI GA back in ~July 2015, I was reading “Meaningful: The Story of Ideas That Fly” (truly inspiring book btw) and really felt I could be reading the story of Power BI there, the mindset, the why. Equally inspiring. heck. Even the book color reminded me of Power BI :)

From the book description…

”We don’t change the world by starting with our brilliant ideas, our dreams; we change the world by helping others to live their dreams. The story of ideas that fly is the story of the people who embrace them, love them, adopt them, care about them and share them. Successful ideas are the ones that become meaningful to others—helping them to see what’s possible for them.”

You see…? :)

Guess when you stop having customers and start having followers instead, amazing things are bound to happen.

Long post I, I warned right? Anyway, and also speaking of Azure, SQL Server,  Microsoft new support for Open Source platforms, R, Hadoop, Spark, so many others, such a powerful data platform we now have in our hands. Huge responsibility for us, data professionals, to take all this and deliver on the promise of immensely valuable data.

Should not surprise many, that I feel we’re not there yet (far from it, never been the one easily pleased with only gorgeous dashboards, sorry!). I believe we’re still capturing a tiny potential of the value hidden in all this data. But those thoughts will have to wait for another day. (hope it won’t take another year anyway…)  

Disclaimer: I don’t work for Microsoft, never have, although I work pretty much exclusively with Microsoft stack since I started my professional life, and also very sure, that I will stick with msft stack to the end of my work life. (that is, aside from my faithful Kindle and those precious things called moleskins )  

So, anyone still reading should have this in consideration. I may be biased… slightly…

Better late than never, Happy Birthday Power BI! :)

Rui

Exploring saved #Kindle Highlights with #SQL2012 Semantic Search

Kindle Highlights

I have said it before, archiving book highlights is one of the best features available on Kindle. You highlight your books on Kindle or any Kindle app and  then you can review them online whenever you want, and with a little effort… you can (you should actually, who knows what amazon will decide tomorrow…) save them locally for personal reference.

Anyway, I was reading about the new SQL 2012 Semantic Search feature and I thought it could be interesting  to run the new semantic search through my Kindle highlights.

There’s an amazing sample by Roger Doherty on codeplex called MySemanticSearch. I don’t know any other good app to explore (as in freely explore…) my highlights, so this seemed to be a very good start to explore the new SQL 2012 semantic search.

Prerequisites

Be sure to :

Getting my Kindle Highlights

First of all, cranking some PowerShell and using the kindle site, I got my very own highlights for several of my Kindle books :

image

Example:

image

Loading Streams into SQL using SQL 2012 FileTable feature

Using FileStreams and the new SQL 2012 Filetable I quickly loaded these into SQL:

— Create Database
CREATE DATABASE MySemanticSearch
ON PRIMARY (Name = FileTableDB, FILENAME = ‘D:\SQL2012\MSSQL11.SQL2012\MSSQL\DATA\MySemanticSearch.mdf’),
FILEGROUP FTFG CONTAINS FILESTREAM
(NAME = FileTableFS, FILENAME=’D:\SQL2012\MSSQL11.SQL2012\MSSQL\FileStream\MySemanticSearch’)
LOG ON
(Name = FileTableDBLog, FILENAME = ‘D:\SQL2012\MSSQL11.SQL2012\MSSQL\DATA\MySemanticSearch.ldf’)
WITH FILESTREAM (NON_TRANSACTED_ACCESS = FULL,
DIRECTORY_NAME = N’FileTableDB’);
GO

Then the FileTable

CREATE TABLE MyKindleHighlights AS FileTable
WITH
(FileTable_Directory = ‘MyKindleHighlights’);

GO

Loading files into BLOBs using SQL FileTable shared folders

Then just copied all my text files into the newly created folder for the FileTable (created automatically by SQL):

image

image

The table updates automatically, content/streams up and running!

image

Creating the full text/semantic index

You can do this with SQL of course (see here), I choose the lazy way, through Management Studio UI :)

image

image

image

image

image

image

image

Using semantic search

It’s now possible to use semantic queries to explore my highlights:

image

Well…. not so great, I don’t know exactly why, but my text files (I think apostrophes and some non printable chars) caused some confusion in the indexer, I get keywords like “isna”, “whata”… not good.

Reparsing the text files was not an option for now, I opt for adding some more stop words to the already existing in the system stop words list. For that I listed all the words in my semantic index:

 

image

I loaded that list in PowerShell and with a very rude dictionary check script borrowed from the web I listed  the words that were unknown nor anywhere in the files. With that I could now create a SQL Full text custom stop word list based on the already existing system stoplist:

CREATE FULLTEXT STOPLIST [HighLightsStopList]
FROM SYSTEM STOPLIST
AUTHORIZATION [dbo];

Using the PowerShell script to produce something like:

image

Configured the Fulltext index to use that stop list instead:

image

image

Run the query again, and as expected those noise words were now removed from my semantic queries:

image

Loading into MySemanticSearch codeplex sample

Time to load that FileTable into MySemanticSearch! Made a few visual adjustments, and a few others  so that it could work on my dev laptop (some strange problem with joining stream_id and document_key, I used path_locator instead of stream_id).

I opted for building MySemanticSearch from the source code, using VS 2010, so that took me some additional time to set up.

But finally all running! :) Connect to my sql 2012 instance:

image

It scans for available FileTables and finds MyKindleHighlights immediately:

image

Some interesting insights (remind you that these are from my biased highlights only, a very small subset of the complete books):

image

image

 

image

 

Final thougths

There seem to be some shortcomings (seems to handle unigrams only and does treats plurals/singular as completely different keywords, ex cat and cats… ). From a “semantic” search point of view I would expect a little better. May be I’m missing something, as this was a very initial test.

Nevertheless,  as an easily configurable and completely automated unstructured text mining tool now available in SQL Server, it is a very valuable addition to SQL Server.

Reference:

 

Rui

Fixing Amazon Kindle Daily Review page usability on touch enabled tablets – a case for “url site scripting” ;)

In a previous post (see Kindle eBooks as a great Learning/Self Development platform) I’ve shown how to get the best from Amazon’s Kindle Highlights feature. This feature and all the amazon highlight personal store features are probably the main reasons for not buying physical books anymore, except for a very few cases.

Anyway, I was using the daily review page occasionally on my laptop. Then I got the chance to start using a tablet to review my notes, and the page suddenly appeared much more interesting and useful.

Useful indeed, but not usable! as the small preview/next buttons are only at the top, and touching the flashcard or the page don’t jump to the next page as it should…

image

Waiting for amazon to fix this… well, good luck! So, I was pleased to see that there were actually a pretty workable solution: just fixing the code inline, injecting an additional click handler directly! Amazon surely is using jquery right? Right indeed, they are!

How do you inject custom javascript behavior on a site? Well, easily through the address bar, just prefix your code with javascript: and you’re done! or off course you can try to spot a cross site scripting  flaw on amazon site… but  I doubt it would be an easy task. :)

So, go to the daily review page and just paste the following code in the address bar:

javascript:$(“.reviewContentWrapper”).click(function(e) {$(“.nextReviewArrow”).click();});

Nothing happens right? Well, just click the flashcard, your next flashcard should load automatically! Much better now! My fingers can finally rest from that “humongous” effort. :)

image

Sure it works on iPad, just add a bookmark with the link. Every time you open /reopen the daily review page just select the bookmark. You won’t go anywhere, but the page will become far easier to use. As long you don’t close the page it will keep working, even when you choose to review another book.

image

Not a permanent fix but a very usable one from my recent experience! ;)

 

Have fun!

Rui

How to save 3000$ in ebooks :) ? Don’t have a kindle yet? Well…you should get one (…not necessarily buy one :) )

Yes, I bet the title caught your attention right on. :) Well, anyway I thought today was a good time to recap that you can get all sorts of interesting ebooks from amazon, for free. Yes, I would also say that an actual Kindle device is a great tool IMO, but you have also the kindle free apps from amazon: ipad, iphone, cloud reader, pc, android… You just need and amazon account to get started.

btw, Please be advised that I am an enthusiastic amazon fan & customer. I do have a kindle keyboard, And, Yes, I regularly buy ebooks!

I actually find it difficult to have a good reason for not starting your own digital library right now…

Currently there is some kind of market place visibility war going on… authors & publishers need better rankings, visibility, one of the fastest ways to do that is to promo the books regularly… so regularly we monitor (thanks powershell, iftt & alerbox). :)

A few days ago I finally set up some powershell & ie automation to be able to compare all my orders (free and not free ones), with current prices and some other book information.

The results are interesting… (even taking into account that several of these books are clearly over priced… ) Got the results into Excel, and I was pleased to see, that at least, my book addiction didn’t reach my wallet! :)

Another side note, I usually follow only two/Three kindle categories, so this is just a small glimpse of the free content regularly available.

image

Hoping that I didn’t mess anything up (and always remembering that data, like perception, is not reality!), but I double checked several items. Bet it’s at least approximate.

Off course Excel is a great tool for the stats but I couldn’t resist to get this data on a small tool I’m using for quick dataset analysis & visualization (based on ms pivotviewer). Maybe we can get this published through SmartPivot, how knows? :)

So, Right click my scrapped CSV where I collected order prices, current prices, stars, reviews, & book covers…

 

image

And we get a much more expressive & interactive view of my digital library:

image

Slice & dice, or kind of  :)

image

Zoom it!

image

Detail it:

image

Unfortunately time to read is much harder to get these days, so most of this content still waits on my beloved kindle. But I can also say that I’ve read amazing books (at the time free), just a glimpse:

Critical Thinking Strategies for Success (Collection)

Do the Right Thing

Wired to Care: How Companies Prosper When They Create Widespread Empathy

image

 

Final note, crossing my fingers that this won’t be the post that shuts the free content down! :)

By the way, this week pick. Bet it will last to next Monday only. ;)

image

 

 

Now, go grab some reading! :)

 

Rui

Kindle eBooks as a great Learning/Self Development platform

Almost a year now since I got my first Kindle. :) Being a somehow frenetic reader way before ebooks, and although I still like to read “regular” books, I must confess that when it comes to learning&self development, ebooks & kindle are hard to beat!

Why is that? Well… lots of reasons, but for now I’ll focus on the  highlights & notes feature. It’s amazing as it allows you to really extract so much more value from ebooks & reading in general. Curious thing, I had never been an “highlighter”… and always left my books in a pure & “pristine” state.

So for the basics, you highlight & add notes on any kindle reader (I prefer the real Kindle to do this , although I sometimes use the free mobile reader, as I’m always switching between them both).

You can then return to the ebook, on any reader (mobile, cloud,etc) and see all the hightlights & notes you’ve made. It all syncs to your amazon kindle profile, so you also have a great backup.

 

photo (13)

Kindle Mobile Reader – Book Highlights

 

photo (8)

Kindle Cloud Reader – Book Highlights

image

 

You can also review all your notes & highlights directly on the Kindle device, selecting  the My Clippings book (you can see it like a dynamic ebook on your device, collecting all you highlights & notes).

photo3

My Clippings on Kindle Device

There’s also a .txt file available with the content if you connect the kindle to the desktop with usb.

But there’s a minor catch here, the “My Clippings file only saves the highlights you do on that specific kindle device. If you highlight on the android/iphone the notes won’t make it to the file. That can be annoying, specially when you switch kindle devices.

Anyway, there’s much more to that. All your highlights are also available online at your kindle profile page here. Like this:

imageKindle Home Page

From there you can go to any book and recover your highlights&notes, and also share them publicly.

 

image

 

In one page you can see all you highlights & notes. A great starting point for building a quick personal  ebook. ;)

image

 

with a little copy & paste, and minor editing you can get a pretty good ebook with all you preferred content, like this one wich contais all my notes, ever!:

 

 

photo

I even created a table of contents with one chapter for each book I’ve read. For periodic & quick recaps that’s amazing.

 

Amazon also got it right that we often buy books for learning & changing, not just reading, and there’s a specific feature that helps you to review & remember  your books highlights, its called the daily review. It will cycle trough all your finished books, periodically, and show all your highlights, one at a time.

image

The daily review

You can also check other books highlights & posts… (Ferriss “on fire”, as always! :) )

 

image

image

And follow some interesting fellows :)

image

 

BTW, don’t miss this week free books. Just create your account and buy them “free” . :) Then, read them on free cloud/mobile readers. You don’t need to have a Kindle!

clip_image001

clip_image001[5]


So, that’s it for now, keep reading, and learning! :)

 

Rui