Guest interview in Power BI Developer Blog and a new Send To Power BI app for Excel

Be sure not to miss the latest blog on the official Power BI Developer Blog :) , the amazing Power BI team asked us to share some thoughts on our latest work & research working with the new Power BI APIs and capabilities. And we were deeply honored to contribute. How cool is that? :)

Read the full post here.

And celebrating the day :) also announcing that even if you can already use SmartCharts to push & sync data to the new Power BI Preview directly from Excel, you now have a dedicated app just to do that :) a brand new Send To Power BI Office App for Excel.

Give it a test drive and leave us your feedback or review. :) It will be greatly appreciated!

Download the new app from the App store.

clip_image002

For more info:

DevScope GitHub repository
Send To Power BI Excel App
SmartCharts Excel App
DevScope Office Apps
Rui Quintino Blog
Rui Romano Blog
www.devscope.net

Awesome day! :)

Rui

How #AzureML and #MachineLearning are helping managing my Inbox–Part I

In this post, how to build an AzureML model to auto suggest folders/labels for email classification & archiving. How to consume AzureML web service directly on Outlook.

note: sample AzureML experiments now published on the brand new AzureML gallery. Email Classifier-Training & Email Classifier – Scoring

Context: for years I’ve opted for an archive by folders strategy to handle my Inbox (not obsessed with inbox zero but  I try to keep it below ~50 inbox mails, and instead use inbox like a buffer – since my first contact with David Allen GTD years ago). true: once in a while I try to test the single  archive folder, but until now…always reverted back to my folders to organize so many parallel projects and threads going on

For this to work, I think one thing  is absolutely mandatory, very fast folder archiving and switching. Although have tried a few tools for this  I ended up coding some Outlook macros over the years to fit this… “peculiar” way of working and unproductive task switching…. 

So to archive a mail I just press alt-4, this window popups up, I type my search terms, enter, and it’s done. mail Archived

image

Or to switch context to a specific project or fast search, just press alt-3, search, enter and I’m there with the latest thread mails immediately available.

image

(that can admit, I obsess over searching…. “as you type” kind of search :) )

Fast forward to “the present”, #MachineLearning #DataIntelligence #AzureML era. I now have a few thousand “labeled” mails (to use machine learning terms) on several active folders. My macros were needing an improvement….  :)

So I exported my archive mails in the form:   |from | to | subject | time offset since fixed day|  format to a csv (tsv in fact)

image

and let AzureML do the heavy lifting of building my very personal email classifier suggestion web service, putting some multiclass classification models & also text handling /feature hashing AzureML features to work in a more useful scenario than classifying flowers :)

 

image

Sticking with the neural network model for now (default params), created and adjusted the scoring experiment & published the web service.

image

image

Used the VBA code sample from the new cool AzureML generated Excel files,

image

few lines of code added on my Outlook macros, based on AzureML Excel VBA code/macros

image

and we get a pretty impressive auto classifier ready to use & help manage our inbox, suggesting the folders where the message belongs when archiving.(note: also triggered when sending messages, archiving both sent and original message if needed.)

So for example,  if I’m disturbing Joana with another annoying mail :) about SmartCharts new features AzureML advises me:

image

On the other hand, if it were an help request to Romano on some stream analytics samples AzureML would opt for:

image

press Enter, for now it’s needed…:) ,  and that’s done. How cool is that? :)

(have to say that the accuracy is not 100% obviously , but pretty damn useful already)

All this running on a free AzureML workspace. Up & running & minutes, from training to online web service

(excluding the time to slightly adapt AzureML VBA code to call the web service & get my mail properly exported)

Training & scoring experiments are available in AzureML gallery fo you to test drive. Off course I loaded a small sample of my mail in those, so you will have to load your own to really see how it works.

…now, to be close to perfection this would benefit from automated data update & model retraining, and that will be the case for use the new AzureML training APIs.

But that will have to wait! :) (true, reduced some time processing my inbox, but not that much…!)

btw- hope to get feedback on this scenario, ex: feature engineering tips and model tuning suggestions to improve the model results

Cheers

Rui

SmartCharts for Excel new features: guest post by Joana Barbosa

Sharing the first guest post on my blog (cool!) and proudly introducing my colleague Joana Barbosa (who I hope soon to have her own tek blog ;) ). She will share some of most recent feature of our tiny (but cool) dataviz App for Office, SmartCharts.

Joana works on DevScope  “multi-faceted” BI Team :)  , among several amazing contributions, she’s the main dev working on SmartCharts coolest features (namely d3/dataviz). (adding to that…. fixing my bugs, sorry Joana! :) )

ps-feedback on features or suggestions? please use the app built-in feedback form . It will reach me or Joana in a moment.

Go Joana!

Hi everyone,

So, recently SmartCharts got some new features for you to play with. And, since the 2014 FIFA Ballon d’Or winner was announced this week, I’m going to demonstrate them using a soccer dataset: Champions League – Best Scorer By Year.

First of all, we’ll load our dataset.

clip_image002

Our table has columns for date, player’s name, country and team as well as goals scored. But we also have three special columns: Images of the player, team and country containing links to pictures.

What can we do with this? We can enhance our charts of course!

One of SmartCharts’ new feature is binding an image column to another dimension column. For that, we need to name it according to the following rule:

– Consider our column ‘Country’. To bind an image column to this dimension, we must call the new one ‘Country_image

Applying this to our dataset, we’ll see the following result:

clip_image004

So, our Bubble Chart has become much nicer, with flag images on it. Also, when you go over a Bubble or over a slice in the Pie Chart, we’ll see our dimension chart over time accompanied by the dimension image we added.

Imagine now that we want to see all players from our country in the set. How could we do it?

Before, we had to alter our dataset in the spreadsheet. Now all we got to do is click on whatever you want to filter by, for instance, the Portugal bubble. This way, by changing the dimension to Player, we’ll only see the Portuguese players. That easy!

clip_image006

Of course, we can go even further by selecting and filtering another dimension (ex. Team: Real Madrid). Thus, we’ll only see the players from Portugal and who were also playing in Real Madrid. In this case, the single result is Cristiano Ronaldo.

Oh, if you want to print one of your charts, or take a screenshot to copy to a document but you don’t want that toolbar messing with your nice chart, just click on the small arrow icon in the upper right corner, and the toolbar will be gone, hidden until you re-click the button.

clip_image008

And that’s it! Our brand new features that I hope you try and enjoy.

The dataset is available for you to check it here:

http://1drv.ms/1CcHLe4

So, want to build some cool fresh interactive charts?

Then, Go Smart and make your own SmartCharts!

Joana

https://twitter.com/JoanaVBarbosa

SmartCharts, SmartPivot & Gantt Chart for #Excel -Latest features & releases @DevScope #Apps

We updated some of our apps and add-ins last week. Brief post introducing some of the new features:

SmartPivot – now with Measure Group/Table Filter in search pane, also Table Report & Filter Tools improvements

A much requested feature now available (should have been in the initial Search release honestly…): SmartSearch pane now allows for measure group/table filtering, much like the regular PivotTable fields browser in Excel.

Download the new SmartPivot release here.

image

Fixed some issues also with Table Report & Filter Tools (both Saved Filters & Filter by list)

image

 

SmartCharts-now with real-time Share feature & Free download during SharePoint Conference 2014

You can now share your charts session, opening it in a browser or tablet/mobile window (disclaimer:some safari/OS issues still to fix). You can then collaborate real time with anyone connected to that SmartCharts session.

Also, you can freely download SmartCharts Task Pane in Office Store during #SPC2014. :) feedback will be appreciated!

Download SmartCharts Task Pane in Office Store

image

 

Gantt Chart & SmartCharts for SharePoint

Soon to be published SmartCharts for SharePoint ;) stay tuned.

image

Gantt Chart for Excel on the other hand is already available in Office Store, and free! :)

Download Gantt Chart for Excel

Gantt Chart for Excel

 

Take care,

 

Rui

To Microsoft Office Apps/Developer Teams: Some requests to improve #Office #Apps Experience–a developer’s view :)

SharePoint conference 2014 is around the corner. The office team & office developer team have been posting some new content on their blog, useful tips for anyone building Office apps for office store. Be sure to check these recent posts for guidance on office apps development:

(also, Office 2013 SP1 was released this week, sp1 new features for sharepoint & office apps api/sdk expected to be announced during the SharePoint conference next week)

Anyway, this post is not for developers, but for Office Apps developer/seller dashboard teams :)

It’s my personal view, after publishing, supporting & improving a few apps (SmartCharts, GanttChart, -a few others coming soon ;) )on how the experience could be improved for end users and developers, hoping to drive a much higher acceptance for Office Apps & the App model in Office365/SharePoint.

So, what have we learned and what would be great to see improved in the Office Apps experience?

1-Apps in the office ribbon are difficult to find (available only through insert/app.. :( )

We frequently get people asking where they can find the app after installing! We need an individual Apps tab, where we could add our favorite apps, our full list of installed apps, quick access to the store & recent apps. And task panes apps  aren’t particularly “insertable” anyway. ~

Sorry, this need an urgent fix! An new Apps ribbon tab!

image

2-No app icons? Except for local catalogs… needs fixing also, why is that? we get the icon in about, but not listed in our apps…

image

Developer/Local Catalogs Apps with Icon

image

Office Apps-no icon :(

(SP1 for Office 2013 seems to bring some minor improvements but far from what I describe above… we get app icons for some apps but you can’t configure the apps that show in the ribbon… currently you will get bing maps and people graph  from msft)

image

Office 2013 SP1 tab for Apps

3-Users can’t get details/screenshots about an app using the Excel built in app browser?

How are they supposed to pay, install & trust an app if they don’t know what its supposed to do?

image

we click try/but/install and the only available info is…

image

come on!? doesn’t make sense when we have rich description and screenshots available in the store.

Also this should be en easy fix by the office team because all we are seeing is a web page from the store. (you can see that page here)

(also note that in this screen there is no way, that I know of ,  of clicking to open the app details in the browser…? you need to click the publisher link and then see all apps…)

4-App analytics in seller dashboard is….well…not great :(

Trust me, the one thing you should do from day one is to configure google analytics, your own custom analytics framework or both (yes, be sure to reflect that in your privacy page).

Office Seller dashboard analytics is not very useful to gain insights on how to tune your app and how it’s being used. Main pain points for analytics in the app seller dashboard (office apps):

  • we get historic data only for the latest 4 weeks :(
  • we only have week totals, not detailed data, not even daily data (no demographics, time, date, browser versions, nothing…)
  • The CSV you can download also has only week total aggregates, not very useful
  • data usually lags 2-3 days behind, and happened recently due to some issues on the office store, even weeks behind current date
  • if you are actually selling apps, there’s no metric whatsoever regarding revenue (you’ll have to wait for the end of month and use very limited tax/payout reports, and even then you’ll have to convert from international currencies to local currency…)
  • paid downloads & free download are bundled together in downloads metric

image

image

Hope all these could be fixed soon, in the world of data, the available dashboards and data are really lacking (being kind here honestly…).

I would gladly skip the built-in Seller Dashboard “dashboards” and exchange all that for the detailed “fact” data on downloads, browser hits. I’ll do my “dashboarding” on that! :)

5-Developer support could be a little better

While developing and getting SmartCharts approved we had several issues with the api, supporting IE9 in Excel web app, strange redirects issues breaking the office API context model, undocumented issues with permissions and data read API calls. I must say that having been in touch with some office apps support teams in this process, we were on your own almost all the time :(. Expect delay for advanced issues, these are the ones difficult to reproduce (ex: excel web app in IE9/chrome).

There a few locations to send feedback (foruns, office sp dev user voice site), but getting through and actually get real help has been a struggle for us. My initial post on addFromPromptAsync /read permission question is still to see an answer, (we just went ahead and requested full read/write permissions instead of read, problem “solved”… but seems a bug/very undocumented to me…)

6-Some bug/issues remain

Some reported issues, that seem to be related to core office app js API  remain,

  • for example excel web app & dates have issues:

clip_image001

  • very annoying issue-some users buy the apps but Excel keeps showing the app as trial, I have seen that happen, there are app reviews from users pointing this issue, and we’ve also received reports from some users with this problem (yet to find a proper workaround… been using  an undocumented one to fix this… ) –
    (note:received two user reports today with this specific issue!…not good…)

That’s it…

Hope this can reach someone on the office teams and help a little bit with some of these issues.  Do you know someone there? can you please pass this along? thanks!  ;)

btw-be sure to check the SharePoint Conference App Awards – People’s Choice! contest. now running into final voting period until February 28, 4:00PM PST. (little disappointed to know that SmartCharts didn’t make it to the final in people awards, wasn’t aware of this voting until very recently :( but hoping we still get a chance on other categories :) )

Take care!

 

Rui

Introducing SmartCharts Data Discovery App for Excel

Finally! :) time to briefly introduce our SmartCharts app for Excel now available at Office Apps Store (download here).

image  image  image image

So, yet another “chart” app/tool for Excel…what’s the point? :) what’s it for?

Well, beyond being a DevScope research project and a place where we will be test driving lots of #dataviz features using the latest technologies, there were some other drivers to build the app:

  • There’s so much data available these days, but still most people can’t even acknowledge that there’s lot of hidden value in data, being it small, medium or big data… so they don’t even start exploring it :(
  • Provide a data discovery tool that keeps user focus on the data, not modeling, not chart designing, just slice, dice & visual data mining
  • There are a lot of patterns for dataviz, but there seem to be a lack of reusable pre-built analytic models for the most simple data intelligence tasks, time series, pattern recognition, drill down views
  • We are building an excessive number of dashboards (really! we are), repeating the same patterns over and over again, same for data mining… can’t we get reusable analytic modules that quickly adapt to different datasets? Data+Services inter-connectable through already available metadata?
  • An Excel app  seemed an awesome way to continue the research work we started with our Data Insights addin few months ago (same core concepts really, we will be back to that eventually ;) )
  • A tool that we can now use to quickly & easily explore small datasets instantly in Excel (under 100k,150k rows) all kind of log tables, log streams, etl logs, usage logs. 
  • (yes, do not expect much value if you’re using it with small tables, or tables without time/date fields) it’s a data discovery tool, not a chart designing tool

All these are of course “experimental hypothesis” that we can now test and research (and already collecting lots of useful feedback :) ).

And stay tuned, we will be releasing some new features this week ;).

Download here & get us some feedback.

http://www.devscope.net/products/SmartCharts

Finishing this post with a few typical usage scenarios…

… these are from real data , loaded the app with different tables I regularly check (clickstream logs, usage logs, etl logs, ssas query logs,…)

(some values masked, sorry!)

Integration load resource usage (~80k rows dataset)

(side note that the first time I loaded this data on the app I immediately found that I was completely unaware on where integration load resources were being spent, and how that patterns were evolving over time)

image

Integration loads vs extracted rows

image

Drill down view of integration time over time

image

Time of day patterns (integration times)

image

Time of day patterns (rows extracted)

image

Now a completely different dataset(120k rows), obtained from an analysis services query trace table log, instant insights: (again note that no modeling or chart designing was necessary…just click, load & explore)

image

 

image

image

 

http://www.devscope.net/products/SmartCharts

Take care!

 

Rui

SmartPivot new build – now with Saved/Quick Pivot filters and Filter by List improvements

Fresh new build for SmartPivot users, download here as usual. This version features:

  •  Improvements on Filter by List feature (interestingly one of the features mostly mentioned by users)
  • And (finally!) a new Saved/Quick filters feature. :)

New Quick/Saved Filters feature

This feature allows saving filter selections on the pivot for later use on other pivots targeting identical hierarchies. The saved/quick filters are persisted locally in user profile configuration, so that you can always refer to them very quickly.

image

Ex: filter your pivot as usual, or through SmartPivot fast search

 

image

 

Now, pick the Quick/Saved Filters option to save your filter:

 

image

 

image

Select your Pivot field and Save Current Filters option:

image

Now just create any other connection to the same cube, and use the quick/saved filters to apply the exact same selection as before.

(as most features in SmartPivot you can fast search saved filters )

image

double click and your previous selection is now filtering the new pivot:

image

Improvements in Filter By List

Filter by list feature now works as a task pane (expect most of SmartPivot dialogs to be migrated to task panes as they allow for better usability).

Improvements in this release:

  • Available as an Excel Task Pane
  • Pick values from range
  • Filter by member names or member keys

image

Other improvements

Fixed an issue with table reports and orphaned calculated measures (undefined measure groups)

Wrapping up

Download SmartPivot through the product page here.

 

Take care,

Rui