Cloud Futures 2010: Panel on Cloud Applications - New Experiences and Expectations

7. April 2010

I am in Redmond this week and am participating in two workshops being hosted by different groups within Microsoft Research. Along with a handful of others, I was asked to participate in a panel discussion on Friday dealing with new experiences that cloud computing would facilitate, as well as things we felt were road blocks to seeing those experiences realized. He specifically challenged us to think "outside the box" and to look beyond (the now typical) conversations surrounding raw performance and to dream a little. I wrote out the following as a means of working through my thoughts for my 5-7 minute portion of the panel discussion and, as it took me longer than 7 minutes to read, I thought I'd post it here as a expansion of the talk and possibly an anchor on which to hang subsequent conversations. Please forgive the casual nature of the talk as it is intended to be, essentially, a script read delivered to a group rather than a formal written version of the same.

---

This topic is certainly interesting to me as I am convinced that cloud computing is here to stay and also presents a platform that can be disruptive to the scientific/technical computing industry (although I would qualify this by saying “disruptive in a constructive sense” – meaning that the disruption leads to the additive good and not the removal of existing work). I have spent a considerable amount of time over the past week contemplating this question (how do we imagine cloud computing facilitating new usage scenarios), and have chosen to present my reply by means of a few examples.

The first example is that of Lego MindStorms. Are you familiar with these? They are kits that provide kids (regardless of how old they are :) ) the ability to build robots using a familiar (although slightly altered) Lego metaphor. These kits come with motors, sensors, and a "brain" that is programmable via a drag-and-drop software tool but also supports more complex tools such as Microsoft's Robotics Studio. Do you know what is so great about these (besides the obvious)? They allow common people, with no prior robotics or electronics experience, to dabble in the field. It is, a gateway, if you will, to a much broader field.

The second example is more of an experience that happened to me recently in that I had the privilege of running into my high school science teacher this past weekend - a quiet, rather unassuming fellow named Randy White. Randy's brilliance is that he has a passion for science and did (at least in my case) an excellent job of transference. If I am ever able to accomplish anything interesting in the scientific domain, a large portion of the credit will lie with him. Probably the most important thing he taught us, was how to think about, or to tackle the complex. I can't tell you how many times I heard him say, "Start with what you know". The idea being, that most often, incredibly complex problems were comprised of nothing more than a series of far simpler, and additive problems. He taught us to focus on solving what we could, rather than attempting to "swallow the entire elephant" if you'll allow me to strain a metaphor.

If you find yourself wondering what these two examples have to do with each other, or more germanely, what do they have to do with my vision for the scenarios that cloud computing will open, let me see if I can explain...

You see, much in the same manner as Lego MindStorms have introduced an otherwise unlikely audience to the world of robotics, I believe that cloud computing (based on its cost model and popular programming paradigms) is a means of introducing normal people (and by this, I mean those not formally trained in scientific or technical computing) to the notion of using computation as a tool for solving complex problems. Possibly to the dismay of some in the field, I think that this will, at least initially be done in a means void of the topics of MPI, or Fortran, much in the same way as a 15 year old "programming" his robot doesn't have to understand the inner workings of concurrency runtimes nor the physics at work when his robot "walks" for the first time. I will be the first to admit that these (MPI, Fortran, concurrency topics, race conditions) are important topics, but I would submit that they should not be gating factors to one's ability to explore the arena and determine if he/she is interested in further study in that field. I think we will see paradigms that are far simpler to adopt, such as master-worker, map/reduce, etc. (or even cloud-backed applications that are hidden behind more accessible tools such as Excel, or MatLab) take hold in significant ways and that we will see the development of novel approaches to solving problems using this new platform. The tired-and-true tools will remain, and will be used when necessary and appropriate, but I think if we force them down the throats of the next generation of researchers as "the only way to accomplish science", we are doing them a great
disservice.

As to where Mr. White and high-school science comes into play - well, this can best be summarized by a comment made by a friend of mine, Wally McClure when he, almost flippantly, referred to Windows Azure as a "poor man's supercomputer". Being one that had been working with Azure for quite a bit at the time, I took a little offense at the accolade due to its semi-pejorative nature, and prefer the "common man", but the point is the same regardless: Cloud Computing (at least as currently manifested in both Windows Azure and Amazon's AWS platform), has a great potential to democratize high-performance computing. You see, the high-school I grew up in was small... we had 23 in my graduating class. While Randy has moved on, he still teaches in a comparatively small school that certainly has no funds for a cluster on which to run experiments. However, with the advances in cloud computing, Randy could devise a collection of simple experiments and actually execute them as part of a class project. He could have a significant computational cluster for the equivalent of a few dollars. He can present "Scientific" computing as something obtainable to his students, and hopefully foster an interest that will develop into the next generation of computational thinkers - solving one problem at a time, incrementally, on the way to solving massive problems that we have trouble even describing today.

It is, in my opinion, incumbent upon us - the current generation of computational researchers and domain-specific scientists - to look at cloud computing not as a threat to the establishment, but as facilitating a new means of scientific discovery. We should consider ways to make large-scale computation more accessible to "normal" people. We should be opening up the community, sharing wherever possible, reducing the barriers to entry. Challenge yourselves and your students to push boundaries, to consider non-traditional approaches, and to enjoy "playing" with computational resources.

Conferences, Theory, Cloud Computing

CodeMash: Azure – Lessons from the Field

14. January 2010

I had the privilege of speaking at CodeMash today and had a blast. The attendance was good, and the conversation both before and after the session was great.

As promised, the following is the slide deck from today’s presentation:

 

And here are some links that may be of interest:

Conferences, Cloud Computing , ,

HUNTUG Meeting 090914

14. September 2009

I have the distinct privilege of speaking at the Huntsville (AL) New Technology Users Group meeting this evening on the topic of Windows Azure: Lessons from the field. Beyond the content of the talk I’ll be sharing a handful of links during the talk and wanted to ensure that the slides and links are here and available for those who attended the session.

Slides:

 

Links:

Cloud Computing, Conferences , , , ,

Codestock Session

26. June 2009

nerdSkull_logo_2009 I’d like to thank all of you who attended my session today at CodeStock. I had a great time talking with you all and sharing my experiences with SharePoint and TFS with you all.

Downloads from today’s session:

Also, I promised a collection of links for the tools I had installed.

Conferences, SharePoint , , , , , , ,

Customizations to STSDev 1.3

23. June 2009

As part of my session on Deployment and build using TFS and SharePoint for CodeStock 09 I took the source code from the STSDev project on CodePlex (http://stsdev.codeplex.com) and made a number of modifications. Some of these I would classify as clearly bugs, but most of them are simply adjustments to the core to fit my needs/desires. I’m documenting them here and providing a zip of the source for the benefit of those attending my session. These changes and source code are completely unsupported and you use them at your own risk. That being said, I hope that they are helpful and speed you in your integration between SharePoint and TFS. NOTE: unless specified, all of these changes are to the “Core” project.

  1. First minor change is that I moved the solution file up one level to the parent folder. This is truly nothing but a nit-pick but seems to make source control trees happier and therefore something that I almost always do.
  2. Upgraded the projects/solutions to Visual Studio 2008
  3. Added an app.config file with an assembly binding redirect pushing old references to Microsoft.Build.Framework to utilize version 3.5.0.0. This is one of the changes needed to get support for .NET 3.5 working properly.
  4. Changed the target framework property for the stsdev.csproj to .NET Framework 3.5. This change, in concert with the previous, allows the 3.5 selection in the UI to work properly.
  5. Major Change: Support for alternate bin paths. The 1.3 version as published on CodePlex always uses the compiler output in projectdir\bin\debug when assembling the *.wsp file. This happens regardless of what build configuration you have selected (yes, even release). This doesn’t work for TFS builds since the output is, by default, in a different location on the build server. To support this use case, the following changes were made:
    • Program.cs: Changes were made prior to calling SolutionBuilder.RefreshDeploymentFiles to support the passing in of an additional parameter indicating the alternate bin path.
    • Changed the Create method of Builders\DeploymentFiles\CabDdfBuilder.cs to accept the alternateBinPath as an additional parameter.
    • Changed the Create method of Builders\DeploymentFiles\CabDdfBuilder.cs such that, if an alternate bin path is provided, it will use that value when adding references to assemblies rather than the otherwise-hard-coded /bin/debug/
    • Updated the first overload of the RefreshDeploymentFiles method in SolutionBuilder.cs to pass the alternate bin path to the second overload of RefreshDeploymentFiles.
    • Updated the second overload of the RefreshDeploymentFiles method in SolutionBuilder.cs to build the TargetPath from the alternateBinPath if provided and otherwise to use the default.
    • Updated the second overload of the RefreshDeploymentFiles method in SolutionBuilder.cs to pass the alternateBinPath parameter to CabDdfBuilder.Create().
    • Updated the CompleteSolution method in SolutionBuilder.cs to pass an empty value for alternateBinPath to RefreshDeploymentFiles since, during the initial project creation, it is ok to utilize the default /bin/debug path.
  6. Major Change: When creating a project, always create a parent solution directory in the same way that Visual Studio defaults to. This is helpful when working in a source controlled environment with multiple projects in the same solution. To support this feature, the following changes were made:
    • Changed the <REFRESH /> property in Resources\Common\Microsoft.SharePoint.targets.xml to utilize the $(ProjectDir) variable rather than $(SolutionDir).
    • Changed the Create method of SolutionFiles\SolutionFileBuilder.cs to accept the solution directory as an additional parameter.
    • Changed the Create method of SolutionFiles\SolutionFileBuilder.cs to change the directory for where the solution file is created
    • Changed the Create method of SolutionFiles\SolutionFileBuilder.cs to reference the *.csproj file in subdirectory rather than in the same directory as the *.sln file.
    • Added  a property called ProjectDirectory to SolutionBuilder.cs to hold the full path to the project directory (now distinguished from SolutionDirectory).
    • Updated the RunCreateSolutionWizard method in SolutionBuilder.cs to set the Project Directory property.
    • Updated the InitializeSolution method of SolutionBuilder.cs to create the new (child) project directory and update the location appropriately
    • Updated the InitializeSolution method of SolutionBuilder.cs to set the SolutionBuilder.TargetPath based on the ProjectDirectory property rather than the SolutionDirectory value.
    • Updated the CreateDeploymentFiles method of SolutionBuilder.cs to use the proper path for the files (based on ProjectDirectory).
    • Updated the RefreshDeploymentFiles() (first overload) method of SolutionBuilder.cs to optionally set the ProjectDirectory property if it isn’t already set.
    • Updated the second overload of the RefreshDeploymentFiles method of SolutionBuilder.cs to set the current directory to the value of ProjectDirectory if it is set, and otherwise to use the SolutionDirectory value.
    • Updated the second overload of the RefreshDeploymentFiles method of SolutionBuilder.cs to utilize the ProjectDirectory value rather than the SolutionDirectory path.
    • Updated the LoadSolutionConfigFile method in SolutionBuilder.cs to use the ProjectDirectory property and remove the pathing ambiguity that was leading to incorrect path lookups.
    • Updated the CompleteSolution method in SolutionBuilder.cs to pass the SolutionDirectory property to SolutionFileBuilder.Create() since it can no longer assume that the solution file should be in the same directory as the project file.
  7. Major Change: Added support for the Release|AnyCPU project configuration to support integration with TFS Build. In support of this feature, the following changes were made:
    • Added <REFRESH-TEAMBUILD> property in Resources\Common\Microsoft.SharePoint.targets.xml file. The value of this property is similar to that of <REFRESH /> but has an additional parameter referencing the alternate bin path that TFS creates by default.
    • Added a Release target in Resources\Common\Microsoft.SharePoint.targets.xml. This is targeted specifically at the TFS build and is therefore similar to the ReleaseBuild target but calls the $(REFRESH-TEAMBUILD) exe rather than $(REFRESH)
    • Added a “Release” value to the Configurations string array in SolutionBuilder.cs. This causes the release config to be added to the generated solution and project files.
  8. Major Change: We utilize SharePoint Installer to create a nicely-packaged version of the resulting solution making it nearly brain-dead easy to install a new solution. The only real thing unique or project-specific in a SharePoint Installer package is the setup.exe.config file that passes a number of parameters to the executable allowing it to adapt for your particular solution package. This feature change causes STSDev to generate the initial draft of this file and to add it to the solution items. To this end, the following changes were made:
    • Created a new file, SolutionFiles\SetupConfigFileBuilder.cs to represent the template and logic for the SetupConfigFileBuilder class.
    • Updated the CompleteSolution method in SolutionBuilder.cs to call SetupConfigFileBuilder.Create and create the new config file.
    • Updated SolutionFiles\SolutionFileBuilder.cs to include a pointer to setup.exe.config and thereby include it in the solution.
  9. Major Change: We utilize Sand Castle and Sand Castle Help File Builder to produce developer-targeted API documents for our projects. Sand Castle Help File Builder needs a configuration file (xml) with a number of property values to configure it to run properly. This feature allows the basic file to be generated by STSDev and for it to be added to the solution items collection. To this end, the following changes were made:
    • Created a new file, SolutionFiles\SandcastleHelpFileBuilder.cs to represent the template and logic for the SandcastleHelpFileBuilder class.
    • Updated the CompleteSolution method in SolutionBuilder.cs to call SandcastleHelpFileBuilder.Create and create the configuration file.
    • Updated SolutionFiles\SolutionFileBuilder.cs to include a pointer to the generated *.shfb file and thereby include it in the solution.
  10. Major Change: By default, STSDev would not only generate manifest.xml and the ddf file, but it would add these files to the project. While in many cases this is innocous, it causes heartburn in a source-controlled environment. Firstly, purely generated files have no business being in your source tree. Secondly, since the user doesn’t edit them, once they are checked in, they are most often left as such, resulting in errors on compile because the files are marked as read only, preventing stsdev.exe from updating them properly. To this end, the CreateDeploymentFiles method of SolutionBuilder.cs has been updated to remove the calls to ProjectBuilder.AddSourceFile() following ManifestBuilder.Create() and CabDdfBuilder.Create().
  11. Added an image, PlanetIcon.gif to the resources section and replaced all hard-coded references to africanpith.jpg with references to PlanetIcon.gif. This is nothing more than a branding change for the output projects and has no other affect on the program’s functionality. Affected files include: stsdev.csproj, SolutionProviders\SimpleFeatureSolutionProvider.cs (AddSolutionItems method), Builders\SourceFiles\FeatureBuilder.cs (Create method)
  12. Adjusted the InitilaizeSolutionProviders method of  UserInterface\SelectSolutionType.cs to select by default the Web Part Solution (C# Assembly) project type on load rather than the empty solution. This change is nothing other than a convenience feature during testing and use (that’s almost always the project type I use).
  13. Adjusted the RefreshDeploymentFiles method of SolutionBuilder.cs to fix some seemingly obvious bugs in Console output using incorrect values.

Following the session on Friday, I’ll update this post with the actual source code and any other changes made during the presentation.

Conferences, SharePoint , , , ,

Does the “Cloud” help the Public Sector?

29. October 2008

[Warning… rambling mind walk following]

I’m here at PDC 2008 and having a rather good time attending sessions, talking with experts, speaking with vendors, etc. It has been a pretty good conference and I’ve done a better job than previously at managing my ability to pay attention (i.e. I don’t necessarily *have* to attend a session during every available slot, and, amazingly, there’s a good chance that taking a break b/t will help me pay attention better at the next one…

Anyway, I’ve been thinking about the Windows Azure announcements and having discussions with a couple of my colleagues and also the CIO where I’ve been working for the past couple of years and I’m left wondering what value the “cloud” and the cloud computing platform provides me with… Don’t get me wrong… I completely get it for the consumer market… there is huge value there, and some of my work with some non-profits and consumer-facing organizations might benefit significantly from these services. I can even see benefits in some internal-to-the-same-business corporate scenarios, however I think the uptake is going to be slow simply due to an unwillingness (warranted or not) for companies to “trust” Microsoft (or Google or Amazon for that matter) with their core corporate data.

The problem for me, is that for the past few years I’ve been working at an institution in the US public sector… we’ve been “pushing the envelope” a good bit and are more willing than most to adopt new technologies… but I don’t think that I as an app architect for this organization, nor I as a citizen of this country, necessarily want us posting federal information to “a cloud” controlled by someone other than the same federal entity… and I don’t think that this is necessarily unique to the US federal space… I would have an equally difficult time seeing any other government being willing to take this leap of faith.

However, the principals of the cloud are, in fact, very interesting… should those of us in the public sector be banished to a life of permanent disconnectedness? I think not… What I’m leaving the conference wondering… is how does a federal entity (for any country) provide a “safe cloud” to its users? One in which much of the flexibility/extensibility that is provided in the “public” cloud is still available, however wherein the data and services are completely controlled by the specific government… This, of course, brings up a number of issues… if the given agency completely hosts/controls the “cloud” services… do they cease to be “cloud” services and simply manifest themselves as renamed corporate services? This is certainly a possibility but then I think those of us within the various agencies/sectors need to think of ways to broaden the scope of smaller clouds that we will be building… can the Gov’t of India (US, Canada, or insert country name here) build a single set of “cloud” services that is available only for use by it’s authorized agencies and still see manifested the benefits of scale, abstraction, etc. that are being touted for the “public” cloud? Does a given government build a single cloud? or one for each agency? or some mix in the middle? Will MSFT (or some other vendor) provide tools and platforms that not only allow you to connect to “their” cloud, but also allow you to build your own cloud?

Cloud Computing, Conferences ,

What I'm looking for...

1. November 2007

In a number of the posts I've been writing on the SOA/BPM conference I've referred to the applicability (or lack thereof) of a given approach to "the problem set" that I'm currently working on. I thought it might be good to describe what it is that I'm looking for and a little bit as to why.

I'm working at an organization with roughly 4,000 employees that is in the process of "drinking the MSFT Koolaid". We are deploying nearly every MSFT product and working hard to bring consistency to the platform both from a services aspect as well as the development paradigm. We are focusing on SAP backend, SQL data store, and SharePoint/Office Suite front-end.

We are also focusing on bringing a consistent story to the workflow problem, and, by "workflow" I really mean (well, at least for the purposes of this post) business process. We have business processes in SAP that have workflows behind them and use various means of interaction to keep it moving (i.e. nag-mails, etc.). We have similar workflows in SharePoint for document approval processes and the like. We also have a third set of workflows (business processes) that are either not automated at all, or not in any sort of consistent interface. It is this last set processes that I'm trying to "fix".

I have a couple of "rules":

  1. The interface must live in SharePoint (at least the end-user facing portion)
  2. The "workflow" aspects of the system should utilize one of the two existing execution engines we have running (SAP or WF in MOSS)
  3. The workflow designer should be comfortable for a business analyst to use, and preferably an extension to a tool they are already using (i.e. Visio add-in).
  4. The workflow designer should be able to map actual requirements to the steps in the process (where applicable) and serve as a documentation source of sorts.
  5. The process execution system must provide a webparts (or at least the ability to expose the data as webparts) for the following scenarios:
    1. Overall health of the system
    2. Current user's workflows currently in action
    3. Visual representation of a given workflow process
    4. Analysis relative to the execution stats for each workflow instance and type
      1. i.e. X step of workflow Y is over the planned duration without having completed. Some indication should exist that this process is out of line
      2. i.e. Workflows of type X average Y days with the minimum being Z days and the maximum being T days. (and trends over time)
  6. There must be a hard link between process definition and process execution. The File | Print approach is not sufficient
  7. The system should be standards-based. Meaning, I should be able to import/export the workflow definitions (at least at some level of granularity) to an industry standard such as BPEL or BPMN in order to be able to share or compare that process with other organizations.
  8. The system should be able to provide governance over the models.
  9. In all aspects possible, the system should provide a consistent story WRT the development paradigm we have selected (MSFT .NET, ASP.NET, Windows Workflow, SharePoint, etc.).

Reality...

  1. I'm not exactly certain I'm ready to back down on any of the items above, however I'm coming to the conclusion that there may be a need for a "third" execution engine - meaning, all of the vendors that I saw that had platforms that did what I wanted, either used their own custom execution engine or hosted their own instance of WF separate from MOSS. Even MSFT pushed BizTalk as the "main" process execution engine for *serious* workflows.
  2. If reality point #1 is in fact true, the process execution platform should be based on WF allowing the developers to have a consistent development paradigm.

General Development, Conferences ,

MSFT/BPM/SOA Session 6

1. November 2007

The sixth session I attended was entitled "Next Generation Business Applications" and was presented by Mike Walker of the Architecture Strategy Team (MSFT).

This presentation dove-tailed nicely with the one I had attended earlier that day on the human aspects of SOA/BPM. The real gist of this session was to challenge the user to reconsider the requirement to write distinct user interfaces for each of the applications he/she is tasked to build. In much the same vein as the earlier talk, he was encouraging us to think of ways to expose the user-facing aspects of our applications in the context of applications that the user is already familiar with, such as using the OBA approach and embedding the application in a common productivity tool such as Word, Excel, or PowerPoint. The point being, that the less you force you user to "learn" in order to use your application, the more likely they are to use it. Further, the user doesn't really give a hoot if your application is fully SOA compliant and uses your best-of-breed enterprise service bus, and blah, blah, blah... They care about, does it work. What do I have to do in order to do my job. Tell me that, let me do that, and then leave me alone. and... that's not altogether a bad view of things.

The point is not that OBAs, or "Composite applications" are the only way to go, but rather that it should definitely be on our "radar" and should be considered (at least) prior to beginning to build a full UI.  Another take-away from this session was the fact that the more consistency you can drive in your applications (or, more importantly across your applications), the better off you are. Building a "unique" or "fully custom" user experience is not likely a good thing.

Conferences ,