A Year without Social Media

Over a year ago, I finally gave up on Twitter, the last social networking platform I had been using. I thought now would be a good time to reflect on giving up on social media.

For the first few weeks, I found myself repeatedly going back to check on various Twitter feeds. I didn’t tweet at all, but was compulsively following stories and comments about the forthcoming Brexit referendum. After the referendum, I then managed to avoid checking Twitter for several months, until the end of 2016, when I was pulled back to Twitter by the unfolding RHI scandal in Northern Ireland (I blame the Stephen Nolan show for igniting my interest in that topic).

Since the start of 2017, I have managed to avoid using Twitter at all, except for very occasional tweets about poor service or delivery complaints to get attention from customer service. It turns out this is the only way to get the likes of Amazon and other companies to actually respond to you, and is now the only reason I’ve keep my Twitter account. I rarely visit the site, I don’t follow anyone on my new twitter account, and I now automatically delete the rare tweets I do make after a few days. In short, Twitter is simply a tool that I use very infrequently.

Of course, it isn’t that easy to give up on social media. After giving up on Twitter, I found myself getting drawn into following Hacker News, the social news website from Y Combinator focusing on technology and start-up culture. Like Twitter with its followers count and likes, it uses gamification techniques (user karma and comment upvoting) to encourage members to participate. However, after spending some time on the site, you realise that there is a real problem with ‘group think’. There is a definite bias towards to US and Silicon Valley points of view, and any alternative views are effectively down voted into oblivion. I managed to wean myself off the comments by focusing on the top links submitted (using alternate sites like Hacker News Top Links, and the excellent Hacker Newsletter from Kale Davis) and avoiding the associated comments by blocking access to the site on my personal laptop and on my work PC. As an alternative to/reminder of the Hacker News comments, I also follow the satirical site, n-gate.com.

In addition, I’ve blocked all use of Safari on my iPhone, and have removed all apps from it that give real time information/updates. In effect, I’ve reduced use of my smart phone to just Gmail and Pocket now.

In short, it isn’t quite as easy to give up social media as it first seems. Going cold turkey can work, but you have to work hard to prevent yourself being drawn back to the same old sites. You also have to be careful about giving up Twitter or Facebook, and replacing them with other sites that consume just as much of your time with the same gamification techniques. As a result, I’m careful to avoid such sites (Hacker News, StackOverflow) where possible. Giving up Twitter has given me more time, but I need to remind myself to do something useful with it, and not to waste it all over again.

SharePoint Development - SharePoint 2016

This post is the fifth in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look the recent SharePoint 2016 release.

More posts in the series:

• Part 0 - SharePoint Development - Past, Present and Future (the series introduction)
• Part 1 - SharePoint Origins (including 2001 and 2003 releases)
• Part 2 - MOSS 2007
• Part 3 - SharePoint 2010
• Part 4 - SharePoint 2013 
• Part 5 - SharePoint 2016 (this post)
• Part 6 - The Future of SharePoint Development

 

SharePoint 2016

There was considerable discussion about the SharePoint 2016 release – notably, whether an on-premise version of SharePoint would even be released. This reflected the constant focus by Microsoft on the Office 365 brand and SharePoint Online over the on-premise offering. This only intensified with the appointment of Satya Nadella as Microsoft CEO in February 2014. He brought a fresh focus on mobile and cloud offerings, and a move away from the traditional Office and OS products.The future of SharePoint on-premise, despite being a product that generated $3 billion for Microsoft in 2015, was seen to be seriously under threat.

In the end, Microsoft did release an on-premise version, after realising that many of their large corporate customers were not prepared to consider migrating their data to a third party cloud service. Microsoft’s “Evolution of SharePoint” blog post in February 2016 confirmed this decision. But even this blog post focused mainly on SharePoint Online/Office 365 features, and how SharePoint 2016 could integrate with them. The hybrid capabilities of SharePoint 2016 were heavily marketed by Microsoft in the lead up to the release, in an effort to entice large organisations to start using cloud based services alongside their existing on-premise installations.

The new on-premise version was an incremental update on the 2013 release, with the major improvements being around the hybrid capabilities already mentioned. The hybrid options required having an Office 365 subscription, in addition to paying for the on-premise licensing. Some of the new features (such as durable links) also required having Office Online Server (the updated and renamed Office Web Apps Server 2013), with Office Online Server now being seen as essential to deploy alongside SharePoint Server 2016.

With this version, the free Foundation edition was dropped, and it was only available in the Standard and Enterprise editions. SharePoint 2016 was released to market on the 14th March 2016, and was finally released to general availability on the 4th May 2016, after missing the planned release date at the end of 2015. As described at the release event, the four areas of innovation in SharePoint 2016 were:

• File sharing on any device
• Intranet/Team sites on any device
• New extensibility options for developers
• Enterprise features around security, compliance and privacy

The new features in SharePoint 2016 included:

• Improved mobile experience
  • Support for touch screens on the web application
  • Mobile apps for Android iOS and Windows devices promised within the next year
• Integration with the new PowerApps and Flow services, again not available upon release, but to be available within the year.
  • PowerApps is a set of developer tools and templates that are simplified for ordinary business users to create business applications. The announcement that they will be integrated with SharePoint means that users will be able to create cross-platform PowerApps that utilize SharePoint lists and libraries as a data source.
  • Microsoft Flow is part of PowerApps and is a clone of the well known IFTTT service, allowing users to mash up data from various services, and set up certain actions to specific defined events. 
• Hybrid features, including:
• SharePoint Insights, a reporting service that combines “usage and compliance data from on-premises and cloud into the Office 365 Reporting Center”. In other words, the analytics logs from your SharePoint on-premise farm are automatically uploaded to Office 365 to give a unified audit view. Again, this feature was not available on the release of SharePoint 2016, but was to be available for the end of 2016.
• The new Hybrid Search allows users to search across both on-premise and online SharePoint sites for relevant content with a single combined list of search results (and not separate results for on-premise and online sites as before). This uses a new Cloud search service application for SharePoint 2013 and 2016 that generated a unified cloud-based search index.
• A Single Sites View for users to view any on-premise and SharePoint Online sites they have access to.
• A new enhanced UI, so that the look and feel is consistent across both online and on-premise sites for a consistent user experience, particularly for hybrid scenarios. This included the introduction of the App Launcher to SharePoint 2016, to allow users to access Office 365 applications such as Exchange Online, Delve and OneDrive from on-premise.
• OneDrive Redirection – use of the user’s Office 365 My Site instead of hosting it on-premise. OneDrive for Business aims to bring users to one place to help them work with their files, regardless of where they are stored. Note, this feature was available in SharePoint 2013 with SP1.
• Integration with Office 365 Delve and the Office Graph (see Office 365 section below) with on-premise, to surface relevant content and activity for individual users
• Users can have a single profile in Office 365 where all their profile information is stored. This profile can then be used for both SharePoint on-premise and SharePoint Online.
• Compliance features were significantly improved, a major selling point for Enterprise customers.
• Data Loss Prevention (DLP) features, similar to those found in Microsoft Exchange Server 2014 and Exchange Online, were introduced to both SharePoint Online and SharePoint 2016.
• The Compliance Center uses the DLP features to allow users to identify and search for sensitive content in both SharePoint Server 2016 and OneDrive documents. 51 built-in sensitive information types, such as credit card numbers, passport numbers and bank account numbers, are defined and used to identify sensitive data in documents. Once found, following a crawl by the search service application, sensitive documents can be reviewed in the eDiscovery Center, and the sharing permissions can be changed, the data removed from shared sites, or the documents can be exported for further review
• The In-Place Hold Policy Center allows users manage policies defined across site collections and OneDrive for Business, and to establish time-based holds on data for a specified time.
• Miscellaneous
• Improved sharing options, with a share button on every page, and the option to view who already has access to an item.
• Support for Open Document Format (ODF)
• The file names can now be longer than 128 characters, and character restrictions are being removed, so that the & ~ { } characters, GUID, leading dots are now allowed in file names.
• About Me profile page to include content from Delve and Office Graph API
• Image and video previews are generated on hovering the mouse over a a file in the document libraries.
• Durable links - make it easier to move files without having to update links.

While this is not a comprehensive list of all the end user features in SharePoint 2016, it confirms that the release was incremental and didn’t offer many significant improvements for end users. This wasn’t particularly surprising, given the maturity of the SharePoint platform.

Developing for SharePoint 2016

This was the first release of SharePoint to be based on the SharePoint Online codebase, with some capabilities backported for the on-premise version (such as PerformancePoint). This meant that with a single unified code base for both the SharePoint Online and on-premise versions, there was a now a single roadmap for the product. Features would come first to the cloud, and when appropriate, would be added to the on-premise version.

Microsoft made much of the fact that new features would be constantly be added to Office 365, and that these will be made to on-premise farms available more quickly using the new Feature Packs (for those customers with Microsoft's Software Assurance volume-licensing plan), rather than the traditional Service Packs. Clearly, most of the future effort and innovation would be directed at Office 365, with relatively few new features coming specifically for on-premises.

The move to a single codebase based on SharePoint Online meant a number of architectural changes:

• As already mentioned, there would be no Foundation version of SharePoint 2016. This was understandable, given the minimal capabilities and very low take-up of Foundation in previous releases.
• As noted above, several new features required having Office Online Server (the renamed Office Web Apps Server) installed. Office Online Server is now seen as being essentially a dependency for SharePoint Server 2016.
• Excel Services has been removed from SharePoint 2016, and is instead included as part of the Office Online Server.
• Microsoft Project Server 2016 (PWA)  is now integrated to SharePoint 2016, instead of requiring a separate install as previously. As a result, MS Project Server content is now consolidated into the SharePoint Content databases and not in a separate database.
• Forefront Identity Manager is no longer included for bi-directional synchronization of user profile information with Active Directory. Instead a uni-directional sync using AD Import is available. Microsoft Identity Manager 2016 can be configured separately to enable bi-direction sync if required.This is part of a move away from Windows based identity management to cloud based identity management (such as Azure Active Directory, which was now trusted by default).

This version of SharePoint was primarily an infrastructure release, with relatively few improvements for end users. Instead, Microsoft was focusing on improving how SharePoint is installed, configured and maintained, and in particular, offering more options for integrating on-premise SharePoint farms with Office 365 services. The changes included:

• Installation & Topology
• The initial configuration of Farm and Server Roles was simplified in the installer wizard, due to the new MinRole server configuration.
• A new MinRole topology. Based on the experiences gained by Microsoft in running SharePoint Online, this allows administrators to optimize the configuration of each server in a Share Point farm based on the role it has been assigned. The different server roles supported are Front-End, Application, Distributed Cache, Search, and the new Specialized Load role. The Specialized Load role is used for services that are to be isolated from the default SharePoint services (such as 3rd party services or PerformancePoint), and servers with this role are excluded from compliance and reporting. The SharePoint health analyser is used to enforce the min-role topology by scanning all servers except those used for the specialized load role.
• SharePoint 2016 can no longer be installed in standalone mode. Farm Administrators would have to choose one of the 6 Mini Roles available during the configuration phase. There is a Single Server Farm option where everything is installed on the same computer for development purposes.
• As there is no ‘standalone’ installation option to install SQL Express, SQL server must already be installed and running prior to installing SharePoint.
• Upgrade Options
• You could only upgrade to SharePoint 2016 from SharePoint 2013,  using the database-attach upgrade method. There were no supported direct upgrade paths from SharePoint 2010 or earlier.
• Improved performance and reliability
• The Zero Downtime Patching strategy will make use of simplified patch management to significantly reduce the size and number of update packages. Rather than deploying large cumulative updates that contain all patches up to that point in time, as currently, the new patching strategy will mean patches are distributed through smaller packages (~100 MB). As the new patches contain smaller, more targeted changes, SharePoint administrators can now choose which patches to install and skip, so that patching with zero downtime is possible.
• The introduction of Fast Site Collection Creation to speed up the creation of a new Site Collection. This uses a master site collection template at the Content Database level to provision a new site, and avoids the overhead of Feature activation. Because of this, the time to create a site collection fell from 40 seconds to just 2 seconds.
• Reliability improvements to the Distributed Cache. This still relies on AppFabric 1.1, which, despite being deprecated as a Windows feature, will continue to be supported for the SharePoint 2013 and 2016’s lifecycles.
• Resilient file system (ReFS) storage protects data from common errors that normally cause data loss. When ReFS is used in conjunction with a mirror space or a parity space, detected corruption can be automatically repaired using the alternate copy provided by Storage Spaces. ReFS also prioritizes data availability so that if corruption occurs, the repair process is both localized to the area of corruption and performed online, requiring no volume downtime. ReFS includes the Salvage feature that removes the corrupt data from the namespace on a live volume and prevents good data from being affected by non-repairable corrupt data.
• Scalability
• Content databases will scale into Terabytes, up from a maximum of 200 GB per content database in SharePoint 2013.
• A maximum of 100,000 Site Collections per database (the limit was previously 20,000)
• The list view threshold will be greater than the existing 5,000 items limit. In reality, the threshold still exists, but SharePoint 2016 will automatically create Indexed Columns on lists with more than 5000 items, so that while the threshold is still in place, it no longer impacts end users.
• Maximum file upload size has increased from 2GB to 10GB, allowing for large media (video) files to be stored in SharePoint 2016. The file BLOBs will still be stored in the content database and will make use of the Shredded Storage implemented in SharePoint 2013.
• Security
• Encrypted connections (TLS 1.2) supported by default 
• Supports sending email to SMTP servers that use STARTTLS connection encryption, and using non-default SMTP ports
• Miscellaneous
• SharePoint Logging API (SLAPI) allows easier ability to record and report on analytics and telemetry across a whole range of objects in the Farm
• Some 115 new PowerShell cmdlets have been added to help support SharePoint, and STSADM is finally officially deprecated

For developers, the main changes were:

• The announcement of a new development model, the SharePoint Framework (SPFX, see details below).
• A new simplified Page model, to give a improved end user experience. The new page model was required for the new SharePoint development model, and gives rise to a new page canvas that gives a simplified editing experience for end users (no more clunky ribbon), and made it easier to add and remove web parts (no web part zones as in classic pages). However, classic web parts would not work in the new page model, so new web parts would be provided to replace the existing OOTB SharePoint web parts. The new page model gives rise to new client application types:
• Page-based apps – an alternative to provider-hosted apps with server side code, these are implemented in JavaScript, but have benefits such as having full page context.
• List-based apps – an alternative to JS Link for transforming the display/edit/new experience around list items
• The new modern page model is used in the updated team and publishing sites, which can mean that when migrating content from existing SP2013 sites, you can end up with mixed content (classic and modern pages) that have significantly different web parts and editing experiences available.
• A new version of SharePoint Designer would not be released with SharePoint 2016. The existing version SharePoint Designer 2013 would continue to be supported in SharePoint 2016.
• Similarly, a new version of InfoPath would not be released. The previous version InfoPath 2013 will continue to be supported in SharePoint 2016. The InfoPath Form Services will continue to be supported to 2023 but there will be no further enhancement to InfoPath. Despite some talk of next-generation forms for SharePoint, Microsoft currently has no replacement story for InfoPath. Instead, PowerApps is being touted as a possible replacement.
• Search Improvements
• SharePoint 2016 also offers the option to crawl current and legacy SharePoint versions such as 2007, 2010 and 2013, without needing to upgrade those versions.
• Supports indexing of up to 500 million items per Search Server application

SharePoint Framework

For developers, the main talking point with the release of SharePoint 2016 was the introduction of (yet another) development model, the SharePoint Framework (SPFx). This was a SharePoint client-side framework that would allow developers to use the latest web development technologies and tooling to create pages and web parts for both SharePoint Online and SharePoint on-premise.  As for many of the new features announced for SharePoint 2016, it wasn’t actually ready for release in May 2016. Instead, a preview was available in August 2016, and the framework was released to general availability in February 2017. This was for SharePoint Online only; it is expected to be released for SharePoint 2016 on-premise via Feature Pack 2 in the second half of 2017.

Key features of the SharePoint Framework include:

• Runs in the context of the current user and connection in the browser. The new framework does not use iFrames, unlike the previous App Model.
• The controls are rendered in the normal page DOM.
• The controls are responsive and accessible by nature.
• It's framework agnostic - React, Handlebars, Knockout, Angular, and other JavaScript frameworks can be used..
• The toolchain is based on common open source client development tools like npm, TypeScript (required), Yeoman, webpack, and gulp.
• Solutions can be deployed in both classic web part and publishing pages and modern pages.

SPFx is effectively a JavaScript based client-side UX framework for SharePoint. While it is possible to use any JavaScript framework with SPFx, the clear preference from Microsoft is for developers to use TypeScript (which makes sense for SharePoint developers coming from the legacy server-side APIs), and the React framework. The new framework would use RESTful API’s to communicate with the SharePoint (Online or on-premises) backend.

This was a major change to SharePoint development which historically was based around full trust C# code running server-side. With the SharePoint Framework, Microsoft was saying that the future of SharePoint development was using client-side JavaScript. As the new framework was client based, the files for any web parts of apps that use SPFx can live anywhere (on a CDN or a separate website), not just in SharePoint. Developers were no longer tied to using Visual Studio (or even the Windows OS) for development, but could instead use lightweight code editors like VS Code or Sublime.

I haven’t been able to play with the new framework, as it is currently only available for Office 365. But I have been following a useful blog series that is asking prominent SharePoint developers to describe their experiences of SPFx in their own words. Some common themes emerge:

• The framework is a good start, but it has a long way to go until it gains parity with full-trust server side code or the App Model. Currently, the framework really only offers web parts; developers are looking to use the framework to:
• Create SPAs built on top of SharePoint.
• Provide more granular customizations (giving alternatives to custom actions and display templates)
• Provision lists and sites (developers are still reliant on PnP Guidance, which is to use Add-ins)
• Automate deployment of SPFx solutions
• Give alternatives to JS Link and script injection
• Ability to target specific site collections, not just the tenant
• The positives about the framework include:
• You no longer need to develop on a SharePoint server. The “local development” model is very different – IIS no longer hosts files on your local machine, since Gulp and node.js are used to serve files instead.
• The use of unit tests (automatically generated when creating a new SPFx solution) will lead to higher quality applications.
• There were mixed views about the use of TypeScript, but most agreed that this would help legacy SharePoint developers make the jump to the new framework.
• The framework, combined with the new page model, helps to simplify end user experience, and gives rise to improved rendering on mobile devices
• Common complaints include:
• Lack of documentation (to be fair, I personally think the documentation is quite good compared to previous releases, but there is a definite lag on documenting the new features, simply due to the large number being released).
• The steep learning curve for Enterprise developers who are unfamiliar with the modern web development stack.
• Lack of support for the framework in Visual Studio (currently there is only a community extension that allows developers to create SPFx projects in Visual Studio).
• SPFx doesn’t provide guidance on how to bundle dependencies, or how to deal with an updated version of solutions.
• The Framework is not as agnostic as it purports to be: all the existing first party web parts and documentation use TypeScript and the React framework, making it difficult for developers who prefer vanilla JavaScript and Angular. This issue was mentioned by almost every developer in the blog series.
• As SPFx is a client-side framework, there is no security model to work against, as in the server-side API. The lack of a security model means that  Provider-hosted add-ins (using iFrames) are a still a valid developer option as they are isolated on the page. Alternatively, a server side component (eg Web API) may be required.

SharePoint Online/Office 365

As noted in my blog post on SharePoint 2013, Microsoft were quickly developing new features for SharePoint Online/Office 365. The pace of change had only accelerated since 2013, and new features and services being delivered almost weekly. Recent features include:

• Microsoft Graph is an underlying Office 365 service maps the relationships among people and information (such as email, Yammer, meetings, recently accessed documents), and allows more relevant and personalized information to be presented to each individual user. The Office Graph uses machine learning techniques to connect people to the relevant content, conversations and people around them. It is based on the Yammer enterprise Graph. In another win for the folks at Microsoft Marketing, Microsoft Graph was known previously as the Office Graph (renamed in November 2015), and before that, known as the Office 365 Unified API (see below).
• Delve is a data visualization and discovery tool that surfaces interesting content to users. It is based on the Office Graph, and in effect, tries to show documents and content to the user that it thinks they will need, before they know they need themselves.
• Sway for Business, an interactive multimedia tool that allows users to quickly create and share interactive reports, personal stories and presentations.
• Modern team sites with a responsive UI.
• Modern SharePoint Lists, with support for Microsoft Flow and PowerApps (discussed above)
• Office 365 Groups - a way to centralize membership for multiple Microsoft products (such as SharePoint Online, Exchange, Teams, Yammer) in one place, and apply policies at the project or team level instead of applying to each separate product.
• Improved integration between SharePoint Online and OneDrive, with users being able to access SharePoint Online document libraries using the OneDrive mobile app.
• Office 365 Public CDN, allowing a library in SharePoint Online to be become a repository for assets to be stored in a CDN.

One of the major changes in Office 365 for developers was the launch of the Unified APIs, subsequently renamed as the Microsoft (Office) Graph. Due to the rapidly increasing number of services in the Office 365 brand, Microsoft recognized that unified API was required to bring together all the various APIs that access the Office platform. This Office 365 Unified API subsequently evolved into the Microsoft Graph, giving developers a single endpoint to access data from Microsoft cloud services.

Conclusions

SharePoint 2016 was a pivotal release, not so much in the new features it offered, but in that it effectively set SharePoint Online as the modern SharePoint platform. This new modern SharePoint is built on top of Azure and uses modern web development technologies, in direct contrast to the legacy SharePoint on-premises platform it evolved from. Microsoft's effort and innovation will clearly be directed at developing SharePoint Online and Office 365, with few new features coming specifically for the legacy on-premise platform.

For the end user, the main features were the improved mobile experience (and improved UI overall), and the increasing number of Office 365 services that they could use from SharePoint on-premise. Apart from this, the 2016 release offered end users very little. For administrators, the improved hybrid support in SharePoint 2016 made integrating with Office 365 significantly easier, and there were significant improvements in installing and maintaining on-premise farms (such as the MiniRole topology, Zero Downtime Patching and ReFS). For developers, this was a significant release, with the new codebase being based on SharePoint Online, and yet another development framework, and a simplified Page and publishing model.

I have mixed feelings about the new SharePoint Framework (SPFx). Obviously, as the fourth development model to be introduced, why should I or others bother investing in this one? Microsoft is selling it as the one and true way to develop solutions for SharePoint, but they pushed Sandbox Solutions and the App Model in the same way, and they’re now as dead as the dodo. Why should we assume that there won’t be yet another development model in the next SharePoint release?

One reason is that Microsoft has actually released the new model correctly. They had learned from their many mistakes in releasing the previous development models. Instead of just laying down the law to developers that this was the development model to use, the process has been more open. Microsoft had been discussing the framework with developers and vendors for over a year before SharePoint 2016 was released, in a series of Dev Kitchen events organised by the SharePoint Product Group. Previews of the framework were available for almost a year before it was released in February to general availability. The use of the SharePoint Framework Roadmap has been rightly praised – the goals for SPFx for the next 12-18 months are openly discussed and prioritised using community feedback, and new framework features and functionality are rapidly being released.

Another reason is that Microsoft has embraced the modern web development stack. The new framework attempts to standardise development for SharePoint around web development best practices and tooling, and to move to client side development. SPFx gives us a clean break from legacy SharePoint, and the increasingly anachronistic server-side development model that we have all been using for too long. While this means a huge learning curve for existing SharePoint developers, it does represent a chance to encourage other (non-Microsoft) developers to build solutions using SharePoint/Office 365.

Finally, the framework is additive: it doesn’t replace anything or make the legacy development models obsolete. Organisations can continue to build solutions using the existing development models, but as the framework matures, customers will begin to use it to build new solutions and enhance existing solutions.

I suspect adoption of the new framework will be very slow. It is still not available for on-premise SharePoint 2016, and the majority of existing on-premise customers are still on SharePoint 2013 or even older versions. The use of the web development stack will be a barrier to existing SharePoint developers making the jump. I suspect it will take at least another 2-3 years before we see any significant uptake of the framework, as organisations start to migrate to 2016, and as developers gain confidence that Microsoft is not going to pull the rug from under their feet (yet again).

Overall, I’m actually positive about the SharePoint framework. It represents a break from the legacy SharePoint development practices that have built up, and allow us to embrace web development techniques that can be used outside of the SharePoint ghetto. Obviously Microsoft is hoping it will encourage greater use of SharePoint from non-Microsoft developers. Regardless of whether the SharePoint Framework survives or not, the future of SharePoint development is clearly client-side. I’m looking forward to developing solutions with it soon!

To be clear, with the advent of the new SharePoint Framework, the App/Add-in model is dead. It will kept around for a while until SPFx gains parity, but it will then be deprecated to death, just as Sandbox Solutions have. I doubt if full trust server side solutions will ever stop being an option for on-premise customization, simply because they offer a way to provide a custom backend API for SPFx solutions. Additionally, I suspect that if you’re serious about developing for SharePoint (as opposed to just consuming a SharePoint / Office 365 API), you will still need to understand the underlying object model.

While the SharePoint Framework was a positive feature of the SharePoint 2016 release, there were also a few negatives. For developers, the lack of investment in new tools to replace SharePoint Designer and InfoPath was disappointing. After initial hinting at a replacement for InfoPath (Forms on SharePoint Lists), this was cancelled, and PowerApps was parachuted in to (partially) replace InfoPath. As there is no full replacement for InfoPath, the InfoPath 2013 client and InfoPath Forms Services will continue to be supported until 2023.

The much heralded new iOS app for SharePoint, released in June 2016, was also underwhelming. As someone who had to use and support it, I’m aware of issues where users belonging to a large number of site collections were unable to access their sites via the app. As can be seen from the iTunes rating graph below, until the release of v2.9.3 last week, the ratings for the app were around 3.2-3.4, which is relatively low. Hopefully, the new version of the app will have resolved many of the issues that Microsoft appears to have ignored for the past year.

Another major failing in the release of SharePoint 2016 was the failure to address what we should be using Yammer for. After all the hype for Yammer in the SharePoint 2013 release, it wasn’t mentioned at all at the release of SharePoint 2016. What is the use case for Yammer? Where does it stand amongst the other conversation tools like Skype, Office 365 Groups and traditional SharePoint discussions?

The major takeaway from this release was that Microsoft wanted developers to stop viewing SharePoint as a platform, and to start viewing it as a service. This conclusively resolved Microsoft’s SharePoint identity crisis. With the new SharePoint Framework and the new modern page model, Microsoft were effectively ignoring the legacy SharePoint development models and were rebooting using the latest web development technologies.

Microsoft now appears to see SharePoint on-premise as a way of promoting hybrid solutions. The improved hybrid support in SharePoint 2016 aims to push the current on-premise customers (typically larger organisations reluctant to store their data in a third party cloud-based solution) into using the hybrid model and increasing their exposure to the rapidly growing range of Office 365 services. As part of the move away from being a platform, SharePoint (Online and on-premise) is now being built more for an out of the box experience for end users, as opposed to being a platform for developers to build customized solutions.

In attempt to quell any of the uncertainty around the SharePoint 2016 release, Microsoft has already told us that it will not be the last on-premise release. As well as summarizing this series so far, I’ll look ahead to the next release of SharePoint and how we will develop for it in the next (and last) article.

Belfast Street Poster Art

If you talk about street art in Belfast, most people will think of the paramilitary murals that mark out the sectarian landscape of the city. Or possibly they know of the number of talented street artists that are reclaiming walls in Belfast for more contemporary and non-sectarian street art, like emic or #VISUALWASTE. Check out this virtual tour of murals if you’re interested in finding out more.

But there is also a strong tradition of street poster art, particularly around the University/Botanic Avenue area. I seen these posters recently, and thought they were worth sharing.

Nothing to hide, nothing to fear…

In Billy We Trust…

Daily Mail – Super Bigot Edition

If you come across any others, please share in the comments. Thanks!

SharePoint Development - SharePoint 2013

This post is the fourth in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look at the SharePoint 2013 release.

More posts in the series:

• Part 0 - SharePoint Development - Past, Present and Future (the series introduction)
• Part 1 - SharePoint Origins (including 2001 and 2003 releases)
• Part 2 - MOSS 2007
• Part 3 - SharePoint 2010
• Part 4 - SharePoint 2013 (this post)
• Part 5 - SharePoint 2016
• Part 6 - The Future of SharePoint Development

 

SharePoint 2013

The SharePoint 2013 release was considerably less ambitious than the 2010 release. It was relatively minor upgrade from an IT perspective, with no major architectural changes. This wasn’t a major surprise. SharePoint was by this stage a mature and robust product that had been around for over 12 years, and was, by 2013, used by some 100 million users and 80% of Fortune 500 companies. The 2013 release was simply an incremental change on the existing software that focused on improving usability for end users, particularly around the social experience.

The different versions available for SharePoint 2103 remained unchanged from SharePoint 2010 (Foundation, Standard and Enterprise). The first technical preview of SharePoint 2013 was released in early 2012, and the public beta was released in July 2012. The public release of the finalized version was on 11th October, 2012.

The main features in this release were:

• A major redesign of the user interface, including:
  • Improved cross browser support
  • Drag and drop documents from your desktop into SharePoint libraries, and also to move content within and between SharePoint libraries (also cross browser)
  • Ability to create and edit lists without leaving the page you’re currently editing
• Sync document libraries to your computer using OneDrive for Business (previously known as SkyDrive Pro, it was simply an updated version of the Groove tool).
• Additionally, the MySites feature was rebranded as ‘OneDrive for Business’. In fact, the existing MySite feature was kept, and each MySite now came with a Document Library specially built for the OneDrive for Business tool. As end users unsurprisingly associated it with Microsoft’s OneDrive cloud storage solution, this caused endless confusion. Chalk up yet another win for Microsoft Marketing.
• Improved task management features, such as:
  • Updated Project Site template, and an improved Tasks list that now included a timeline and Gantt views.
  • Ability to assign multiple resources to tasks, and to create subtasks
  • Improved integration with Microsoft Project
  • Option to view all tasks (across the entire SharePoint farm)  assigned to you in your personal site, without the need for custom or third party solutions.
  • Tasks could also be synchronized with Outlook, and could also be pulled in from other providers (such as a CRM or TFS) using the extendable framework provided.
• The search features in SharePoint 2013 were considerably improved, with an eye to the search features used in Google that users were already familiar with. These new features included:
  • Type ahead (auto-complete) when entering search terms
  • Previous search terms are now displayed as query suggestions at the top of the search results page
  • Use of Boolean operators in search queries
  • Wildcard searches
  • Quick preview of documents within the search results, allowing the user to preview the document without leaving the results page.
  • Additionally, users could now see how many times an item has been viewed from within the search results
  • A new one click option to view search item in their parent library 
  • Users could follow, and be updated about changes made to, a document from the search results page
  • The new continuous crawl meant that there was no lag period following the addition of new documents to the documents appearing in the search results
  • Improved discussion list, with a more ‘Facebook’-like activity feed.
• The most important search related improvement was the ability for a piece of content to be surfaced in different locations, across site collection boundaries. This included a number of new category of web parts called ‘Search-driven content’, where the web parts were all pre-configured variants of the Content by Search Web Part (e.g. “Items Matching a Tag”). Similarly, any list/library in SharePoint could be nominated as a ‘catalog’ and then shared across site collections as part of the ‘Cross-Site Collection Publishing’ Feature.
• Improved social features. The social networking features in SharePoint had been poor, and organisations were forced to use third party add-ons, like Social Sites from Newsgator. The 2013 release had a number of improved social features:
  • The introduction of micro-blogging with (searchable) #hashtags and @mentions
  • Ability to follow and be updated about changes to people, sites and documents, as opposed to the older style alerts
  • New Community site template focused on conversations between users. This was very different to the existing team sites - users could observe conversations and then “join” a community when they want to post. The user’s photo was shown beside all posts they had made, and there was gamification (ratings, badges) to encourage participation
  • An interactive newsfeed to aggregate and follow a user’s activities across all their sites, and to view the activities of users and content that they follow
  • One click sharing to easily share documents and sites with others
• Mobile
  • In SharePoint 2010, it was possible to create a mobile views for sites, but it was very constrained, and didn’t allow for any site customization for specific mobile devices or tablets. With the SharePoint 2013 release, the mobile browser experience was optimized, and allowed the creation of device channels to render SharePoint content in a design for a specific device type
  • SharePoint Newsfeed apps were released for Windows Phone and iOS devices, and later for Windows 8
  • SkyDrive Pro (later re-branded as OneDrive for Business) applications were also available for Windows Phone, and later for Windows 8 and iOS, allowing for offline viewing of SharePoint lists and document libraries
• Business Intelligence
  • The BI options in SharePoint 2013 enhanced the existing features, such as Excel integration and the PerformancePoint service (including a new Dashboard designer)
  • Improved support for rendering dashboards on iPads and Windows 8 tablets
  • A new Business Intelligence Center site template to manage reports, scorecards, dashboards, and data sources in a single location
• Miscellaneous
  • OneNote integration with team sites (when configured with Office 365)
  • Improved integration with Microsoft Outlook
  • Improved support for playing video (HTML5 and Silverlight players) and searching for videos
  • New eDiscovery (electronic discovery and audit) site template
  • Use of Metro styling for the UI

As always, this is an incomplete list, but it covers the major improvements in user features for SharePoint 2013.

Developing for SharePoint 2013

As already noted above, there were no major architectural changes in SharePoint 2013. Despite this, the upgrade process from SharePoint 2010 to SharePoint 2013 was still problematic, due to deprecated features and new or modified features (such as the updated discussion forums) which could cause existing SP2010 solutions to stop working following the upgrade. Additionally, for this release, there was only one supported path, from 2010 to 2013; you could no longer skip a release and go from 2007 to 2013.

The visual and in-place upgrade options were discontinued in this release, with only the database attach upgrade method being now supported. However, there was now an option to retain a content database as a 2010 content database and to perform a “deferred site collection upgrade”. In practice, this means that the 2010 site collection continues to be a 2010 site collection, just running on a 2013 farm. The solutions for the 2010 site collection continued to be deployed to the 14 hive, while solutions for 2013 site collections were deployed to the new 15 hive. In theory, the existing 2010 sites would continue to work when migrated to the SP2013 farm, including your customizations. When you performed the deferred site collection upgrade, the site collection now uses the 15 hive, which is when the custom solutions might break. As someone who has recently upgraded a major SharePoint farm to 2013, this was a nice feature that allowed the migration to proceed in phases as the existing 2010 customizations were made 2013 compatible.

At the time, Microsoft stated that SharePoint 2013 would be the last version released on a 3 year upgrade cycle. In future, both the on-premise and online versions will be updated much more frequently, with SharePoint Online to be upgraded every 90-120 days. Future releases of the on-premise release would simply be snapshots of current SharePoint Online.

The main changes to the administration of SharePoint were:

• Improvements in the ability to manage multi-tenancy deployments (again, based on Microsoft’s experience of running the SharePoint Online service)
• Significant improvements in the User Profile Synchronization Service with the re-introduction of the AD Import (faster than the User Profile Sync), faster performance for full synchronizations, and the ability to sync with more directory services (such as the Java System Directory Server, Novell eDirectory and IBM Tivoli). The User Profile Sync was still difficult to setup, but it was considerably improved on the 2010 release.
• Introduction of AppFabric to give a distributed, in-memory object cache to scale out high-performance .NET applications. AppFabric was installed as a prerequisite for SharePoint 2013 and could only be configured via PowerShell. While an important new feature, it was difficult to configure SharePoint’s Distributed Cache Service (a wrapper around AppFabric) to work across a high availability SharePoint farm with redundant servers.
• Request Management to control how the SharePoint farm handles incoming requests by evaluating logic rules against them in order to determine which action to take, and which machine(s) in the farm (if any) should handle the request. This wasn’t to replace load balancers, but to provide more flexible configuration options for the SharePoint farm.
• Internet Explorer 7 was no longer supported
• Various improvements in the Web Content Management (WCM) features, including:
  • Ability to copy and paste directly content from Word, without having to strip out the formatting
  • Improved support to upload and embed videos in pages, including a new video content type, and the auto-generation of thumbnail preview images
  • Ability to insert iframe elements into a page, allowing external content to be embedded
  • Improved support for multilingual sites
  • The introduction of cross-site publishing, allowing specified site collections to be published to other publishing site collections. Also, the new Product Catalog (simply terrible naming) site template allowed you to create lists and libraries whose content could be published to other sites
  • Introduction of managed navigation, new feature that allowed site admins to define and maintain the navigation on a site collection by using managed metadata term sets
  • Improved SEO for public facing websites, such as user friendly URLs, allowing verification of site ownership, the ability to edit sitemap settings (such as the robots.txt) for search engines, and the ability to create SEO meta tags.
  • Improved branding options, allowing the use of standard HTML, CSS and JavaScript designs and workflows to brand SharePoint, rather than constraining designers to using only Visual studio or SharePoint Designer
  • As mentioned above, the introduction of device specific channels
• Related to the above improvements in WCM, SharePoint Internet sites no longer required an expensive licence. The use of cheaper standard server licence for public facing sites meant that organisations could host their internet and intranet on the same platform, reducing costs and providing more flexibility.

The new features for developers included:

• The new App Model (see below)
• Major improvements in SharePoint search, with the core SharePoint 2010 Search product and Fast Search for SharePoint 2010 being merged and extensively overhauled into a single product called SharePoint 2013 search that was common between the Foundation and Standard versions.
• Introduction of Shredded Storage to store large files in the database as small shreds instead of a single BLOB (binary large object). The file shreds are combined into a single document when retrieved from the database. This aimed to reduce the I/O of updates to existing files (only the file shreds that are modified are uploaded). So if only a file’s metadata is updated, the file shreds are not updated (in SharePoint 2010, a copy of the document’s BLOB was created). Unfortunately, this feature was poorly implemented, and could result in increasing rather than decreasing file storage in many common scenarios, as well as increasing the time required to retrieve documents from SharePoint. Additionally, this also caused some issues for admins, as they could no longer retrieve documents directly from content database backups as was possible in SharePoint 2010.
• Addition of OAuth authorization (used in the new App Model)
• As already mentioned above, the introduction of Device Channels for mobile/tablet specific page rendering.
• The JavaScript Link property (JSLink) that allows the user interface of fields and lists to be modified using a JavaScript file, as opposed to using modified XSLT views as in previous versions. A small but very useful change that allowed easy customization of the UI.
• Introduction of the Minimal Download Strategy (MDS) to download only the differences between pages when navigating between them. This improves rendering performance when browsing content where large parts of the page do not change. MDS is enabled by default on team and community site collections.
• Support for HTML5
• Significant improvements to Workflow. As well as being backwards compatible with SharePoint 2010 workflows (based on Windows Workflow Foundation 3.5), SharePoint 2013 used the new Workflow Manager service, based on the Windows Workflow Foundation in .NET Framework 4.5. The new Workflow Manager is external to SharePoint, and can run on completely separate server(s), allowing for greater performance and scale. The Workflow Manager and SharePoint WFEs communicate using a REST API secured by OAuth. Note, it isn’t possible to automatically upgrade from older SharePoint 2010 workflow to the new workflow types.

App Model

For developers, the main change with the release of SharePoint 2013 was the introduction of the new App Model (later renamed the Add-in Model). Previously, developers wrote custom server-side code that was run within the SharePoint process (either as a fully trusted solution or within the sandboxed user code service). SharePoint Apps, however, are run external to the SharePoint process, running within the browser as a client-side solution, or remotely on some other server or service external to SharePoint.

What did this mean in effect? Developers would no longer develop SharePoint solutions using C# to run on top of SharePoint, but would instead develop apps (essentially web applications) using HTML5, CSS and JavaScript using current web development best practices. The apps would be loosely coupled with SharePoint, consuming data using the SharePoint CSOM, JSOM and REST APIs, rather than using the server side API. These new apps would be compatible across both on-premise SharePoint and SharePoint online instances, and would provide users with ‘No Code’ (as in no server side SharePoint code) customization solutions.

The new App Model had 3 separate deployment models:

• SharePoint Hosted App – the solution components are hosted on either an on-premise or Office 365 SharePoint farm. SharePoint hosted apps are installed on a SharePoint 2013 website, called the host web. They have their resources hosted on an isolated subsite of a host web, called the app web.
• Provider Hosted App -  also known as a self-hosted app, this solution include components that are deployed and hosted outside the SharePoint farm. They are installed to the host web, but their remote components are hosted on another server. The separately hosted components are typically an ASP.NET application, but they can be developed in any server side technology (such as Node.js, PHP) that can use OAuth and consume the SharePoint REST API.
• Automatically Provisioned Azure App – also known as auto-hosted apps, these were cloud-hosted apps whose remote components are provisioned and deployed on Windows Azure. As with the provider hosted app, it could interact with a SharePoint website but also uses resources and services that are located on a remote site that is hosted by Windows Azure. This deployment option was only available to SharePoint Online customers.

As app stores were all the rage (Apple’s App Store, Google Play for Android, the Windows Phone Store, Windows 8 Store), SharePoint also got one (the Office Store). Corporations got an internal app store, the internal app catalog where only apps approved for use within the organization were available.

SharePoint Online/Office 365

The SharePoint Online service was updated to use the SharePoint 2013 version at the end of February 2013. Initially, it was identical to the on premise version. However, Microsoft quickly started to add new features to SharePoint Online/Office 365 that were not available with the on-premise installations. These unique features included Yammer integration (for Enterprise subscriptions), and then, later, Delve, a personal search and discovery tool (think Cortana/Sir for work). The quick iterative nature of the updates to Office 365/SharePoint Online reflected the new 90-120 day release cycle that Microsoft was using. Other changes for SharePoint Online included the new App Model and the Office Store mentioned above.

The use of a hybrid architecture, with some SharePoint sites and services running in Office 365/SharePoint Online, and some on-premises, gained popularity after the release of SharePoint 2013. In particular, many organisations opted to host their MySites with SharePoint Online, while retaining the majority of their data in an on-premise farm. For those organisations, this removed the need to police the contents of an employee’s MySite and meant they could simply offshore the problem to Microsoft’s data centres.

Conclusions

For end users, SharePoint 2013 was a major improvement with its improved UI and focus on usability. For those administering SharePoint, there were significant improvements,  particularly the new Web Content Management features. For developers, however, the release was more of a mixed bag.

The notable improvements were the introduction of JSLink, and the new Search and Workflow engines. Unfortunately, there were quite a few ‘meh’ features, including the new Shredded Storage which had a sloppy implementation. Another was Device Channels – why base this feature solution on User Agent strings, and completely ignore CSS3 media queries? Similarly, HTML5 support was great, but being able to build components for SharePoint with ASP.NET MVC rather than legacy Web Forms would have been even better.

In addition, I felt in this release that there were several major failings. The first, and the most surprising given that the trouble that Microsoft had went to to improve the UI and to introduce Device Channels for mobiles and tablets, was that there was no official Microsoft SharePoint app for iOS or Android (or even Microsoft Phone, not that anyone actually owned one…). There was the newsfeed app, but if you want a native app to browse and view content on a SharePoint site, you had to use third party offerings.

The second major fail was the new App Model. It sounds like a brilliant idea. No longer would developers be forced to use clunky old C# full trust solutions to customize SharePoint that would risk the stability and performance of the SharePoint server (and more importantly, Microsoft’s SharePoint Online service). Instead, they could deliver solutions across both on-premise and hosted SharePoint sites using the latest and greatest web development practices. Developers obviously rushed to use the new App Model to develop their SharePoint solutions, right?

Well, actually, no. In fact, practically no one used the new app model, as SharePoint developers could see it for what it was – a desperate attempt by Microsoft to solve the no server code customization issue for their SharePoint Online service. They had been burnt by Microsoft pushing Sandboxed solutions in SharePoint 2010, in a previous attempt to solve the same issue, and were understandably wary of jumping on another development model that could be killed off with the next release.

The App Model wasn’t helped by the fact that it was more complex to implement and deploy than full trust solutions, or that it didn’t provide the same range of customization options as either full trust or the more restrictive Sandbox solutions. In particular, it was unable to:

• Deploy resources to the SharePoint hive, such as master pages, application pages, CSS, JavaScript or image files
• Provide any delegate control elements or user controls (*.ascx files)
• Deploy custom Site Definitions
• Perform Custom Action Groups & Hiding Actions
• Deploy Custom themes
• Create a Timer job

Effectively, if it wasn’t possible to do something with the CSOM API, it wasn’t possible to do it with SharePoint Apps.

In time, with the help of the excellent Patterns and Practices project (PnP), some of these major gaps were (partially) filled in, but for some problems, you would still have to use full trust solutions. This meant that when customising for an on-premises solution, you were being asked to support 3 different development models: full trust code, Sandboxed solutions and the App Model.

Of the 3 different deployment types for the App Model, the Provider Hosted App was considered the most useful. Very few people used the SharePoint Hosted App due to issues with authentication and upgrading, and so few used the Azure Hosted Apps that Microsoft removed the deployment option within 2 years of release, stating scalability and manageability concerns.

In the end, most SharePoint developers simply ignored the App Model, despite all of Microsoft’s marketing. The ideas behind the App Model were correct. The SharePoint architecture and development methods were still based on ASP.NET 2.0 Web Forms, which was a decade old. Customers wanted SharePoint solutions that made use of web development best practices and frameworks, and a UI that was responsive. Unfortunately, the App Model simply wasn’t going to deliver that.

A third failing was the Yammer integration. While it was possible to integrate it with an on-premise SharePoint 2013 farm, there was never a clear reason to do so. Yammer never offered enough value to business to be used instead of the existing SharePoint social features.

The Hybrid approach was heavily promoted by Microsoft, as a way of introducing organisations to Office 365. But actually implementing a hybrid architecture was difficult, and was the most complex option (compared to simple on-premise or SharePoint Online solutions). Hybrid really only offered:

• Search across both environments (but with a very poor user experience, as the search results from the 2 environments weren’t merged together)
• BCS allowing access to data in on-premise and Office 365
• Duet integration allowing access to data in on-premises SAP from Office 365

There was no shared global navigation across the 2 environments, or a global site directory. The branding across the 2 environments could not be synchronized, and likewise, configuration settings for one environment were not replicated across to the other. The link between the 2 environments was really quite minimal.

A Hybrid architecture was, however, attractive to Microsoft, as you were effectively paying twice for the same solution – the licencing costs of your on-premise servers, and the monthly subscription for Office 365.

Overall, SharePoint 2013 was worth upgrading to, due to the real improvements for end users, and a number of notable new features such as the WCM improvements. Would the next SharePoint release, which Microsoft hinted would be available in under 2 years due to their new release cycle, be as useful? I’ll look at this in my next post in this series.

SharePoint Development – SharePoint 2010

This post is the third in a series looking at how SharePoint development has changed over the years, and how it is likely to change in the future. In this post, we will look the SharePoint 2010 release.

More posts in the series:

• Part 0 - SharePoint Development - Past, Present and Future (the series introduction)
• Part 1 - SharePoint Origins (including 2001 and 2003 releases)
• Part 2 - MOSS 2007
• Part 3 - SharePoint 2010 (this post)
• Part 4 - SharePoint 2013
• Part 5 - SharePoint 2016
• Part 6 - The Future of SharePoint Development

 

SharePoint 2010

The 2010 release built on the growing popularity of SharePoint following MOSS 2007, and it was clear that Microsoft had ambitious goals for SharePoint. Notably, the Microsoft CEO Steve Ballmer referred to it as "like an operating system" in the keynote presentation at the Microsoft SharePoint Conference in 2009. This release was seen as positioning SharePoint as a business collaboration platform that would fill all the needs of an information worker. As a result, there was huge marketing drive for the SharePoint 2010 that targeted end users. These changes were reflected in the updated functional pie chart, from the emphasis on IT Solutions in SharePoint 2007, to the emphasis on end user and business solutions in SharePoint 2010.

Comparison of SharePoint 2007 and 2010 features (taken from https://nikpatel.net/2010/02/14/sharepoint-2010-end-users-improvements-and-new-features-overview/)

In this release, SharePoint was positioned separately to both Office and Windows brands, with references to both removed from the version names. There were 3 versions:

• Foundation (free, corresponding to WSS in the earlier releases)
• Standard (with additional search functionality)
• Enterprise (additional content management capability)

SharePoint 2010 was launched globally in May 2010. There were a large number of new features in this release:

• An enhanced user interface:
  • Introduction of the fluent ribbon to mimic the changes to the Office UI, minimizing the number of mouse clicks required
  • Multi-browser support
  • Use of Wiki pages rather than web-part pages in templates
  • New themes to customise a site's appearance
  • Mobile Device Access (to render mobile views of the default SharePoint templates)
• Improved ways of integrating with existing business data
  • The InfoPath Forms service was updated to allow customization of the default SharePoint list forms
  • Introduction of PowerPivot, a special Excel workbook that can be shared and updated in a SharePoint site, as part of a drive to add Business Intelligence capability to SharePoint
  • Two way Business Connectivity Service (BCS , formerly the Business Data Catalog in MOSS 2007) to read and write data from external systems
    • Use of External Content Types to define external data sources, and to allow creation of external lists that can be used in the same way as SharePoint's own lists
• Improved workflow options
• Updated user profile and introduction of social networking features:
  • User profiles now included photos and presence indicator, status updates
  • Tagging of both user profiles and content
  • Users could now create their own blogs
  • Colleague suggestions
  • An Organisation chart to show a user's place in the organisation hierarchy
• Records Management (retention and auditing policies, records repository, email content as records, legal holds)
• Improved integration with Microsoft Office, including the ability to open content and edit in SharePoint directly in Word and Excel
  • Office Web Apps (OWA) was introduced as an add-on to SharePoint as the online companion to Word, Excel, PowerPoint and OneNote applications that enables users to access and edit documents from anywhere. This allowed for the replacement of the traditional desktop Microsoft Office clients.
• Improved governance and IT management tools, to control what end users can do with SharePoint. Notably, this was to prevent duplication of content and to also prevent users creating sites without any oversight, as these were major problems with the MOSS 2007 release.
• Improved Content Management
  • Document Sets
  • Shared Content Types
  • A new Managed Metadata Service was introduced to allow for tagging of content, and the use of taxonomy hierarchies for implementing logical data architectures
  • Rich Media management

This is by no means a comprehensive list of the new features in SharePoint 2010 (but you can find a full list of the new features and comparisons to previous versions here, here and also here).

Developing for SharePoint 2010

The SharePoint 2010 release involved massive changes for developers. This wasn’t just an incremental release on top of the previous 2007 release. There were significant architectural changes, mainly aimed at removing the scalability and performance issues that had limited MOSS 2007 as an enterprise platform. Notably, Microsoft had experienced these problems when managing their online SharePoint service, Business Productivity Online Services (BPOS). These architectural changes included:

• Moving from a 32-bit to 64-bit architecture, giving a significant performance boost. This also meant that upgrading from MOSS 2007 involved significant changes to the hosting infrastructure.
• The replacement of Shared Service Providers by Service Applications to share resources across sites in different web applications and farms. The new service application architecture allowed for the use of dedicated application servers, and also lent itself to multi-tenancy scenarios, which was particularly important for Microsoft’s online offering.
• Improved the boundaries and thresholds for scalability, including:
  • Increased the list view threshold from 2000 (in MOSS 2007) to 5000 (20,000 for administrators)
  • Increased the number of content databases supported per web application from 100 to 300
  • Maximum content database size increased from 100 GB to 200 GB
  • The full list of boundaries and limits for SharePoint 2007 can be found here, and for 2010, here.
• The introduction of the Sandbox Solutions to allow site collection administrators to monitor and control the resources consumed by web parts and other  solutions in a site collection. Sandboxed solutions were hosted in the new SharePoint user code solution worker process (SPUCWorkerProcess.exe), and the code could only affect the performance of the site collection it was deployed to. As the code was not run in the IIS worker process (as in the full trust farm solutions that had previously been used), the sandboxed code could not affect the performance or stability of solutions in other site collections. This had been a common issue for Microsoft when running their multi-tenancy BPOS solution – poorly coded web parts in one tenant’s site collection would affect the availability of other tenants solutions running on the same web application. The new Sandbox Solution framework were created to solve this problem. Microsoft was so insistent on Sandbox Solutions being used by developers that they made them the default solution type when creating a new SharePoint solution in Visual Studio, and SharePoint Online (the BPOS successor, see below) would only accept Sandbox Solutions.

The changes in the architecture are highlighted by below between the two diagrams:

SharePoint 2007 Architecture

SharePoint 2010 Architecture

In addition to the Sandbox Solutions, Microsoft had a major push to educate developers on common programming pitfalls and performance issues when developing SharePoint solutions, and released the SPDisposeCheck tool (a static code analyser) to help identify poorly coded solutions.

There were a number of important changes to improve the administration of SharePoint:

• The Central Administration application interface was updated, including the introduction of the ribbon and wizards to step through common configuration tasks (including the initial configuration of a farm).
• In addition to the STSADM executable, PowerShell was introduced to provide command-line administration. As an early adopter of PowerShell, I particularly welcomed the addition of PowerShell to the SharePoint developer’s toolbox.
• Managed Accounts were added, allowing for the management of service accounts directly in Central Administration, instead of having to manage these accounts directly in the AD.

The tooling and best practices for developers changed radically with the release of SharePoint 2010. Notably, there were now 3 separate APIs available to develop against:

1. Server-side Object Model – the existing .NET API, updated to use .NET Framework 3.5.
2. Client Object Model – this was a new API designed to allow developers to produce solutions for use in remote client-side solutions that run on computers where SharePoint 2010 is not installed. This included a Client Class Library (for use in  .NET managed or Silverlight applications) and a JavaScript class library (an unmanaged ECMAScript object model for use in the browser).
3. SharePoint 2010 Web Services- updated to have a RESTful interface.

The changes to the API model reflected the improvements in web development techniques and a new emphasis on creating rich Internet applications (RIAs) in the browser for an improved user experiences using Ajax and Silverlight. It also reflected the new emphasis on SharePoint as an enterprise platform – the new APIs allowed integration with other enterprise systems.

In addition to the new APIs, the Business Connectivity Services (BCS), the successor to the Business Data Catalog, had significant new features to connect to external data sources outside of SharePoint and literally treat them as a SharePoint List. These connections were now two way (read/write) connections. Out of the box, the BCS could connect to SQL databases and web services, and also provided a pluggable Connector Framework to allow connectors for new external systems to be developed.

Similarly, the User Profile service was revamped in SharePoint 2010, with the associated synchronization service also made a two way connection, so that changes made in a SharePoint user profile could be written back to an organisation’s AD. Personally, I came to dread dealing with it (and the associated Microsoft Forefront Identity Manager (FIM) engine). The initial documentation on TechNet was very poor, so I came to rely on Spencer Harbar’s excellent Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization.

For developers, there were significant improvements in the tooling for SharePoint 2010, including the introduction of a number of different SharePoint project templates in Visual Studio 2010. It was now possible to deploy and upgrade features and solutions from within Visual Studio, instead of having to rely on third party tools and extensions.

Developers were encouraged to start using the new LINQ to SharePoint provider, building on the popularity of the LINQ to SQL ORM that was released in 2007. LINQ (Language Integrated Query) is a .NET Framework component that adds native data querying capabilities that was introduced with C# 3.0. The LINQ to SharePoint provider promised to abstract away the complexity of writing CAML to query SharePoint lists. Sadly, the implementation of the first release was poor – you can read the details in Bjørn Furuknap's article. There were also a number of other open source libraries that provided similar functionality, such as Camlex.NET.

Accessibility was improved in SharePoint 2010, after being famously dire in MOSS 2007. This had been a major issue in the take-up of SharePoint solutions by UK public sector organisations who had a duty to provide accessible websites to their users. SharePoint 2010 was marketed as being WCAG 2.0 AA compliant, and also standards (W3C XHTML) and cross browser complaint. In reality, while still a major step forward compared to MOSS 2007, SharePoint 2010 was not fully compliant to the WCAG 2.0 AA standard. I detailed the issues in a past article (the article is no longer online but is still available via the Internet Archive), but to develop a fully accessible web application using SharePoint 2010 required a significant developer effort, using custom control adaptors and JavaScript.  SharePoint 2010 could now support Firefox, Chrome and Safari in addition to Internet Explorer, but the differences in browser support for various features and web parts (mainly due to the use of ActiveX) caused a lot of work for developers. Many organisations were also forced to work around the lack of (official) support for Internet Explorer 6.

Other notable changes for developers included:

• In addition to the change to a 64-bit architecture already mentioned, SharePoint could also be installed on developer workstations (using Windows 7 or Vista OSs), instead of only on servers as previously.
• Improved search functionality with an emphasis on promoting information discovery across both structured and unstructured data. Additionally, the FAST search application was integrated into SharePoint only 18 months after Microsoft had acquired the company. Unfortunately, there were 3 separate search options available in SharePoint 2010 (Basic Search Center, Enterprise Search Center and FAST Search) which was needlessly confusing. This was finally resolved in the later SharePoint 2013, when the FAST search become the standard search engine in SharePoint 2013 and the SharePoint Online.
• Addition of web templates and provisioning providers, as well as updates to the existing SharePoint deployment approaches, site definitions and site templates (detailed here).
• The introduction of claims based authentication, using the Microsoft Identity framework (code name Geneva), allowing non-AD users to be authenticated (such as users using Hotmail and Gmail accounts).
• Major improvements in the information architecture, with improved ways of adding metadata to documents with managed terms and keywords.
• Introduction of Remote Blob Storage (RBS) to store large binary files such as video.

I don’t have space or time to do justice to these new features, or the many others that I have failed to list. But even from this brief overview, you can see that SharePoint was fundamentally rebuilt and extended with the 2010 release.

BPOS/SharePoint Online

As already discussed, many of the changes made in SharePoint 2010 were made to improve Microsoft’s BPOS offering. This led to to an identity crisis that Microsoft still hasn’t fully resolved - is SharePoint an online service or on-premise software? While Microsoft’s direction of travel is now clear (more on this later in the series), the contradictions between these two views of SharePoint have certainly affected how developers see the platform. However, it has been evident that SharePoint Online has contributed significantly to driving new features in SharePoint and in also reducing the product release cycle time.

With the development of Sandbox solutions, custom code could now be deployed to SharePoint Online. However, the feature set for SharePoint Online lagged considerably behind SharePoint 2010 on-premise. Following the release of Office and SharePoint 2010, the BPOS services were refreshed and re-launched as Office 365 in October 2010. Microsoft chose the name ‘Office 365’ to reflect "dependability every day of the year", due to BPOS’s very poor record of uptime.

Existing BPOS customers were given 12 months to upgrade to the new online service, with possibly complex migrations involving updating the client Office applications. This was an issue for many existing BPOS customers who were predominately small or medium-sized businesses. The demand for BPOS/Office 365 was seen as low, and the uptake for SharePoint Online at the time was even lower.

Conclusion

As described above, SharePoint 2010 was a massive improvement from MOSS 2007, both in terms of features for end users and organisations, and also for developers. But there were some notable failures. SharePoint attempted to do everything, but did very few things well. The new blogging and wiki features were underwhelming. Similarly, the accessibility story fell considerably short of the marketing hype. I recall repeatedly having to spend considerable amounts of time developing SharePoint solutions that were fully WCAG 2.0 AA compliant for UK public sector clients.

SharePoint 2010 was widely marketed as a suitable platform for Enterprise development, and I worked on a number of large scale projects (with 20+ development teams) to develop various platforms. Some were successful, most were not. One notable failure I recall was an attempt to develop an ecommerce platform on top of SharePoint 2010. This was a clear case of believing marketing hype over reality, as SharePoint was never suitable for developing this type of solution.

With the huge growth in SharePoint usage (detailed in my previous post), a large number of ASP.NET developers started to develop solutions in SharePoint, with mixed results. At one point in my career, I was employed in a large SharePoint team in a major consulting firm that was responsible for cleaning up SharePoint solutions gone wrong. Without fail, the root cause of these failures were architects and developers attempting to develop SharePoint solutions without prior experience. Reading one of the many developer books published on SharePoint 2010 didn’t make you a SharePoint expert, though I came across many who believed it did.

One common anti-pattern was attempting to use SharePoint as a relational database, with developers using SharePoint lists as data tables for massive numbers of records. Another was attempting to install and configure SharePoint using only 1-2 service accounts, instead of the 15-20 accounts actually required. While fine for creating a developer environment, this would cause no end of problems in Production, particularly with the User Profile Synchronization Service. And yet another common failure was re-using service accounts used in an existing MOSS 2007 farm for your new SharePoint 2010 installation – this would always result in constantly locked accounts when password changes failed to synchronize across the two farms.

Despite the major improvements in SharePoint 2010, I now see the release as damaging Microsoft’s relationship with developers. While the new APIs were significant, Microsoft’s push to have developers build Silverlight clients for SharePoint would rebound when the company would side-line Silverlight in favour of HTML5 towards the end of 2010. Likewise, the LINQ to SharePoint provider was killed off along with LINQ to SQL, when Microsoft settled on Entity Framework as their chosen ORM.

However, probably the biggest setback was the introduction of Sandbox solutions. Introduced to overcome performance and instability issues introduced by badly coded farm solutions, particularly in Microsoft’s BPOS service, they were made the default solution type in Visual Studio. However, these solutions offered fewer features but were significantly more complex to implement, especially when attempting to use APIs that were not by default supported within the sandbox (full-trust proxies and the dreaded service locator, anyone?). As a result, very few developers bothered to use them.

The introduction of Sandbox solutions was clear case of Microsoft’s SharePoint identity crisis – was it online or on-premise, service or software? Some very quickly came to realise that the future of SharePoint was to be subsumed in the Office 365 brand. At the time, despite being aware of the major architectural changes going on, I tended to sneer at SharePoint Online as a toy offering, given the very low take-up of the service.

These and other issues (like the completely unnecessary debate over the future of .NET in Windows 8)  would lead to SharePoint developers becoming reluctant to adopt new frameworks and features being touted by Microsoft - more on this trend later in the series.

In the next post in the series, I’ll look at SharePoint 2013. Apologies for the delay in publishing this series – the combination of the Christmas holidays and the baby has meant that I’m running significantly behind in my blogging schedule. Also, thank you to all those who have sent emails about this blog series. All comments or feedback are gratefully received.

Goodbye to Delicious, Hello to Pocket

After some 6 years of using Delicious to store my bookmarks, I've finally admitted that the service is effectively dead, and have now moved to use Pocket instead. I’ve updated my Links page accordingly, and I have also deleted the old bookmarks RSS feed on FeedBurner (another service in terminal decline).

The final straw for me with Delicious was finding that the export option was no longer available, and the developer API was gone. This meant that I had no way to export and back-up my some 5000 bookmarked links. Luckily, I had an export of my Delicious bookmarks from last year. I went through my most recent Delicious bookmarks and manually transferred the most useful to my Pocket reading list.

I’ve been using Pocket for over 8 years, back when it was know as Read It Later and it was a simple reading list add-on for Firefox. Since then it has developed considerably. I’m particularly fond of the excellent iOS app, something I use daily. The premium subscription is a bit steep ($45/£37 per annum), but given I’ve been using their service for so long for nothing, it seems only right to pony up for it. I did consider combing Pocket with PinBoard (and this may happen at a later date), but Pocket alone is sufficient for the moment. I’m not interested in saving 1000s of links that I’ll never look at again, as I did with Delicious.

Again, I’m struck how important it is that you keep control of your data, even something as minor as a list of bookmarks. This is something I touched on earlier last year when I wrote about the shutdown of the Shelfari service. Often, you will lose this control when using a free/freemium service, as the service provider gradually attempts to lock users in.

As a result of thinking about this, I plan to move to use either paid services that give me more control of my data, or to use self-hosted open source applications. I’ll be writing more on this soon. If you have any thoughts on this, I would love to hear from you.