Saturday 22 August 2015

Why Open Source Software Defined Storage Will Win

Last week NetApp posted it's first quarterly loss in many years. It's not something I take pleasure in, since I have many friends still working at that company and it still has some very interesting technologies. The storm clouds are gathering though, and I can't help but liken NetApp's situation to that of Sun Microsystems, another former employer, as the tech bubble burst in the early 2000's, especially when I look at some of the comments reported from the NetApp earnings call.

Back in the day, Sun was making a lot of money and good margins with high performance proprietary hardware

Then along came Linux running some simple tasks on commodity hardware. it was good enough to do the simple jobs, had a lot of the essential features & functionality that unix provided, and the hardware was at such a low price that the economics were impossible to ignore. Some of the first adopters were the same companies that been the early sun customers, those who had replaced mainframes & minicomputers with cheaper, yet effective Sun hardware.

Unfortunately the lesson of their  own early success wasn't remembered by Sun's senior management, who thought they could win customers back by offering Linux on Sun hardware. Of course, the software wasn't the point here – it was the low-cost hardware that was attracting attention. Some of us in the field & parts of engineering tried to convince Sun's management to put more effort behind Solaris for x86, but just at a critical juncture, Solaris 9 was released on SPARC, along with the message that x86  support would be put on the back-burner ... the die was cast and Sun's fate sealed as commodity hardware pushed the boundaries of performance and drove the need for proprietary hardware into an upper niche. I still contend that if Sun had instead fully committed support for x86 & embraced a software-first approach,  the Solaris would dominate the market & Linux would have been relegated to a similar position as FreeBSD occupies today.

What does this have to do with software defined storage & NetApp? Well it appears that there is a similar approach from NetApp's senior management:  they see that customers "want scale out and software defined storage functionality",  but seem to think that the only response is a solution running on NetApp hardware. Like Sun, they (and the other major storage vendors) are chained to their high-margin proprietary hardware. Breaking free of this kind of entanglement is at the crux of Christensen's Innovator's Dilemma.

Meanwhile open source, software-defined storage solutions running on commodity hardware, such as SUSE Enterprise Storage based on Ceph – the Linux of enterprise storage if you like – are starting to gain attention, especially for large bulk data stores facing exponential growth and price-sensitivity. For the moment these solutions are best suited to relatively simple (albeit large) deployments, but the technology is not standing still. Ceph already features high-end technologies like snapshots, zero-copy cloning, cache-tiering and erasure coding, and enterprises are finding that it is "good enough" and at a dramatically lower price-point.  The open source nature of the development means that progress is rapid, and the software-defined nature means that hardware costs are driven relentlessly down. This is the same dynamics as we saw with the transition of UNIX to Linux, and it likely to have the same impact on the proprietary enterprise storage vendors.

So, change is here: I hope NetApp can navigate the course better than Sun did, though it will be interesting to see how.

Meanwhile, enterprises looking to rein in the exponential costs of enterprise storage can now look to open source for answers, and take advantage of the power and economics of commodity hardware systems.

Friday 15 May 2015

Conflict within Teams – Inevitable & Necessary ?

This was originally a scholarly article written as a discussion of whether conflict in teams is both inevitable and necessary.  The style is a little more formal for that reason.

Modern organisations are increasingly reliant on teams and teamwork to conduct the bulk of their business. These teams exist at all levels – from manufacturing and customer support at the front-line of operations, through management layers and even into the executive management of a organisation and its board of directors or trustees. Teams can be formed from groups of workers within a single area of responsibility, or can be cross-functional teams including participants from many different parts of the organisation working together on a single project. Following improvements in telecommunications and information technology, team members may not even be in the same physical location, but make up “virtual” teams representing different geographic locations or simply taking advantage of the ability to work remotely.

At the same time as teams and teamwork are becoming more important, the working population is continuing to diversify, including people with obvious differences in gender and racial background, but also including the less-obvious diversity of people from a wider range of disciplines and training backgrounds than were previously evident in business. With the aging population of many Western nations, there is also greater likelihood of a wide range of ages being included in any given work group, which in turn leads to diversity of experiences, attitudes and expectations between team members.
This increase in diversity leads to the question of conflict occurring between team members. Although conflict is traditionally regarded as a negative thing, and not conducive to productivity and satisfaction, some recent theories of management practice suggest that conflict may have beneficial effects on performance. This essay examines the question of whether conflict in teams is both inevitable and necessary, with a focus on the modern working environment. The evolution of thinking around this question – in particular the necessity of conflict – will be studied along with the results of experiments performed by researchers in the field.
It is clear from the literature that this topic is very controversial and the subject of continuing debate amongst management theorists. This paper will examine and present the argument that although conflict of some form is highly likely in modern teams, and although it can lead to some benefits in innovation and performance, the negative factors are such as to make it highly undesirable, and that conflict management techniques should be used to mediate these negative effects.


What is conflict?

In order to properly understand whether conflict is inevitable or necessary, it is important to understand what is meant by conflict in the context of workplace teams. Intragroup conflict can be broadly defined as perceived incompatibilities between team members (Jehn, 1995), but it is self-evident that incompatibilities can take more than one form, and so the nature of conflict must be further investigated.
Early theories of the types of conflict include structural models (Deutsch, as cited in Korsgaard, et al, 2008), and process models (Pondy, 1967). Structural models theorise that conflict occurs as a result of interdependence and incompatibility between parties; when goals of one team member can only be achieved at the expense of another team member, conflict is increasingly likely to result. It is the structure of the inter-relationship between parties which gives rise to conflict, and in this theory focus is given to the importance of task characteristics, social context, and the relationships of participants.
Process models take the view that conflict is a series of episodes, each of which commences with some pre-existing conflict potentials, and concludes with an aftermath which affects future episodes. The participants in the conflict go through a process of interpretation, or sense making, after each event which leads to feelings, thoughts and actions in response, which in turn leads to consequences for future interactions of the parties. The process model highlights the role of perception in conflict development, and is infused with affective reactions such as anger (Korsgaard et al, 2008).
The combination of structural and process theories suggest that conflict can be seen as three linked factors leading to conflict (Korsgaard et al, 2008), viz:
  • Inputs (individual differences; individual traits; task structure; social context) leading to...
  • Behaviour (conflict provoking behaviour; 2-person exchanges; group interactions) leading to...
  • Sense Making (identifying an event as an offence; laying blame on another party) which leads to conflict
Manifestations of conflict include frustration, anger, intentions to rectify and/or retaliate, and interference of cognition and effectiveness.
More recent studies of intragroup conflict have focussed on distinguishing between different types of conflict, and their effects on teams where the group is responsible for some collective product or decision (De Dreu & Weingart, 2003). The current typology of conflict includes three broad areas; task conflict, relationship conflict, and process conflict (Jehn, 1997; Korsgaard et al, 2008)
Task conflict (also known as cognitive conflict) is disagreement related to the job at hand: what should be included in the task, how to approach it, ideas, issues and content. Task conflict has a predominantly intellectual basis, and is strongly influenced by disparate backgrounds in the participants of a team (Li & Hambrick, 2005).
Relationship conflict (also referred to as affective conflict) involves interpersonal incompatibilities, tension and antagonism (Jehn, 1995). It has an emotional basis and impacts the ability of team members to share and assess information within the group (Jehn & Mannix, 2001)
Process conflict, according to Jehn (1997), is “conflict about how task accomplishment should proceed in the work unit, who's responsible for what, and how things should be delegated. It is more about the logistics of “who does what” rather than the task conflict nature of “what should be done”. Process conflicts impact on the team's ability to coordinate effectively (Greer, Jehn & Mannix, 2008).
It is also important to understand that conflict is dynamic – it evolves over time, and the above three types of conflict can interact and transform from one kind into another during the lifespan of a team (Greer, et al, 2008).


Is conflict inevitable?

It can be asserted that conflict in teams is inevitable since creative collaboration is often the team's reason for existence (Kling 2009), and indeed although this assertion may not be completely borne out in all cases, the literature generally seems to take conflict of some kind as a given in team environments and any other kind of group or organisation (eg Jehn, 1995; De Dreu 2008).
In a study of 60 cross-functional teams in high technology companies (Lovelace , Shapiro & Weingart, 2001), it was found that all teams experienced some degree of intra-team disagreement. Hobman & Bordia (2006) also found direct associations between dissimilarity in individuals and the likelihood of conflict; given that everyone brings some degree of difference to any group, conflict of some kind is likely to occur.
However, it is worth considering whether the inevitability of conflict in the workplace is influenced by cultural constraints. Individual performance and achievement is generally well-regarded in the individualistic Anglo-American cultures (including Australia), but in collectivist cultures such as those found in Asia and Southern Europe, greater emphasis is placed on intragroup harmony and personal relationships, and conflict may be perceived as a threat to the interests of the team (Passos & Caetano, 2005; Hempel, Zhang & Tjosvold, 2009).


Is conflict necessary?

From the 1990's until the present, conflict in teams has been interpreted as a vital component of team success (eg Amason et al, 1995; Karnani, 2008; Flanagan & Runde, 2009), and studies have shown that some degree of conflict may be required to develop more creative thinking than might be achieved in a group with convergent thinking (Jehn, 1995). It has also been suggested that in cases where pressure is being applied by an external source, the “groupthink” of convergent teams may lead to the premature selection of a less than optimal solution (Paulus, 2002). Conflict, however, can introduce “vigorous debate” on issues requiring critical analysis or innovative thinking (Flanagan & Runde, 2008).
Through interviews with managers of teams, Amason et al (1995) found that the presence of task conflict enhanced the ability of teams to make creative decisions. This improved creativity was reported in cases where participants brought different points of view to a problem and were supported by the right environment to manage the conflict. In their interviews, participants related the belief that the improved creativity of teams with some degree of task conflict resulted in valuable business decisions, and that the lack of conflict in some decisions resulted in substantial costs or failures.
It is important to note that the type of conflict apparently yielding positive results in the above is task conflict. In cases where relationship conflict is evident in a team, the effect becomes dysfunctional, and can be a serious impairment to team performance (Amason, 1996). Similarly, process conflict has be found to have both a negative impact on team performance itself, and also the quality of transforming to other types of conflict over time, especially if experienced in the early life of a team (Greer et al, 2008).
As mentioned above, the correct supportive environment must be in place to gain the maximum innovative benefit of task conflict. This includes the freedom of team members to express doubts without a “win-lose” position being taken by other members of the team: that ideas can be freely expressed without the danger of participants self-censoring. When combined with collaborative communications, this allows for differences in ideas to be exchanged and developed by the team as a whole, rather than becoming the focus of antagonism and dissatisfaction (Lovelace et al, 2001). Participants must have a sense of trust and safety in their interactions, and possess sufficient emotional intelligence to be aware and cope if and when constructive conflict deteriorates into destructive behaviours (Flanagan & Runde, 2009).
Whilst the introduction of task conflict into a team in order to generate more creative ideas is often regarded as a positive thing, West (2002) has argued that it is implementation rather than creativity and idea generation that is the important factor in team performance, and that this requires high degrees of collaboration. Furthermore, De Dreu (2006) shows that the positive benefits of conflict on performance follows a curve – although a small amount can help stimulate ideas outside the status quo, too much conflict has a greatly negative impact (see Figure 1)
Figure 1: Curvilinear Relationship Between Task Conflict and Team Innovation (from De Dreu 2006)
This curvilinear effect was also noted by Jehn (1995), and can be explained in part by the conflict transformation discussed by Greer et al (2008): each of the three types of conflict (relationship, task, process) can feed into the others over time. The drop-off in innovation can also be attributed to situations where teams have positions that are so divergent at the start that they may never come to an agreed conclusion, or may not completely synthesise their ideas into a commonly-agreed solution within the timeframe required.

Intentionally introducing conflict (by, for example, assembling extreme diverse teams) is extremely difficult to manage; Chen (2006) notes that careful timing of positive conflict is necessary, along with training in conflict awareness and the development of significant trust and psychological support within the team. This may be a significant additional effort, since the diverse groups assembled in order to introduce task conflict are also likely to be sufficiently divergent that process conflicts will also arise (Jehn, Northcraft & Neale, 1999). The time taken for conflict awareness training and the development of team cohesion must be accounted for in a project's schedule and costs.

Similarly, in their study of cross-functional teams, Lovelace, Shapiro & Weingart (2001) found that innovation and time & cost efficiency was most positively influenced by the way their disagreements were managed by teams members as well as leaders. Successful conflict resolution techniques require the team members to have a shared sense of responsibility and purpose (Flanagan & Runde, 2009), yet developing the skills for effective conflict management are an additional cost to the organisation in terms of both money and time.

Careful management of conflict is also needed because, according to De Dreu (2008), there is only a very narrow set of conditions under which conflict has a positive function. For example, conflicts should be task conflicts only, be of no more than moderate intensity, and are only of use if the team would otherwise have preferred a “suboptimal” solution.

It should also be noted that for teams working on a well-defined procedural task rather than a problem requiring innovative solutions, task conflict is perceived as a significant source of dissatisfaction and negative performance impact, since it distracts team members form the “real” work they are trying to achieve (Jehn, 1995).

Meta-analysis of many research reports has found that while relationship conflict had the expected negative impact on team effectiveness and satisfaction, task conflict was equally negative in many the reported studies, and could not be clearly identified as purely positive in any (De Dreu & Weingart , 2003). De Dreu (2008) also notes the various costs of conflict and conflict management: in time, health and well-being of participants, and the displacement of costs to a third party (for example, the end-customer). The introduction of these (often initially hidden) costs must affect an organisation's decision on whether to introduce conflict, and how best to manage conflict overall. With this in mind it has been suggested that constructive conflict management is essential for mitigating the broadly negative effects of conflict in teams (De Dreu 2008).


Conclusion

The investigation, classification and analysis of different types of intragroup conflict has been going on for several decades, leading to ever-more nuanced understandings of how people interact in teams. That conflict will occur seems to be universally accepted as inevitable in the Anglo-American world, although the prevalence of intragroup conflict may be muted in more collectivist cultures where group harmony and interpersonal relationships are an intrinsic part of business.

Relationship conflict is almost universally regarded as being detrimental to team performance, and process conflict has been shown to be very damaging especially when encountered in the early stages of a team's lifespan. What remains contentious is the degree to which task conflict is necessary or even desirable. It certainly appears to be the case that for teams required to produce innovative results (as opposed to those performing well-defined and standardized procedures), some degree of task conflict is necessary in order to generate a wider range of ideas that might otherwise be developed by a group with convergent thinking. This concept has even given rise to the saying “thinking outside the box”, which is often hailed as a positive concept in modern business.

It is important to understand that all conflict - however benign - comes at a cost of some kind, and the additional time required to work through the increased range of ideas brought about by task conflict may not be available in certain types of project. Similarly, process conflict can substantially affect the efficient delivery of the end result, and if implementation is an important factor, then group cohesion and well-defined roles should also be considered vital. Finally, attention should be paid to the idea that conflict is dynamic, and without careful management can easily transform from task-based to relationship- and/or process-based, and hence cause significant damage to team cohesion, performance and participant satisfaction.

It appears that introducing conflict in order to improve team performance is a technique fraught with peril, and requires a great deal of skill to manage effectively: if conflict is necessary, it is a necessary evil. In any case, proper management strategies should be put into place in order to deal with the inevitable conflict that will arise in teams, with a focus on reducing the negative impact, rather than any attempt to engender productivity gains which may never be realised.



Tuesday 5 May 2015

Cloud Adoption in Asia

I was recently asked to compare and contrast the adoption and attitude towards cloud in Asia vs the UK. This is a little less straightfoward as the questioner intended, I think, since “Asia” is a very broad market, with several distinct regions and attitudes to Cloud and Software as a Service.
Still, let's look at how things stand in Japan, Korea, China, Australia, India & South-East Asia.


Japan is still generally a very conservative IT market, however Cloud & SaaS is being looked at carefully. There are definitely examples of major corporations making significant investments in cloud and especially SaaS technologies (for example Toyota is the largest customer of SalesForce.com).   While big enterprise has the resources to move into cloud, and especially private cloud, the chief adopters of public cloud are small-medium enterprise, with adoptions patterns very similar to western countries and for the same reasons of economy.

Korea can be even more conservative than Japan, and generally heavy reliance on older Windows technology (ActiveX, etc) from the early 2000's makes migration to Web 2.x & cloud/SaaS less straightforward in many cases. However, once one large enterprise moves, the rest of the market will follow quickly.

In Australia, cloud technologies are being investigated by large enterprises who have the resources to spare (eg the big banks), and up-coming businesses looking to leapfrog the old fixed-system way of working. Still, the "cloud first" mentality is still a couple of years behind the US.

China shows lots of interest in cloud, with several large service providers offering cloud and SaaS services, and there is a lot of focus on how to develop cloud services with a Chinese flavour rather than straight adoption of western (particular US) systems. In general, government still generally favours traditional application deployment, and security is a significant concern. The adoption trend shows that smaller organisations are most likely to adopt cloud, either by migrating existing process or taking the opportunity to move directly into public cloud. Private cloud are much less common. Larger organisations may see cloud as a way of leapfrogging old deployment methodologies, but the automation and TCO benefits of private cloud can be less attractive where low-cost, high-skilled labour is available. In general it’s still true that there is not a lot of radical new innovation in Chinese IT, but the scale of deployments often pushes the boundaries of technologies.

In South-East Asia, especially outside of Singapore, the scale of IT deployment is generally too small to really gain advantages of on-premise cloud. As with Australia, there are places where small businesses are looking to leapfrog the old way of working & go directly to SaaS, or at least public cloud.  Even this is really more prevalent in Singapore than in the truly emergent economies.

For home-grown enterprises in India  (as opposed to outposts of US, etc companies), cloud & SaaS is an area of  early investigation.  Even more than in China, TCO benefits of private cloud deployments are often undercut by generally low labour costs, which in turn hampers adoption of a lot of operations automation technology.

In comparison to the rest of the world, my general perception is that Asia's cloud-adoption maturity lags behind  that of the US (where adoption of the new can be much faster than elsewhere, even if it's not necessarily a good idea), and also behind the UK and similar western economies.  On the positive side, this means that Asia can take advantage of the mistakes made and lessons learned, and possibly  entirely leapfrog legacy IT concepts that western world has to continue to deal with for many years to come.

From an employment point of view, there is opportunity for people from the US, the UK & the rest of the world to bring their cloud expertise to Asia – most importantly it is business-integration-related skills and experience that are needed, rather than purely technical exposure. The biggest headache most organisations face when moving to cloud is how to integrate business needs with access to the right resources. This is a big change from the way we've been working with IT over the past 30 years (or more) and so needs different approaches & ideas. Experience from people successful in this area will be invaluable to organisations in Asia.

Wednesday 22 April 2015

A Turning Point for OpenStack Cloud in the Enterprise

This past couple of days I've been at the CONNECT Expo in Melbourne, which included an OpenStack conference, where I participated as a speaker & panelist.

The focus of that conference was about whether OpenStack is ready for the enterprise, and included contributions from Dave Medbury from Time Warner Cable, Mike Dorman from GoDaddy, and Rik Harris from Telstra, who presented their real-world experiences with OpenStack in commercial settings.

One of the things that was very clear from the conference is that OpenStack is ready for the enterprise. This is probably not news to many who have been following the project for the past few years, but there definitely seems to have been a turning point in recent times, especially enterprise versions of the Juno release (such as SUSE OpenStack Cloud 5) that have become available in the past few months.

The turning point I've really noticed is that OpenStack is becoming much less of a developer's toy or interesting plaything, where new features are the most important aspect of development, and is transitioning (in the core areas at least) to a more practical enterprise-class framework, where component stability, infrastructure high-availability, and robust support options are necessary.  This was reflected in the composition of the audience for the event, which included many more "enterprise architect" types than have been present in the past.  It was also reflected in the questions asked during the panel sessions, which often seemed focused as much on business & organisation as on implementation and technicalities.

So OpenStack is definitely ready for the enterprise, yet as with any complex system and organisational change, there are a few considerations to bear in mind (and many of these are relevant, regardless of the private cloud infrastructure software to be used):

  • Implementing cloud computing is an organisational as well as operational change, and should be handled carefully
  • Successful cloud implementations rely on an existing IT operations mindset that includes automation & policy-driven deployment (see my previous article).
  • Executive-level sponsorship (ideally CIO-level or more) is required to effectively marshall all the disparate IT disciplines towards making cloud deployment a success
  • Integration with the lines of business is critical – internal customers of the private cloud have to agree to use the standardised cloud resources, rather than expecting bespoke IT services at cloud prices.
  • Not every workload is suitable for cloud deployment right now, as applications really need to be well suited to that kind of environment; this is OK – it's all about using the right tool for the job, so it's best not to try to force the issue and encounter failure.
  • The OpenStack control plane must provide High Availability (99.999%+ uptime). Without a highly-available control plane, access to the cloud resources disappears, and even cloud-aware services can fail.
  • OpenStack deployment and management can be difficult without the right tools and support, so it makes sense to work with an open source infrastructure software vendor (such as SUSE) to provide the technology, integration, support and training necessary to build up your own capabilities.

There is a growing number of enterprises (including PayPal and BMW) that have adopted OpenStack as a significant part of their IT infrastructure, and this trend will continue as the framework becomes more mature, and as the vendors and other members of the OpenStack Foundation build the body of knowledge in terms of documentation and training.

On that note – SUSE is hiring. At the time of writing there are at least 13 roles open at SUSE related to OpenStack & Cloud. Check https://suse.com/jobs for details.