Tuesday, 30 March 2010

Integration Consultants

Through Sporting Solutions, our B2B subsidiary, we provide a product that we believe is of significant value to our partners. But of course we'd say that, right? We also believe - as did Drucker - that value is not what you put into your product, it is what your customer takes out of it. Therefore we appreciate that how valuable our product actually is to any given partner is entirely down to how they use it.

Data from our API could be put to so many uses - from making a fairly simple ticker application which scrolls upcoming sports events across an operational screen through to automating the complete bookmaking lifecycle; setting up events, updating website content, pricing and probabilities in real time, and resulting after events. Obviously my examples here represent two opposite ends of the value continuum from a customers perspective - same feed, same data, very different business returns.

With that in mind, we're introducing a new role into our organization - integration consultant - with a presales and postsales remit to help our customers get maximum value out of our product by analysing their existing systems, planning their use of our feeds, and seeing their integration project through, all side-by-side with their own engineers. Free of charge, anytime you need it.

We want our customers to love our product and to stick with it, and helping them to get the most out of it seems a lot like a shortcut to that to me. Another first in the online gambling world I believe.

Saturday, 20 March 2010

Why don't we have a common data format?

I've written before about the changes we're going through as an industry and the fact that we don't have a common data format - or even consistent terminology for describing things like events, games, markets, etc - is a constant barrier to exchanging information between organizations.

More operators are becoming both producers of their own B2B product and consumers of other operators B2B products and this is pushing up the need for interoperability. It's difficult enough during initial discussions to clarify what both parties mean by fixture (are we talking about a betting opportunity withing a game or the sporting match itself?) and - although it might seem like pedantry on the surface - these things matter when customers are trying to get a clear idea of what they're buying and what they're committing to in a contract.

And then, when we move on to implementation, the fun really begins. Because all our schemas are unique, closing that gap becomes an integration problem. Typically this is overcome with a mapping service or parsing raw data and inserting it directly into a target database. Whatever the approach it is usually a little time consuming and, due to the theoretically unlimited number of variations, requires a bit of ongoing manual intervention. There are only so many options for joining together dissimilar data structures.

Many other industries have been through this journey, and have established some common formats to make exchanging data easier. At Sporting Index we've taken SportsML - used by many news and content organizations as an open format for exchanging data about sports - and make some extensions to it to include e-gaming type information.

As a B2B operator in any industry you always want some standardization in this area - anything else is simply a barrier to the adoption of your product. Of course as an incumbent data provider proprietary protocols and schemas act as a barrier to your customers adopting someone else's product instead but, if you are confident in your product and you provide a good service, why would this be an issue for you?

Wednesday, 17 March 2010

Expressing Non-Functional Requirements

NFRs have recently been a topic of debate in our hallowed halls and, as usually happens in most good technical teams, there isn't any shortage of ideas, good patterns, and approaches to quality - just a lingering feeling that life would somehow be better if we had a more consistent and explicit definition of done.

A few years ago I wrote a short series on what I called The 6 Principles of Webscale Computing and, now that I revisit them older and [depending on whom you ask] wiser, I can see a little room for a bit of a brush up. Those key elements which I hold most dear are:

1 - Scalability
2 - Maintainability
3 - Quality
4 - Security
5 - Availability
6 - Usability

What I found particularly effective was documenting NFRs as user stories; this let us express them in a format that developers are already familiar with and make them a bit more specific and actionable than just 'do a good job'. Using the as-proven-by story clauses allowed that extra degree of flexibility which good engineers will use to make interpretations appropriate to the project they're working on - deftly sidestepping the joint pitfalls of waste (gold plating little used or less critical components) and undercooking (not putting enough in where it counts most). And, after all, a non-functional requirement is still just a requirement and if you want something done in software development there is no better way than to 'require' it.

Over the coming few weeks I'll be dusting those old stories off, giving them a bit of a polish, putting them back up here and - to strike terror into the hearts of my team - turning up to their demos to see them implemented in practice!

Monday, 1 March 2010

Where are the brown M&Ms?

If a picture is worth a thousand words, what value would you put on a bowl of M&Ms?

An article in this month's Fast Company tells us about how Van Halen used to have a clause, buried deep in the contracts with concert venues, that stipulated that the venue would forfeit all costs if there were _ANY_ brown M&Ms backstage. At the time many people thought that clause 126 was simply the ultimate example of rock excess. In reality it was there for a very specific purpose.

The contracts were very, very, very detailed and a successful concert meant that everything had to be in place and working. The M&M clause was a simple, visual feedback indicator that showed whether the organiser had actually read, and fulfilled, the contract. If there were brown M&Ms backstage then that would be a tell tale sign that a venue had not fully read the contract. The group would make sure that a thorough examination of the venue was carried out. Invariably they would find a major problem that could have derailed the concert.

You don't always need thick, verbose, progress reports to know if things are on track. Often, all you really need, is a simple, visual indicator of potential trouble.

Sunday, 28 February 2010

The Emerging B2B Market

It seems like everyone is getting into the online gambling B2B space at the moment - it feels like every couple of months another operator brings a white label or a feed to the market. This is a trend which has repeated itself many times before in other industries, and web's unique properties as an integration medium between organizations simply accelerates the process.

Over the next few years the gaming industry is going to polarise into marketing-led organisations which own the customer, and technically-led organizations which own the product. This type of supply chain specialization isn't new to most of us and, excluding a few exceptions that might manage to keep a foot in each camp, we'll all need to choose whether we gather around the data and the platform or the customer and the presentation.

History teaches us that, as this unfolds, the waters will be muddy for a while. When operators analyze their portfolios most will find that they have something which can be leveraged into this new B2B market as well as aspects of their business which they are better underpinning with someone else's product from that same marketplace. Add a conservative view on risk and the relatively low level of efficiency that is a property of any newish market and, for the foreseeable future anyway, most organizations will be keeping hold of capabilities which overlap with (and perhaps even have a poorly defined relationship with) external products integrated into the enterprise.

The trick to making sense of this is in the meantime is going to be clearly defined boundaries. We have many good consumers (we all know how to run a book or some games) and few good producers (not many of us have worked out how to take something highly personalized for us and package it up generically) and B2B operators are going to have to take the lead by clearly setting out the scope of their services, how they can be easily integrated, and - perhaps most importantly - what bookmakers are able to do with those services.

At Sporting Solutions we don't sell feeds or access to databases of sports content - those things are just delivery mechanisms - what we sell is automation, cost effective product expansion, and cost reduction tools. The important distinction here is defining our product relative to it's use in an operators environment. After all - we can hardly claim that we offer more cost effective solutions to the sports gambling industry if, for example, an operator has to take our feed and still keep a large contingent of manual data entry or content management staff busy.

Friday, 19 February 2010

Devops design patterns

If you are a developer of a certain age (i.e. old like Dan and I) you would have lived through the "Gang of Four" Design Patterns era in the mid '90s. When the GoF book first came out in 1995 it had a spectacular impact on the development community - it was up there with "The Internet for Dummies" in terms of the best read tech book. You couldn't move with out hitting a design pattern. They were everywhere. They were in requirement specs, architecture specs, job specs and even in people's code.

It was usually an ominous sign when you saw them in code because you could be pretty sure that you would find all the patterns, everywhere - a clear indication of buzzword bingo being played by a development team that wanted to make sure their CV was chock full of the latest Abstract Builders, Observers, Memento's, Flyweights and everyone's favourite: the Singleton. (Never has so much damage been done by one pattern as the Singleton!)

As with all meme's the design pattern concept has started to drop out of favour in recent years, which is a shame as Patterns are actually a very powerful metaphor for describing systems.

The originator of the design pattern concept was Christopher Alexander. He was an architect who was interested in creating a pattern language that would help people to build houses and communities of any scale. Design patterns in the technology community have largely been restricted to developers and it is rare to find systematic use of patterns to describe other areas of engineering life. At the risk of getting shouted at by Doug, I have never, for example, heard of SysAdmin Design Patterns. Why is that?

I was therefore interested when I found a post this morning on a devops blog about design patterns for operations teams. The blog is really little more than a teaser but it got me wondering whether having a "pattern language" to describe a whole range of engineering tasks, not just development code structures might be useful. Would it be useful to have patterns describing various deployment options or ways/means to monitor systems? Would a set of operational pattern languages help us in describing the tradeoffs between various failure handling scenarios? Would it help us to cross boundaries between development, platform and ops? Would being able to express things as a pattern help people from different teams to think about what they really need to get done and why?


Friday, 29 January 2010

Why Toyota's global recall is actually a wakeup for your sprint build process

There are very few truly new ideas in the world and the unfortunate fact of life is that neither you, I, nor anyone we know is even remotely likely to ever stumble across a ground breaking new concept. The mundane reality is that most ideas build on top of other, older ideas. Newton once observed that "If I have seen a little further it is by standing on the shoulders of Giants". If Newton needs a little help then what hope you or I?

If we want to improve our daily lot then one of the best things we can do is to go in search of the giants on whose shoulders rest the ideas that drive our daily life. By taking a look at these older, deeper ideas we might be able to learn an old lesson about one of our "new" ideas.

I believe that the recent global recalls that Toyota have had to make due to faulty accelerator pedals and floor mats can teach us something about the quality of our product and software development practices. First, however, we need to take a history lesson.

The roots of agile

One of the big, "new" ideas in software development over the past decade has been that of agile development. The first stirrings came with the announcement of the Agile Manifesto in February 2001. From there the ideas of pair programming, unit tests, continuous integration and so forth began to flourish and the concepts that would eventually become Scrum started to form. Although it is easy to forget now, but many of these ideas were considered heretical at the time, particularly by "properly" trained project managers i.e. those who had been brought up on waterfall style methodologies.

The underlying ideas that became the Agile Manifesto did not suddenly dawn on Kent Beck, Ward Cunningham and the rest of the signatories at that meeting in February 2001. They are based, in large part, on the work that had been going on for decades previously in manufacturing. In particular they were influenced by the lean manufacturing principles that underpin organisations like Toyota, the car manufacturer. The ideas that would eventually become the Toyota Production System had, in turn, built on the work of people like William Edwards Deming who was so influential in Japan in the years following the total and utter destruction of the country in the Second World War.

Constraints breed creativity

One of the biggest constraints for Japanese industry at the time was the almost complete lack of capital. Most machinery had either been destroyed in the fighting or confiscated. Much of the valuable foreign exchange had either been spent or was being used as war reparations. The primary question therefore was how to get hold of cash as quickly as possible? How do you improve the cash flow?

One fundamental insight was that if you could reduce the time between the order being made and the product being delivered then you would get paid quicker. That would clearly improve cash flow. If you could delay building the order until the very last minute then you would not have as much stock tied up, sitting around idly. You wouldn't have to pay for it and that would improve your cash flow. If you could reduce the amount of stock returned due to faulty production you would not have to spend time and money refixing them. That would save cash.

The aim of many of these early practitioners therefore was to build a system that allowed product and work to flow through as fast and reliably as possible and delay making decisions until the very last moment. Products were to be built "Just In Time".

Any turbulence or impediment that disrupted that flow was to be quickly removed and never allowed to re-occur. It was soon realised that one of the primary causes of turbulence in product delivery was quality (or the lack thereof) and the sooner that a quality defect could be removed the better for the overall system. The best possible solution would be if you could set up the system so that the defect never happened in the first place. Ensuring high initial quality thus became an economic imperative.

This is where the concept of Jidoka was born and for many years underpinned the inexorable growth of Toyota. The production line employees of the myriad Toyota factories were held responsible for ensuring that no defect passed their station. If they saw a problem then they had to do anything to resolve it. Ultimately they would be able to halt the entire production if they needed to.

Fail fast

To "modern" western manufacturing executives and strategists this was complete and utter madness. Have a single employee potentially stop an entire factory because of a broken widget? Think of all that wasted time and money with expensive resources standing idle! Crazy! Surely it was better to let the defect go through and try to patch it up later!

One of the advantages that Toyota had over the West was their lack of cash. This had forced them to build an entirely new type of flow based system. The western executives did not "suffer" from the lack of cash in quite the same way and thus where unable to really appreciate what they were being told. They operated in an environment where it was believed that economic efficiency arose from making maximum use of resources. If a machine or person was standing idle due to a lull in orders why not let it make more of the component and build up a buffer? This would undoubtedly be used later anyway and would provide some insurance against a down stream failure in the mean time.

What the West had not realised was that creating components that were not needed at this very moment had costs. And they were high, albeit often invisible. Unused components used up cash that could have been put to more productive uses. Production runs often took so long that requirements often changed radically before all the buffer was used up and great amounts of stock had to be abandoned or reworked.

Even more insidious was the effect on bonus and compensation structures. More was considered to be better. The goal was to maximise output irrespective of need and this often meant factories and production lines pumped out millions of faulty (and unwanted) components simply to meet targets. People felt they didn't "have time" to fix problems, even though no one was actually consuming their output.

Western economies emphasised volume over quality, the Japanese valued the inverse. This is still the case. It soon became clear that customers put real value on the quality and reliability of Japanese technology. This resulted in a huge economic boom and within decades Japan became one of the most powerful economies in the world.

One by one the existing western car manufacturers started to wilt against the on slaught of the Japanese quality machine and they disappeared or started to rack up huge losses. This eventually culminated in the massive US government bail out of the Big 3 Detroit car companies (GM, Chrysler and Ford) a couple of years ago.

Eye off the ball

Unfortunately, somewhere along the line, Toyota stumbled.Maybe it was the fact that it had become the number one global car manufacturer. Maybe it started to value growth for growths sake and began to reward people for absolute output rather than tailored to demand. Whatever the reason, it began to allow defects into its production line and failed to follow Jidoka. Quality started to suffer. Ultimately they allowed millions of cars to be produced and shipped to customers with faulty accelerator pedals and floor mats. The line was not stopped.

The cost to Toyota has been huge. 4.2 million cars were recalled in the US last October due to faulty floor mats. Last week more cars were recalled due to the faulty pedal issue. Overall 8 million cars have had to be checked in the last 4 months due to quality issues. In order to get to the bottom of the issue Toyota senior management have now physically stopped selling many of their models and have ordered a complete stop of several complete US factories until the matter is resolved.

Someone broke the build

Stopping sales and halting factory production is an awe inspiring decision. It is a recognition, admittedly belated, but very public that the economic value of quality in a flow environment is paramount.

Software and product development are flow based systems. Rather than passing widgets and gadgets down the line we are dealing with unanswered questions. Is this feature what the end user really wants? Will this patch fix the bug? Whatever the hypothesis we are trying to answer, we need to make sure that we can find the answer as soon as possible.

A failing build tells us that there is a quality issue somewhere on the production line. The flow of code has hit some turbulence and needs remedial action. What do you do in that situation? Will you call a halt to the line and make sure it is fixed immediately or will you let it roll and deal with it later? If you do not feel able or willing to stop a sprint to fix an issue will you really be willing to do the digital equivalent of shutting down entire factories and recalling several million cars later on? Somehow, I doubt that....