Copyright
Avancier Limited. All rights
reserved.
Previous sections have showed there are practical and circumstantial reasons why agile principles are difficult to achieve in practice. The psychology of the players may not be suitable. The politics of the employment situation can make agility impossible. In all these cases, one looks to agilists for specific advice on how to deal with the challenges.
The trouble is that agilists irritatingly tend to retreat from these very real challenges into making trite assertions about successful projects, or they reduce agility to questionable principles such as “select your own practices” or narrow points about “high communication” that can lead people away from the proper disciplines of an agile method.
This section distills several months of occasional and fragmented email exchanges (in an international data management discussion list) into a relatively coherent discussion thread. It had been suggested that data management people are ignorant of or resistant to agile methods and out of touch with developers. The consequent arguments tended to generate more heat than light. I tried to help the chief agilist to reach an accommodation with data managers, to review their issues, and explore the possible reasons why agilists don’t get the response they want from data managers. I do not believe most data managers are opposed to agile programming or its principles. They are simply not as interested as agilists want them to be, for several reasons.
GB: Many agilists focus on "soft" people/cultural issues. They put their emphasis and people, communication and cultural issues. Many good project managers do the same. And when telling the story of a project success or failure, they focus on issues surrounding culture, relationships and team spirit.
That’s OK up to a point. But the cause and effect relationships are obscure and complex. Team spirit tends to be good when the requirements are clear, the estimate is adequate and you aren’t threatened by overwhelming technical challenges. Technical factors remain critical, and pressures from them can destroy the best culture, relationships and team spirit.
Technical factors are many and varied. E.g. the usual agile principles are not so immediately helpful when developing a system that is:
·
legacy replacement, must work right on day one
·
very large, very complex business rules
·
lots of batch processing, little or no user interface.
Agilist: People are writing about many of these things. Go poking around www.Agilealliance.org. One thing to consider is that you've listed technical issues, yet agile focuses on "soft" people/cultural issues. In short, the factors you've just listed are orthogonal to what you're exploring. I'd be looking for "soft issues" where agile doesn't work instead, the technical issues aren't going to do it for you.
GB: You have to include some technical factors in your selection criteria when and how to use an agile method. Not least because technical factors influence people factors.
A system that process serial files has no end users to engage in the project - that is a very people-oriented consideration. In a legacy replacement project, it may be impossible to install an intermediate release that customers are willing or able to test the acceptance of. That's another people-oriented consideration.
Agilist: This is why you need to include stake holders, not just end users (a subset), in the process. If you're building a system then surely there are stake holders, if not, why are you building it? Why can't they acceptance test a batch job?
GB: You can't roll out a half-finished batch program suite in the way you can publish a half-finished web page. Customers cannot usually acceptance test the suite on an ongoing basis, in JAD workshops; they usually have to wait until the programming is complete.
Agilist: Yes, you might not be able to roll out a half finished batch program. However, you could still very easily develop it incrementally and iteratively.
GB: In practice, it is harder to iteratively develop batch programs than UI-centric programs where end users can watch the design they are later to work with. To test a partial batch job, you may have to devise tests that are useless for the completed job.
But my main point here is different. Ongoing customer acceptance is a key to agile development, yet it is hard enough to get a few potential users to look at screen designs! Customers are usually unable or unwilling to do acceptance testing on a half-finished batch job.
Perhaps your comment on technical v. people-centric approaches explains why you are fighting in cross-wired debates with people in this DM list. Obviously, we have to get the technical things and the people things right. The people things are easier to talk about, more socially engaging. But rather too many software project managers have come regard people issues as their whole job. They avoid grappling with more intellectually demanding technical issues, including the need to structure their project life cycle with a modicum of up front analysis and design and some reference to an enterprise architecture.
Agilist:
Yes.
GB: Most of us (even data managers) agree that an agile approach has a place in software development. The questions are about defining that place.
Agilist: Barry Boehm and Richard Turner's latest book about Risk-based software development is pretty good. They give some advice for when to use agile stuff and when not to, but more importantly they conclude that you need to mix and match appropriately.
GB: Thanks for the reference, though I am content with agile method selection criteria we documented some years ago.
Agilist: It's worth checking out. They even include some pretty good decision flow charts that companies could modify for their own unique needs.
GB: I suggest secondly: you agree with data managers that an agile approach has to be tempered in some situations, especially where the system might otherwise duplicate data storage done by existing systems.
Agilist: In previous posts you seemed to assume that agile developers will automatically duplicate data storage.
GB: I don't assume agile projects automatically duplicate data storage any more than they automatically duplicate process on different data, though I see both happening. I do believe that agile methods increase these risks. The macro-economic argument is that agile methodology needs to be tempered by enterprise-wide disciplines designed to prevent duplication of effort and software. Not an original thought, people have been saying it for years. And it is a raison detre for many data managers; so of course they bridle when this perspective is belittled.
Agilist: It seems to me that far too many data professionals are looking for an excuse not to be agile. This was the same sort of thing I saw a decade ago with object orientation. Gut feel tells me that agile is here to stay, so the data folks might not want to make the same mistake again.
GB: That may be true of the enterprise context. Most in a service provider context want to be agile, few think the agile paradigm is going away. What I see is the reverse - people using agility as an excuse to avoid proper professional discipline.
Much if not most development is now outsourced to suppliers where the pressure to cut costs and meet deadlines is intense. Many projects are reduced to something akin to frantic XP, with too little analysis and design up front. Too many customers lack sensible enterprise-wide disciplines. Too many managers expect developers to start coding a prototype on day 1 of the project, then expect that prototype to be viable as the basis of the final system.
I suggest thirdly: you agree that agile principles are not so immediately helpful when developing a system that is hindered by the technical factors mentioned above. You have to include some technical factors in your selection criteria for use of agile methods - though the ones in my wee example are hardly very technical.
GB: It seems to me that your approach is antagonistic to data managers. They find it hard to welcome somebody coming in at right angles telling them to communicate better, hold stand-up meetings every morning, refactor their databases or whatever.
Agilist: If I'm coming in at "right angles" then that is very likely more a reflection that they haven't kept up over the years. I've written several times that the "split" between the object and data communities in the late 80s and early 90s have hurt both communities -- the object folks aren't as up to speed on enterprise issues as data folks are (IMHO), and the data folks clearly aren't up to speed on evolutionary (iterative and incremental) development techniques. Both groups need to work together more effectively and learn the techniques of the other. I suspect the data community has a much harder learning curve to overcome.
GB: Nevertheless, DM list member Mike Gorman does agile development. List member Charles Betz supports agile principles. I do too. Many of us do. Probably most of us do. The question here is why you aren’t getting the interest you want.
Agilist: Why do so few data-oriented publications discuss agile processes? Why is there almost no conversation, at least positive conversation, within the agile community regarding the role of data professionals? Why is there a virtual absence of agility talks at data conferences and similarly and absence of data talks at agile and/or development conferences?
GB: Even though some of us advocate all the agile principles in their place, I can suggest several reasons why people in this list don't show the *interest* in talking about agility that you want them to. I mention a few below and more in the next section.
Some DMs have been working for years to establish in an enterprise some respect for shared data definitions, some recognition of the long term value of imposing disciplines on project data models and database schemas. They resent this being undermined when enterprise managers are deflected from this by the promotion of agile methods, following which small empowered teams make their own decisions, leading to duplicate data and/or duplicate processing. If this is the future, DM fear for their jobs.
Some DMs have chosen the DBA corner for a quiet life. They have found technology that changes only slowly and a place they can organise and get under control. They thought they would always be needed. You frighten them by telling them they are no good, too slow, and will be replaced by agile team members sharing work around.
Some DMs remain suspicious that OOPLs are simply not a good tool for database processing. They resent agile methods riding along with the bandwagon of OO developers who know nothing but this paradigm, and defer only to the OOPL gurus.
GB: Many good data management folks feel they are the last bastion of enterprise-wide sense, fighting against the dominant forces of head-down rapid application development. You can't blame some of them for finding your digs at them so provocative.
Agilist: Perhaps they need to stop fighting it and start finding ways to become and effective part of it? My writings at www.Agiledata.org should provide some insights for them.
Veterans do get irritated irritated when agilists claim bragging rights to well-known principles already embedded in structured methods, while also decrying those methods.
e.g. The importance of engaging people, communication and feedback is the territory of conventional project management methodology.
e.g.
Something akin to "user involvement is paramount throughout" was on
the first page of the
Agilist: I'd be very surprised to discover any agilist claiming that the ideas of people working together are new.
GB: I don't say they claim it is a new idea. They do however imply it is missing from the non-Agile methods people are familiar with. That is untrue. And the agile pitch often starts with an attack on a mythical waterfall method that I don't recognise. That is irritating.
But my bigger point here is the ho-hum factor of motherhood and apple pie. Managers are often veterans who have seen several software wheels turn several times. They resent agilist announcing their expose of issue X, when we know issue X was documented and dealt with ten, twenty, thirty years ago. Repetition can irritate people to the point where they find fault with the obvious, just for the hell of it.
GB: While many methodologies are reducible to a smallish set of SBOs, it does seem that agile methodology gurus are especially prone to start and finish their sales pitch with SBOs. Real project challenges often to start beyond the obvious.
Agilist: I suggest that you take some time to actually understand the agile movement before you make clearly inaccurate claims. [Do] you actually want to learn about agility? Right now you seem to be at the "hard end" of the learning curve, albeit further along than most within the data community.
GB: My claims are accurate. I won’t embarrass you by quoting SBOs from your own agile discussion list. I have listened to other leading agilists speak. One of them was very much in SBO territory; the other very close. And I receive a regular agile methods circular which is so full of SBOs I can’t bear to read it. Let me quote from the most recent circular: "I predict that 2004 will be the year of agile management <snip> I also predict that changing the project management culture in many organisations will be even more difficult than changing that of software development groups."
Following experience of DSDM, which started here around 1995, I know people been saying the last sentence above for at least five years. So predicting it for 2004 is a SBO and safe bet!
Agilist:
DSDM, a very good process, hasn't gotten much traction elsewhere [outside the
GB: Deeper than that, we have been saying that agile/iterative development is harsh discipline, and it frightens project managers. That is what the agilists have to confront PMs with, and address.
Agilist: Yes. Many PMs have been trained in spectacularly ineffective and often bureaucratic techniques. It'll be difficult to get them to change their ways. I was recently at a client where I was asked to speak to over 100 project and senior managers in their IT organisation. It was very revealing how many issues they were struggling with and how much trouble they were in. The disconnect between management and successful development was spectacular. The good news [for me] is that situations like this make for good consulting opportunities.
GB: My aim in this context is to offer explanations for why people get irritated. I do agree that the agile manifesto’s SBOs are profoundly important, but they are also trite, easily understood, and rather boring to the kind of person who wants to spend their weekend reading emails about data management.
Let me say more generally that what makes any methodology interesting is the statements it makes that are *not obvious*. The interesting thing is what an agilist recommends where the normal SBOs don't readily apply. That’s where the agilist has something valuable to offer.
·
Specific advice on how to iterative development of a batch program
suite would be *far* more interesting and valuable than how to do on for screen
design.
·
Specific advice on how to speed up database structure changes would be
*far* more interesting and valuable than classifications of what kinds of
changes might be made.
There are other difficult and contrary cases for the agile methods guru to address, rather than stick in the comfort zone of the happy project where agile principles are readily applied.
GB: Some methodologists and gurus stay in the comfort zone of the happy project where the preconditions for success are readily established. They simply won’t step up to challenges outside their comfort zone. e.g. The mission is a large fixed-price legacy-replacement project. e.g. The project is mostly batch programming.
You can't roll out a half-finished batch program suite in the way you can publish a half-finished web page. Customers cannot usually acceptance test the suite on an ongoing basis, in JAD workshops; they usually have to wait until the programming is complete.
Agilist: Yes, you might not be able to roll out a half finished batch program. However, you could still very easily develop it incrementally and iteratively.
GB: In practice, it is harder to iteratively develop batch programs than UI-centric programs where end users can watch the design they are later to work with. To test a partial batch job, you may have to devise tests that are useless for the completed job.
But my main point here is different. The interesting thing is what an agilist recommends where the normal SBOs don't readily apply. That’s where the agilist has something valuable to offer. There are many difficult and contrary cases for the agile methods guru to address, rather than stick in the comfort zone of the happy project where agile manifesto principles are readily applied.
Agilist: Data architects should get out of their ivory towers and communicate with developers more.
GB: True, but data architects need more specific advice than that. What made Mike Gorman's recent piece on running data workshops interesting? He didn't just say "hold workshops that engage players from different applications". That would not be interesting or worth a consultancy fee. He proposed exactly how to do that using a specific analysis methodology and specific tools. Whether Mike’s methodology can rightly be called agile or not is of little interest to most people. They want very specific advice.
GB: Every well-known methodology shares the majority of its principles with other methodologies. What makes a methodology interesting is the statements it makes that are different.
As an agile method grows to embrace some enterprise-wide discipline, so it moves away from the agile end of the spectrum and becomes a compromise middle of the road methodology. The middle of the road is where most of us spend most of our time. When a guru claims to have advanced their extreme methodology by moving it closer to the middle ground, where we already sit, that may raise a wry smile, but no excitement; it tends to confirm the view of people who thought the guru/method was misguided in the first place. e.g. Ed Yourdon's reputation never recovered around here from saying something like "the DFD is dead".
GB: By the way, I agree that agile teams 'blocking' corporate interference (including enterprise data architects) can sometimes be ethical. But I hope we can keep to solid ground, to discussion of short term v. long term costs, short term v. long term benefits and micro economics v. macro economics.
An argument that says “my approach is right on ethical grounds” is at best sanctimonious and irritating, at worst a kind of political correctness, a blackmail applied where the economic argument cannot be made with conviction.
Agilist: You do have to agree with list member Charles Betz though, that there are ethical issues with the concept of blocking. If you've exhausted all other possibilities then you find yourself in a situation where you have to either walk away from the organisation, block the people, or have them forcibly removed from the organisation. In short, you end up with an ethical issue either way.
GB:
The economic case for and against off-shore programming is interesting, and
could go either way. But the ethical case for not exporting US programming jobs
to
Agilist:
Exactly. All
GB: You don't give up do you? Actually, it could be one way to create a clearer requirement for on-shore data architects!
Why do managers get irritated by agilists? When faced with challenges of the kind above, agilists tend to backtrack into mushy generalizations and platitudes. The mushy middle ground is a swamp; if we start there, we may never reach the high ground of a truly agile project. And agilists do stray from their own disciplines into the mire.
I am a fan of simplicity in a methodology, but not to the point of vacuousness. I am fan of clarity, but not to the point of SBOs (statements of the bleeding obvious). Above all I am a fan of stating what the disciplines of a methodology are, what its limits are, what situations it does not handle so well, what the trade offs are. If the agile manifesto gave more space to these things, it would be immediately more persuasive.
Never try to present agile as the answer to every question, even if you think it is. You will only provoke people to prove it isn’t.
Let us try to preserve a common sense of what it means for a project to be agile, lest in trying to present agile principles as the solution to every issue in enterprise system development, at every stage, we undermine the very values and characteristics that can make a highly agile project successful.