While the position, or progress, of a software development changes all the time, with results produced and specifications evolving, its speed tends to change much more slowly. It is interesting and often useful to understand what causes that speed to change, hence how one can work with it and influence it. This in turn helps in choosing the “safer route” mentioned in the previous blog.

Many Software Metrics are fickle at best (see for instance the recent article, Lies, Damned Lies and Project Metrics (Part 1)), which reinforces the interest of tracking trends (where individual errors get aggregated and masked) rather than positions (where the only two reliably accurate positions in a software project are the starting point, when nothing is done yet, and the end point, when the project is completed).

Some productivity factors, already shown in The Mythical Man-Month are well known. They include the effort involved in the training of a newcomer to a project team, or the influence of team size on the management effort, hence on the overall productivity of the team (the “speed” mentioned above).

Individual productivity also varies while working on a project. Tracking this productivity as such -and not as compared to the initial expectations- yields useful results, such as the definition of a longer initial iteration due to various “administrative” overheads, or realistic expectations on how long a given task-implementer combination will require to reach completion.

Changes in trends can be tracked and yield useful forecasting results. Indeed the earlier one notices changes in productivity, the earlier one can work with that, by finding:

• what lowered the productivity, and correcting this cause, or
• what increased the productivity, and reinforcing the consequences.

Some changes are to be expected, such as for a newcomer in a team, where, on top of the training effort imposed on the team, it typically takes two tasks to reach full productivity. As a consequence ISIS recommends that the first two tasks assigned to a newcomer to a team be short ones rather than long ones.

This person’s productivity should show a significant jump between the second and the third task. If this jump has not occurred by the end of the third task, the Project Manager must investigate what the issue is, and in particular if:

• the consultant started with a very high productivity and should not be expected to improve that much, and this is OK, or
• the consultant started with relatively low productivity and
• there’s little hope of improvement (and an alternate team member may have to be found), or
• improvement is just around the corner (and the next task should also be a short one).

ISIS requires tracking effort, and encourages the tracking. beyond effort, of other important aspects in an engagement, such as:

• the top risks,
• the number of defects (fixed, investigated, identified),
• the testing coverage,
• the response time.

The ISIS Test Plan template recommends tracking the defects apparition and resolution rates. These rates and their trends allow the PM to known well enough in advance when to expect the completion of each test phase.

As another example, ISIS has a pretty complete list of standard risks that apply to projects that ILOG PS is involved in, and gives ways (as well as initial estimates) to:

• Avoid,
• Transfer
• Mitigate, or
• Accept these.

Tracking these risks, or at least the most influential ones (ISIS figuratively speaks of “the top ten risks”, with practical advice on how to identify the top risks), gives an important edge to the Project Manager as this helps pointing out both those risks which don’t move and those that keep increasing, hence should be met before they become overly expensive, be it in terms of time or development resources. This will be furhter developped in another blog.

A useful spreadsheet-based effort tracking template can be found in ISIS, its data entry sheet is shown below, followed by the corresponding graph.

This templates is filled in up to time interval 4, which the project had just reached. The corresponding graph gives:

The trends in the above graph show an expected end date for interval 10, at the intersection of the yellow (effort spent) and orange (work to be done) curves, with an ending effort of 78 instead of the original 61 or current 73.

In brief, three recommendations from this blog:

• track trends, not just status;
• integrate a newcomer into a project with smaller tasks;
• track defetct and risks, not just deliverables and efforts.

Like on the road, check your position but your speed too, in addition to the other instruments on your dashboard!

• if you don’t know where you go, you don’t know when you get to your destination, and
• if you don’t track your progress, you don’t know which way you are going.

Early ocean goers had no accurate way of locating their position or destination, or even of tracking their progress. This is quite similar to many current software project managers. Yet these ocean voyages had a much higher rate of success (i.e. reaching their destination with most of their goods sellable and most of the crew alive) than current software projects, where it is said that in fewer than half the cases their results are actually used.

Software Project Managers (PMs) can find some very relevant lessons in the experience of early ocean seafarers. I will develop two what I see as the four most important ones in the next paragraphs:

1. Know where you want to go.
2. Track your progress, estimate where you are.
3. Track your speed, estimate where you are going.
4. Choose the best route.

I’ll skip for now points 1 and 4, leaving them for a later discussion and focus here on the tracking part of a software project.

Track your progress, estimate your position.

Both the time it took you to reach a certain point, and the location of this point count. Project tracking is naturally about time and expenditures, and number of remaining bugs, but also about delivered results.  

As shown in the above screen shot, ISIS encourages the “earned value” tracking method [1] that gives good location estimates by focusing on what is actually delivered to the customer (the green curve above, top graph) , while the ISIS-required properly filled time-sheets tell the project manager where the effort was spent (the red curve above, top graph) .

Tracking of efforts and actual deliveries give Project Managers an approximate snapshot of the project’s position. Yet PMs typically have a difficult time estimating what work is required to finish. They need to get help from the developers who should report the efforts spent, and what they see as the remaining effort, whether it was planned initially or not (“remains to be done”).

Track your speed, estimate where you’re going.

Project Managers can fairly easily tell their approximate position (based on what deliveries have been accepted by the customer on one hand and the time-sheets on the other); but they usually don’t take trends into account. Back to the initial analogy, being next to a shoal and drifting towards it is very different from being next to a shoal and drifting away from it!

ISIS recommends tracking position over time, e.g. through the integration of successive snapshots. This helps getting an accurate estimate of the delivery speed, thus a good estimate of how long the remaining work in the project will take. This is a much better estimate than using the current position snapshot and betting the house that the rest of the project will go according to some initial plan.

Similarly ISIS recommends tracking the “speed” rather than snapshots of bugs and issues, as this helps predict when a test phase finishes.

The following graph (lifted from the ISIS documentation) gives a hint of how such a dynamic tracking can work:

Using the above graph, with snapshots are taken at each vertical line, a project manager could have a good idea of the overall project effort and duration by the 3^rd snapshot, predicting an end of project around the 12^th time interval, rather than either the 5^th as was the initial prediction, or the 6^th as the 3^rd snapshot would have.

A future blog will develop the “Track your speed” aspects, and some of their unexpected consequences on effective mechanics of project management. Please send us your comments and suggestions!

Informs Practice is one of two annual conferences of Informs, the Institute for Operations Research and the Management Sciences. A group extremely vested in building strategic decisioning systems with thousands of practioners around the world, definitely worth spending a couple of days with. What impressed me the most is the ability of this community to address business needs with smart, pragmatic, mathematical and computer-based solutions. A great example of the benefits of Business Analysis and a great lesson or inspiration for IT as a whole.

My goal was to leverage the experience we gained building both optimization and rule-based systems to generalize some fundamental best practices applicable to the implementation of decisioning systems in general, what ever technology is involved. When you look at such systems, here are some common characteristics which come to mind: 

Yes, some decision processes are non deterministic and a decision made through different paths. In other situations, some users change their mind in the way they make decisions and want a computer to assist them in the process. That makes decisioning systems hard to test. In some situations, decisions are made on intangible elements, or intuition, a challenge for our usual way to model things and implement heuristics. Or the human brain is just too powerful and outperform the computer (e.g. visual pattern recognition, optimized way to allocate resources developed over years of experience). Some decisions are not black and white, some are based on fuzzy information, some others are really complex either in terms of number of variables involved or the type of search algorithm. Some users or policies require the decisions to be explained or traced. Some rules are unwritten. And to complicate things even further, decisioning systems are usually involved in processes which are subject to frequent change, for instance a change of policies (internal or external), strategy, organization or business model.

Such qualifiers clearly would not apply to a standard accounting or record-management application such as a CRM system, or an embedded real-time program such as in an ATM or your car ABS.

Although technology can help addressing many of these above challenges, and object-oriented programming in particular, it is not the only solution. Project Management has also to adapt and leverage new paradigms to make sure these projects can cope with so many potential hurdles.

Here are our top-5 picks in ISIS which we will develop further in subsequent posts:

1. Iterative development. Ultimately, you want to put a working system in the hands of your end-users every 4 to 6 weeks. So they can provide early feedback to avoid late discoveries of misunderstood needs or help uncovering mismatch with changing requirements. This is the key concept of the Unified Process, as well as the Agile Manifesto. An approach which is unfortunately not natural in the IT industry yet, but clearly a plus to handle the challenges identified in the above picture.
2. Focus on Time To Profit. It is key to define iterations which each bring some value to the business to anchor the project on the right track, get acceptance and traction from the business and solidify the business case of the solution under development. Business-driven iterations as opposed to purely technical and/or contractual releases or builds like we too often see in classical project life cycle such as the waterfall approach.
3. Intelligent tracking. Most of the IT projects get in trouble because of non realistic tracking. Weeks after weeks, you hear the usual “I’m 80% done” and the percentage keeps the same… Even more so with decisioning systems, estimates need to get revised to account for changes and implementation challenges. Not only you need realistic “pictures” (a project status at a given time), but you need to juxtapose these pictures to make a “movie” and highlight trends which will help you resetting expectations or reevaluating the scope and objectives. In ISIS we have added charts to our weekly status report to explicit potential drifting in workload estimate or track the quality of a deliverable through the evolution of issues found, analyzed and fixed.
4. Project governance. Governance has definitely got a lot of spotlight since the Enron debacle, and a lot of negative connotations for many people since then. However, increased scrutiny surely helped businesses avoid costly mistakes. There are all sorts of governance (e.g. corporate governance, IT governance, SOA governance, rule governance); the project governance should be seen as a positive oversight, an assistance to the project manager to maximize the chance of success. In particular, a steering committee can help the project manager negotiate changes with the business or suggest mitigation actions to address implementation-related issues. ISIS provides checklists and guidelines to make such a governance process work for all the stakeholders.
5. Risk management. We sometimes hear “if we only knew what we know” and this is indeed the title of a great book on knowledge management. Unfortunately, we actually know much more than we want to admit when we start an implementation. For instance, in ISIS, we have a tool to evaluate 193 pre-identified risks and 48 classical project management mistakes. Needless to say, with such a long list, the saying should become “if we had known what we knew then…” Definitely worth spending time in a fair, transparent and upfront risk analysis.

These are only 5 of many foundations of ISIS but key ones to ensure success. As we have seen over the past 20 years and learnt from hundred of projects. They do require quite some discipline though, but the rewards of getting a decisioning system used for many years make them worth the effort!