A few things stand out to me.

The idea that management has that the team commits to a set of work is inconsistent with the latest versions of the Scrum Guide. The word "commit" or "commitment" is only used twice in the most recent (November 2017) version of the Scrum Guide - once when listing the Scrum Values and once to indicate that "people personally commit to achieving the goals of the Scrum Team".

The idea of goals is important to Scrum. Not only do organizations and teams have broad goals, but in Scrum, each Sprint has a Sprint Goal that is defined at Sprint Planning as a collaboration between the Product Owner and the Development Team. The Sprint Goal is met by implementing items from the Product Backlog, but it is not simply "finish this body of work" and it often doesn't represent the complete Sprint Backlog. That is, you should be able to achieve the Sprint Goal without completing every single Product Backlog Item selected for the Sprint or every single item in the Sprint Backlog. My current thinking is that the work needed to accomplish the Sprint Goal should be somewhere around 60-70% of your team's capacity, however you account for capacity. Different organizations will be different, though, but that's likely to be a good starting point.

Working overtime and weekends is also an anti-Agile Software Development practice. One of the underlying principles is that all stakeholders of an effort are able to work a sustainable pace. Long days and weekends, even if they were paid, is not sustainable for a team.

At this point, there are a few next steps. The team's Scrum Master should be educating management on the values and principles of Scrum and Agile Software Development (such as "commitment" and "sustainable pace"). The team should work on its ability to forecast work and negotiate scope with the Product Owner. The team should also identify and work toward resolving or preventing the impediments that led to this situation (eliminating or reducing the impact of external dependencies).

The situation that you describe, where management requires that the team works overtime to complete all planned stories, is one of the reasons why the Scrum literature has stopped using the term "commitment." A true commitment can only be given when all uncertainty is taken out, including uncertainty about 3rd-party dependencies, how much work each item is, how much time everyone will be available taking sickness into account, etc.

One of the basic ideas of Scrum is that the team works at a sustainable pace, which essentially means working normal hours without overtime (paid or unpaid). This directly means that you are not experiencing an acceptable variation of Scrum.

What you can do about it depends on your role.

If you are a developer, then the best you can do is

• (collectively as development team) refuse to "commit" to more work than you are absolutely certain you can deliver within a sprint. This will probably be way less than what management wants you to do.


• keep focus on finishing work, rather than starting on new tasks. If you see that some work can't be completed, indicate this as early as possible so that the plans can be adjusted.

If you are a Scrum master, then you can really prove your worth by educating the management about Scrum. Some points that stand out to me:

• As stated in the first paragraph, the team does not give a commitment during the sprint planning, but they give a forecast of the work they expect to have finished.


• Although the team should avoid having unfinished work at the end of a sprint, this situation is nearly unavoidable at the beginning of a project (or after a change in the team's composition). How much work the team can realistically accomplish in a sprint can only be determined based on historical figures of the last few sprints of the team in this composition. Early in the project, such historical figures just don't exist, so a team accomplishing in the first 3 sprints exactly what the planned for each sprint is more accident than good planning. After about 5 sprints at a sustainable pace there is enough data to get a reasonable idea how much work the team can realistically finish within a sprint.

Your PM has no business being involved in your scrum.

Your PM has no business asking you for unpaid overtime.

The obvious thing to do is to estimate all tasks in such a way that you can guarantee they will be finished in time. Then you should be able to go to the pub in the second way, since clearly if underestimating a task means you finish it for free, then overestimating means you have time without work.

There are a number of aspects to this, but at a high level, yes - the PM will absolutely want to clearly understand why the planned work has not been completed. However, this should be brought up (and resolved) in the retrospective. From the dev side, there are many factors that can contribute to sprint failures.

Some things you may want to consider:

Too much in the sprint

If you are regularly committing to too much work, then sprints will fail. The sprint velocity should be tracked over time to find out what the optimal number of points (or days) is.

Resource allocation

Ensure that sprint planning adequately accounts for non-development activities like the ceremonies, holidays, training, admin, support and other projects etc. Automatically assuming everybody is developing every minute of every hour they are physically in the office is just not practical and will immediately put you on the back foot from the get go.

Estimate variation

You're doing refinement, but are there certain sorts of tasks that always overrun? Commonly these are down to missing or vague requirements. If the requirements are woolly the story should not even make it into the sprint unless it has been adequately refined or a spike has been planned.

Velocity

If the velocity is being properly tracked, the true number of stories should become clear. That isn't to say they'll always be done in time but it should make things a lot easier.

Good will

On any project, good will is limited. If you're constantly working out of hours to deliver, morale will suffer and devs will burnout - this is a project management failure. As I've already outlined, make sure sprint planning only schedules a realistic number of stories using velocity and spikes to help you along the way.

Spikes

If an item is badly refined or just plain woolly, don't be afraid to put a spike in to provide a better estimate for later sprints. Yes, some people are just bad at estimation but most of the time, the full facts just aren't known at the time. Ideally, this should have been covered in the refinement or picked up early by the PO, but sometimes they can slip through into a sprint. Developers should be pushing back on these hard as these can easily torpedo a sprint that is otherwise going well.

No, this isn't a recognized form of Agile, for one very important reason: if you're committing to deliver everything, you're not doing Agile, you're doing Waterfall - and if you think you're doing Agile instead, you're probably doing Waterfall poorly, at that. I'm sure you've heard the old saw "good, fast, cheap, pick any two," right? With software development, it's more like "all features delivered, on time, on budget, pick the first or the other two". Waterfall picks the first, and Agile picks the second two.

If you're going to be Agile, you need the flexibility to drop Stories from the Sprint that you're unable to complete on time. One possible way to do this is to ask your Product Owner to assess the stories using MoSCoW prioritization. This would involve selecting no more that 60% of the Stories (by total Story Points) as Must Haves that will be completed, 20% as Should Haves that you should complete by the project concludes and the Minimum Viable Product is released, 20% as Could Haves that might be completed if you have the time, and anything outside the scope of the current release as Won't Haves. It's also important to note that when combined, the Must Haves should have enough meat to their bones to create the Minimum Viable Product without needing to include any items from the other categories.

Determining whether or not to drop items from the Sprint Backlog is the responsibility of the Product Owner, after it has been requested by the Team, and any items that get dropped from the Sprint Backlog get assessed by the the Product Owner, and then are either dropped from the project entirely, or placed on the Project Backlog in an appropriately ranked position.

In this case, I'm guessing that your Product Owner is your Project Manager, right? It would be his job to determine which tasks to drop, so he certainly shouldn't blame you for overcommitting, since it would be his job to drop the tasks to compensate for that, and it appears that he isn't doing so.

He is correct, that there should not be any carry-over between sprints. Scrum teams having a carry-over between sprints is an anti-pattern and not something that canonical Scrum accepts as valid result.

But, his approach is not a good one.

During a sprint, team should constantly monitor work being done and if they can keep their commitment of sprint planning about finishing selected stories. If team identifies, that it cannot finish all the stories, it should meet up with PO and select a story to remove from the sprint. By doing so, everyone STOPS WORKING ON THE STORY, and focuses on getting the remaining stories done. Remember: It is always better to finish one story fully than having two stories half-finished.

The problems of external dependencies and imprecise estimation is exactly why Retrospectives exist. During Retro, the team should reflect and identify these problems. And the team should then find and implement solutions to those problems. Estimations can be made more precise by better knowing the system and plain experience. External dependencies are harder, but not impossible to fix.

Your PM (what is even PM doing on a Scrum team??) should not be allowed by Scrum Master to force you to finish all the stories. Instead, if he has complains, he should keep them for Retrospective. And even better, should be part of solving the problems that prevented the stories from being finished on time.

Is this an acceptable/common variation of Scrum I am not aware of?

No. It's completely wrong. I could maybe sympathize with paid overtime, if the PO made the mistake of giving out the estimates as facts before the sprint end, but unpaid overtime is completely ridiculous and would make me look for another job ASAP.

How do you suggest that I should act about this?

In my experience, people who are that much of the rail won't listen to logical arguments. The only way for them to see how broken it is is show, not tell. So next time when you estimate and commit, commit to the smallest amount possible. Commit to finish a small ticket by the sprint end. One you could do in a day. See how your PM reacts. Then start a discussion from there on what the system is used for and what the system should be used for.

Generally, at the start of the project, when we decide the velocity of the team, we go for a conservative (lower than usual) number considering the fact that it's a new team, it would take a little time for the team to settle down, understand each other, understand the functional and NFR requirements, etc. Basically, after the first few sprint's we optimise the team velocity and obviously the velocity will only improve from that point.

There is no point in committing a higher velocity at the beginning and stretching the team to achieve it.

One more thing is that, in a one-off scenario, where there is a delivery commitment which cannot be missed, project managers might put pressure on the team for stretching, working late and working over the weekends. But then that must be considered as an exception and not the norm of working. Working like that might increase the velocity on a short term, but in a longer term it would be counterproductive, as it could result in code quality issues, team burnout issues, unhappy team with low motivation, etc.

Commitment FAQ

Is this behavior normal?

I'm not sure.

Is it surprising?

No. Some people will always understand "commitment" to mean everything in the commitment must be completed.

Is it a good idea?

No. Agile development has sustainability as a central topic: Work only as much/long/hard as you can do indefinitely. That is a sensible idea at most times. (If your management does not accept this, they should not call their organization agile.)

What should we do?

• Explain that the sprint content is based on an estimate.
  "Estimate" means that it will only sometimes be correct -- and usually be wrong.
  When it's wrong, it is sometimes too low and sometimes too high.


• Explain it is far easier to change estimation behavior than work speed.


• Explain that when the team is punished for estimating too high, it will just estimate lower and management will lose a lot more progress that way than by pushing some of the content from one sprint to the next occasionally.

The nice thing is: Your project manager will know all these things already and recognize them as being right. It is only that in the short term one may prefer to ignore them.

I don't agree with your management team. They should not have made you work overtime to finish the sprint.

However, the idea that commitments are not possible comes from a misunderstanding of software development. I have seen many teams try to predict the stories they can complete in a sprint by the number of story points they have completed in previous sprints. If that was possible it would say that software development is linear, i.e. if I work two hours I get twice as much done as in one hour.

Software development is creative, and therefore, not linear. It is a better practice for the team to tell management what they can do in a sprint and then deliver those stories. If you are consistently missing your commitments you either had no idea of the scope of the story to begin with or you are being pressured by your product owner to take on more.

In the former case, you must make sure that you understand the scope of the story before you agree to take it on. If it is the latter, you have a culture problem and there is little that can be done.