What do German World War II submarines and modern software development have in common? Surprisingly, quite a lot: Overengineering, ignoring testing, poor modularity, and cascading failures, to name a few.
How could a 4 million Reichsmarks ℛ︁ℳ︁ (between 50 and 100 million $ today) high-tech Nazi Submarine U-1206 fail so catastrophically by something simple as flushing the toilet? This sounds eerily similar to modern incidents like a faulty configuration file that crashed ~8.5 Million computers worldwide, causing global disruption of critical services.
The Overengineered Submarine Toilets of World War II
Major submarine forces in World War 2 had vastly different approaches to the design and engineering of submarines. The Imperial Japanese Navy focused on innovative designs of submarines for specialized missions. The main goal of U.S. submarine design was practicality and general-purpose vessels. The Germans, with similar goals as Americans, had a slight technological advantage but used, in some cases, unnecessary complex solutions.
A great example of differences in design was toilets. U.S. Submarines had simple septic tanks that could be used while submerged. However, septic tanks could only be flushed to the sea on the surface. Germans decided to push the sewage directly into the sea whenever someone flushed the toilet. By not having septic tanks, Germans saved space and weight but had to make the flushing mechanism much more complex. For much of the war, Germans lacked the technology to flush toilets while submerged, forcing crews to use buckets instead. To mitigate the problem, engineers created an even more complex flushing mechanism that could be used underwater. More complexity meant that it required specialized crewmembers and proper training for the toilet flushing.
Modern military submarines use septic tanks that could be flushed into the sea while submerged. Germans knew how to build toilet systems akin to modern submarines, with a small septic tank that could be flushed periodically even while submerged, but they were too stuck in their way of thinking.
U-1206 Toilet Flushing Incident
U-1206, German Type VIIC, was one of the later produced submarines. During the maiden combat voyage, it sunk after a series of accidents that started (allegedly) after Kapitänleutnant Karl-Adolf Schlitt misused its high-tech toilet.
Timeline of the incident:
- Captain, without proper training, attempted to flush the toilet.
- A trained crewmember tried to complete the flushing process but instead flooded the toilet compartment with sewage and seawater.
- Due to poor sealing, water breached in the battery room, located bellow the toilet.
- Batteries reacted chemically with seawater and produced chlorine gas.
- Chlorine gas leaked outside the battery compartment and spread through the submarine.
- With little choice, the crew urgently surfaced and even ditched the torpedoes.
- British Air Patrols discovered the submarine and attacked it.
- The crew scuttled the submarine.
- British forces captured the crew, just a few weeks before the end of the war in Europe.
Lessons in Software Development
Here are a few lessons I have found most interesting:
Lesson 1: A trivial Task may not be so Trivial after all - Stakeholders, please listen to your engineers. There could be alternatives.
Thinking about what we need and making it are two very different things. I can see important generals and military leaders discussing it with engineers:
- But why is it so hard to make a flushing toilet? American Swines have it!
- Americans use utterly different concepts. They do not flush the sewage while submerged.
- Nein! I expect a working prototype in 8 sprints!
- But...
- Mutter Deutschland and the board of directors expect results!
I may not be that kind of engineer, but having to run complex machinery every time you flush the toilet is unnecessary. You could combine a few "packets" and send them as a group. You would not have to have a large septic tank, only a small one. Even if a septic tank were just for ten "flushes," it would still be much more efficient than calling a specialized crewmember whenever you go to the toilet.
There were several paths to a goal, but somehow, they chose the worst one. How often do we introduce unnecessary complexity in software because we try to do the work the same way we have always done?
Lesson 2: Test in Real-World Conditions - It works in my laboratory
Even a minor mistake on war vessels can mean death and a failed mission. If something works during development, it does not mean it will work during military operations.
Interestingly, complex hardware worked fine in this case, but the user interface failed miserably. How many flushes a day are necessary on a submarine with 50 people? Even the conservative estimation with strict rations is 50-100 flushes or 2-4 per hour. It did not make sense to make the system as complex and require dedicated trained crewmembers for this task.
Deepwater high-pressure toilet was a marvel of engineering, but it was not a finished product that could be used on military submarines during wartime.
Lesson 3: Cascading Failures
Cascading failures are interesting because failures are not isolated but are intertwined mess of several failures that would not happen isolated or would not be as big a problem.
No matter how much monitoring, sound engineering practices, or powerful tools we have, cascading failures are unavoidable (given enough time).
Ironically, the better the system, the larger the chance of cascading failure. With bad engineering, even more minor errors would bring the system down, but it takes several connected problems to break the process into a good one.
Sometimes s*** happens (pun intended). We must be prepared for application problems, even if we take all possible precautions.
Lesson 4: Modularity
The water should have never reached the battery compartment, even if the toilets were flooded. Even if the battery compartment was flooded and full of chlorine, the poisonous gas should not have been able to circulate through the entire submarine. Just by having strict modularity, accidents could have been avoided.
In naval vessels, modularity plays a crucial role in accidents where there is a danger of sinking. Strongly dived sections also help with structural integrity.
Modularity plays a more significant role in software. The source code is organized through modules with good boundaries, which makes development, maintenance, and updating easier. Developers are only focused on a tiny part, so there is no need to "jump" through the entire source code. Software organized in modules gives the ability to reuse modules in other projects.
Here are a few advantages of modular software I find most important:
- Scalability - Independent modules give us the ability to scale independently.
- Parallel Development - Different teams can work in parallel on the same software but on different modules.
- Testing - Modules can and should be tested independently. It is much easier to test modules in isolation, than as a part of the system.
Conclusions
Like the engineers of U-1206, software developers must avoid complexity for complexity's sake. We may be more than 80 years apart and work in different fields, but we, software engineers, make surprisingly similar mistakes to submarine builders. It's in every engineer's nature to seek intricate solutions, but we must always strive for elegant simplicity.
Ultimately, the most stable systems, whether in submarines or software, prioritize simplicity and efficiency over complexity.