This is a post by Amit Aggarwal, Head of Development at BloomReach.
“Simplicity is the ultimate sophistication.” ― Leonardo da Vinci
Complexity can lead to systems that are unmaintainable, hard to test and debug, and difficult to scale. Good engineering systems, on the other hand, are simple — easy to explain, easy to test and debug and intuitive. However, simplicity should not be mistaken for “easy” or “fast.” Bloom filter is an example of a simple system — it exposes a clean and easy-to-understand interface; yet the implementation is non-trivial. MapReduce is an example of a simple programming paradigm that captures a lot of distributed computation use cases, but it is not necessarily easy to implement. As Steve Jobs said: “Simple can be harder than complex.”
One of the key goals for an engineering leader should be to tame complexity in order to allow the technology stack and team to scale over time. So what are the causes of complexity and how can engineering leaders best combat them? I’ve identified a few common problems and some ways to avoid them below:
Sources of Complexity
- Complex requirements: Complex requirements with limited value lead to unneeded complexity in the system. You need to understand which of the features that you are implementing are complex and question the value that they add to the product. Know what problems you are not trying to solve. Does your system really need to be real-time? Do you really need the user interface feature that is not necessary for the main workflow, but adds significant complexity to the backend?
- Narrow Requirements: Narrow requirements lead to inflexibility in design, future patches and downstream complexity. Good design anticipates evolving requirements (without going overboard). As an example, be wary of designing a feature that works for your current customers but doesn’t scale beyond your largest customer.
- Premature optimizations: Optimizations lead to complexity. Have good data to support them. Don’t optimize your schema without understanding your workload. Don’t introduce a caching layer without understanding the potential hit rate or the freshness requirements of your application. If it isn’t a problem, don’t optimize for it.
- Lack of engineering basics: Failing to follow good engineering/design practices, like thinking about invariants for every component, having clean interfaces and sharing code and data lead to complexity as well.
One practical way to avoid some of these complexity pitfalls is to conduct a simplicity test as you build your product. The four key simplicity tests are:
- “One sentence” test: Can you describe every component in your architecture (or tab in your user interface or table in your schema) crisply in one sentence? As Albert Einstein said: “If you can’t explain it to a 6 year old, you don’t understand it yourself.”
- “What if” test: What breaks if I don’t have X? What if I don’t solve this problem? What if I don’t build this new component? What if I don’t have this extra input box in the user interface? When it comes to systems, laziness is sometimes a good thing.
- “Is my cost too high” test: Complexity shows up in costs (machine, operational and integration). Know the cost of the system you are building and ask yourself if the cost is too high.
- “Open source” test: For every component that you build, ask yourself whether you can open-source it. Have clean and well-defined interfaces. A good test of a clean interface design is that others can easily reuse it.
No, simplicity isn’t easy. But taking a deliberate approach — considering the sources of complexity and asking the right questions — is a good start.
This post also appeared on LinkedIn.