As Middesk grows, our software system is becoming increasingly complex. We’re adding more features, integrations, and use cases. One of the engineering team’s primary mandates is to manage that complexity. Success in this effort will allow Middesk engineers to safely and effectively make changes to support our customers.
We recently landed our Technical Strategy, which considers edge cases and states preferred tradeoffs when writing code. It is intended to simplify decision making, pushing engineers who communicate with less frequency (our org is growing) to make similar decisions, and thus creating implicit coordination.
Strategy must also stay close to implementation. “Build modular code” is strategically interesting but unhelpful to those who actually write code; it doesn’t express an opinion about what shape modular code should take here. “Build modular code by following domain-boundary patterns enforced by packwerk” is a much more opinionated, implementation-friendly strategy. The likelihood that disconnected engineers make the same decisions when building is much higher. And that creates consistency.
Modularity is part of our strategy.
One of the pillars of our strategy is code modularization. Software design experts like Sandi Metz [POODR], Kelly Sutton [Sutton], and Tom Long [Good Code, Bad Code] highlight modularization as an essential tool to manage complexity. Modular code has the following properties:
- A component (a collection of folders and files) obeys the Single Responsibility Principle by having one unified purpose. Said another way, objects in the component have high cohesion in that they are highly related in terms of their functionality.
- There is loose coupling between components, meaning one can change internally without requiring changes in another. This can be accomplished by exposing a minimal public API to be used by call sites external to the component.
Here, we’ll cover the options we considered for modularizing our code. After looking into microservices, we ultimately recommend building a modular monolith. We discuss the service object design pattern to accomplish this. We also recommend a tool called packwerk that will help the team write and maintain modular code.
Problems
Bloated models
Today at Middesk, we have models acting as services and data-access objects. Some models have a large number of responsibilities and dependencies, becoming tightly coupled with each other. In our experience, this is common in Rails monoliths like ours. Tomorrow, as our application grows in complexity, making changes to methods implemented in models will be even more complex unless we address the issue. Sandi Metz writes in POODR:
Left unchecked, unmanaged dependencies cause an entire application to become an entangled mess… A class that, if changed, will cause changes to ripple through the application will be under enormous pressure to never change.
For example, consider this version of our Application model. The submitted! method introduces subtle behaviors and dependencies on the Company class (via update_company!), Referral class (via complete_referral!), and our email system.
Complex side effects
We also have many actions triggered by callbacks. This mechanism can make it difficult to reason about what will happen as a result of calling a method. For example, saving a Business has cascading effects of which the developer may not be aware, like emailing, sending webhooks, and updating review tasks. Calling update and then having five other actions occur is difficult to fit in a programmer’s mental model.
Instead, we value explicitness in code.
Callbacks can also force a model to behave a particular way due to assumptions that were made about it in a certain context. As new product requirements inevitably get introduced in the future, a callback that’s tied to a model has the potential to no longer become relevant in a different context. You end up wanting to turn off callbacks in certain cases, increasing the complexity of the model.
For example, the version of our TaxRegistration model below has nine side effects on creation. It’s hard to reason about all this code executing as a result of one method call. Additionally, create_foreign_qualification has a complex conditional. Over time, we’ve realized we don’t want it to run on every create and the effect may be better located at the callsite where it’s desired.
Options Considered
Microservices
As discussed above, loose coupling between components makes development faster and safer. One way to accomplish this modularity is via microservices, in which a large application is composed of many small services, each of which is deployed independently. This approach encourages modularity by requiring services to explicitly declare their interfaces via public APIs only accessible via network.
There is contemporary wisdom that discourages the use of microservices for modularity. Kelly Sutton of Gusto says, “If you do not have compliance or scaling concerns prohibiting data sharing with your monolith, building within the monolith is the safest route.” [Sutton] Shopify, perhaps the most complex Rails app in the world, also chose a modular monolith. [Westeinde] This is because microservices bring with them a host of challenges. Each service requires its own test and deployment pipelines. Calls to other services use the network, adding latency and decreasing reliability.
Further, without clean separation of our business domain in code, microservices would essentially scatter a given domain across myriad runtimes; we need to modularize before we consider microservices.
While microservices can be an effective way to increase the scalability of our systems and address compliance concerns, we do not recommend them for modularity.
Service objects
Service objects have become popular at large Rails shops like Airbnb and Gusto, where developers place the vast majority of business logic in service objects. These classes are instantiated to perform a particular purpose, usually by depending on a group of related classes.
Using service objects helps avoid bloated models. They encourage embracing the single responsibility principle, wherein an object focuses on the one task for which it is responsible. In Tom Long’s Good Code, Bad Code, he explains that when a class is cohesive in this way, it is more easily read, reused, changed, and tested.
These objects also avoid complex callbacks. To take an example, consider a Company object that sends an email on create. A conventional way to do this would be a Company ActiveRecord callback that passes the Company to a CompanyMailer object, thereby introducing a circular dependency. Instead, a CreateCompany service object can be used to remove the circular dependency, keep the models as leaves in the dependency graph, and make the application easier to reason about and change. [Sutton]
In the above example, we see an opportunity to introduce a service object to improve Application submission. This approach delegates referral creation to a higher-order, context-aware class:
The ApplicationSubmitter reduces the complexity of the Application model and removes circular dependencies. It can also grow to handle additional submission complexity that currently lives outside of the model.
Enforcing modularity with packwerk
Writing modular code is hard. Without careful design consideration, components become highly coupled to one another, increasing maintenance costs and slowing down development. While creating a culture of thoughtful design is a strong start, our engineering organization can take this effort further by enforcing modularity with tooling.
When Shopify committed to building a modular monolith, they recognized the importance of tooling to aid this effort. They built and open-sourced a tool called packwerk, which has since been adopted at Gusto as well. packwerk integrates into CI/CD pipelines to flag dependency violations. The usage manual highlights key features:
- Any folder can be made into a packwerk “package” by adding a package.yml file
- By default, packages do not flag any dependency violations. Packages must opt in to enforcement, allowing an existing codebase to transition gradually
- Privacy enforcement makes any constants (e.g. the Business class) in the package private. Only constants placed in the public folder can be referenced by other packages
- Dependency enforcement requires packages to explicitly list the packages the depend on
We’re adopting packwerk at Middesk to encourage engineers to think about modularity as they design systems and write code. We’re starting small with a couple of subdomains within the monolith. In the future, we’re going to do a full audit of the existing classes in the repository, assigning them to packages and then enforcing privacy and dependency violations using packwerk.
In Summary
By using service objects and packwerk, we're empowering engineers to write modular code at Middesk. We believe this will make it easier to reason about our codebase and make changes to better serve our customers.
