Distributed architecture smells. — A simple guide to avoid common mistakes

Sometimes I saw lots of configurations being set inside the codebase, this force a new build at every change and can have worrying security consequences.

Configuration values needs to be detached from both from the environment and the artifact repository using them, either on a separate repository or feature flag provider.

In code and its tests, inject only the portion of configurations that are needed, possibly inject them with a dependency injection container (to lower coupling).

A good rule of thumb should be that no file-based configuration should be required on unit tests, this imply that the configurations will need to be provided in the test itself (hi cohesion), thus making the configurations also decoupled from the file system.

When, inside your system, you use the full data model provided by a service, even if you only need parts of it. Focus on the slice of data relevant to your microservice and disregard everything you don’t need, avoid even serializing the fields you don’t use.

Make eventual consistency your friend and increase resilience by asking for event-provided data, instead of over rely on RPC calls and caching, but pay attention not to cause eventual inconsistency by ignoring its lifecycle.

Be sure of the data you own, and even if you could, never share directly database resources.

Don’t repeat yourself (DRY) is a rule that invite us not to write the same code twice, but sometimes that needs to be postponed. That’s why, it’s nice to keep in mind to write everything twice (WET) before jumping to hasty abstractions!

When I found out that sharing code with binaries could lead to slow downs, I realized that even basic reusing of code should not always be prioritized over single-responsibility, decoupled code. So even inside the same artifact, for example, models that are used to respect outbound contracts are evolving differently from domain core ones, so it’s important to consider them distinct entities.

This applies to all sorts of structures and functions, that lead me to use more and more feature packaging. So please, never reuse request and response models inside your domain code and use different domain models to solve different problems.

If you are storing external libraries inside your repo, stop immediately and start using the most appropriate tool of your ecosystem to handle dependencies (such as NPM, NuGet or Gradle), make sure all environments either retrieve dependencies in the same way, or even better, act on the same produced immutable package, store in the same dependency tool.

Make sure you are always able to determine which commit that artifact/version refers to, and remember to consider it important on the log, especially when doing blue green deployment.

Complex stored procedures, RPC calls without a standardized interaction model (like SOAP, which might be hard to use for many languages), full duplex communication, database functions which you are not sure your testing framework can invoke, interactions between microservices.

If it’s easy for you to test those technologies, or if you can accept the risk involved in not testing it, go ahead. Do not consider adopting some technologies altogether, if you cannot test them.

More importantly, if the solution to something hard is “this will require manual test”, you should know you will end up in a bad place.

There are several³ ways⁴ to go if you want to make sure you are not building the next distributed monolith, these are my suggestions:

• Aim to use better languages instead of relying on complex libraries: many java libraries can be replaced entirely by basic language provided by kotlin, or by using the latest .NET features
• Check your logs and test your alerts regularly in all the environments, like in dev or release
• Keep a similar data flow in all your non mocked environments
• Never store production configurations on code repository, just copy them on the target environment
• Automate as much as possible the creation of contracts and data transfer models
  o   Use open API documentation
  o   Remember that errors should be documented as well
  o   Auto generate it from your code (it’s ok to have comments on your code if you use it to create documentation)
  
  
• Realize the fact that sometimes you need RPC⁵, and stop worry about real/mature REST⁶
  o   Consider using protocol buffer to define and generate those calls and avoid common versioning problems
  
  
• Enforce cross functional requirements in all your environment
  o   Use integration tests to make sure observability works
  o   Remember to keep logs consistent to the definition of the various levels of logging (hint: ERROR is only for failures in current activity, not for application-wide failures)
  o   Check tracing and logging in all environments
  o   Establish sensible defaults for alerting system
  
  
• Use the right tool for the right task
  o   REST is for rapid iteration (human readable) of stateless resources on HTTP
  o   gRPC is gold when resources are scarce
  o   GraphQL when client decide how to aggregate data, otherwise it might be unnecessary in most cases
  o   Kafka when you need to sustain a complex and scalable event driven architecture with hi throughput⁷
  o   RabbitMQ or similar (AMQP) less feature-rich queue processing systems when you need speed of development and simplicity
  
  
• ·Test on a mock environment
  o   Prioritize having an environment where all major features can be tested during the automated deployment pipeline
  o   Use mock server or similar things to guarantee reproducible and predictable E2E runs.
• Consider carefully if you can go for a simple monolith⁸
  

Thanks to all the Digital Dealer developers who provided feedback and suggestions to early drafts of this article. Thanks to Martin Blomster for his advice, insights and support.

/Giulio Caccin