The Business-DSL DevOps Process

During my talk this week at the JAX conference one of the audience asked how the devops process works for systems where domain experts contribute the Fachlichkeit directly using DSLs. This prompted my drawing of the Big Picture picture here which illustrates how the roles of domain experts, language engineers, infrastructure guys and technical architects collaborate to deliver DSL-based software in an efficient, fast and reliable way. Here is
the picture; I will discuss it in the rest of this article.

Modeling and Testing of Fachlichkeit by Domain Experts

Not surprisingly, the pipeline starts out with the domain experts. They use
non-formalized processes to build an intuitive understanding of “the world”, i.e., the the aspects of their domain they are tasked with contributing to the software system. For example, they might analyze tax law.

Based on this understanding, they create models that faithfully represent the domain. They use one or more suitable DSLs for this task (we will return to this below). Importantly, they also write tests. Good business DSLs always have ways of expressing tests for the Fachlichkeit at various level of granularity . The IDE supports the execution of those tests, rendering the test results directly in the IDE. The domain experts thus have an efficient and immediate way of expressing and validating Fachlichkeit. If tests fail, this feedback arrives at the domain experts, and they are primarily responsible for fixing the problem with the Fachlichkeit.

In-IDE execution is often achieved through in-IDE interpreters.

Automated Test Execution on a CI server

Next, we use existing version control technology to forward the updated models to a continuous integration server. There, again, we run tests, just like in the IDE, but we run all of them; a particular user might only have checked out a part of the overall model, and thus also only run a part of the tests, so he might inadvertently break tests “outside” his current scope. Running all tests on a CI server for each commit addresses this risk. If tests fail, we notify the domain experts; we still presume that there is a problem with the Fachlichkeit or its tests.

Both the VCS and the CI server are painted green, because these are managed and setup by the infrastructure people. While, in the spirit of devops, we don’t want to draw the lines between those roles too strongly, it is quite clear that it will not be the domain experts who set up and run this infrastructure.

Code Generation and Testing of Code

Usually the interpreter that is used inside the IDE is not the means of executing the Fachlichkeit in the actual target system (if it is, this step can be skipped). Instead, in many cases, we actually generate source code for downstream compilation, or we generate an intermediate format which is then interpreted. In both cases, this is, if you will, a secondary definition of the language semantics, and it has to be aligned with the primary one, the interpreter. In other words: we have to run all the tests in the generated version of the Fachlichkeit again. Of course we do this on the CI server as well:

This is where the language engineers come into the picture. It is their responsibility to write the generator. And if tests that work in the interpreter fail in the generated code, then it is a problem in the generator; thus, failed generated tests, or failures in the generation itself, end up in the lap of the language engineer, not the domain expert.

And since we’re talking about the language engineer here: of course, they also develop the languages used by the domain expert, and they are also responsible for the language workbench.

System Integration

The code generated so far is only the direct representation of the Fachlichkeit. We can run the Fachlichkeits-Tests against it, but it is not yet integrated with technical concerns of the system such as databases, external system connectors or means of handling -ilities (yes, I am vague here, because these really depend on each particular system). Those aspects are handled by the technical system architects. So in this step, we integrated the Fachlichkeits-core with the frameworks and facilities of the overall system. The result of this is essentially a deployable package that can technically be released into production.

Note that we have made a small change to the previous step: the generators might of course already know something of those technical aspects as well (see the little yellow-ish arrow on the brown generation arrow). So the generators are usually developed jointly by the language engineers and the technical architects.

Staging

Once we have the overall deployable package(s), those can be deployed to a staging infrastructure, where the Fachlichkeit is ultimately executed in its “natural habitat”, i.e., in the context of all the connected systems and infrastructure,

Now, if something goes wrong here, it is much harder to tell whose contribution caused the problem. So everybody is notified, and they need to figure out together what the problem is. However, it should be quite rare that something goes wrong here because of all the previous testing steps; in particular, a failure in the Fachlichkeit should be exceedingly rare.

Going Live

The final step is going live; essentially, the staging environment is “mirrored” into the real world. Nothing should ever go wrong here; if so, it’s probably the CEO who’s notified :-)

This closes the circle for the domain expert. As you can see from the highlight, they mostly care about modelling, local testing, fixing problems that get reported from the first stage of the build server and then, finally, smoke testing the staging environment and releasing the new Fachlichkeit into production. Exactly as it should be :-)

Automation, Iteration and Roles

Note that, in the spirit of devops, the complete process is automated (except, of course, the release into production, which requires explicit triggering). And despite the explicit mentioning of roles and their responsibilities, the process should be collaborative. Especially during the initial stages of establishing the process, the assumption that failing “blue” tests are always a problem with the Fachlichkeit or its test might not be true: there might be a bug in the (not yet mature) interpreter or other aspects of the language definition.

The Language Development Process

I didn’t cover the development of the languages in this article. I will do that in a future post. But suffice to say, that this also happens collaboratively between the at least domain experts and language engineers, sometimes also with the technical architects, and it is highly iterative, as the following picture hints at. As I said: more details in a future post.