View Details

How to choose the target language for a migration? There are moments in the life of a software system—and in the life of those responsible for it—when you feel trapped.

Trapped by a language that once served you well but now feels like a cage. You have an important application, something your business depends on, but the technology it’s built on is making your life miserable. Every change feels like climbing a mountain and you find yourself thinking, “Why is this so hard?”

You don’t think about migrating just because things are slightly inconvenient. You consider migration when the pain becomes too much and you know something has to change.

So you wait longer than you should and then you start thinking: “What if we move to a different language?”

But this brings you to the first, most critical obstacle: which language? You know migration isn’t a small task. It will take time, effort, and resources. And the last thing you want is to jump out of the frying pan and into the fire.

Maybe you’re not the one who chose the current language. Maybe you inherited this situation. Maybe you’ve spent years patching things up, keeping everything running. But now it’s your decision—and if you pick the wrong target language, you will have to own that mistake.

Luckily, choosing wisely isn’t rocket science. There are a few key principles to guide you, principles that can help you make a choice you won’t regret.

In a previous article, we explored how to choose a target language specifically when migrating from RPG. Many of the same concerns apply here, but in this article, we’re going broader. We’ll explore how to pick the right language for any migration, from any outdated or problematic technology.

And if you are specifically dealing with RPG? Go check out that original article for in-depth advice.

Now, let’s see what really matters when you’re deciding where to go next.

Factor 1 – Why are you considering a migration?To pick the language that will solve your problems, we first need to figure out what your problems are.

You might be facing one—or several—of these recurring issues:

1. Talent Is Drying UpMaybe your system was built in a language that no developer under 40 or 50 even recognizes. Your developers are hard to find, harder to keep, and they’re heading for retirement faster than you can replace them.

This is the most common reason we see for migrations.

Even if you’re managing for now, you can feel the walls closing in. Fast-forward a few years, and it won’t just be a problem—it’ll be a crisis.

1. You’re Stuck on Outdated HardwareSome languages are tied to specific hardware, especially older systems. You’re stuck running on proprietary machines, while the rest of the world has moved on to standard servers or the cloud.

You want freedom:

• Freedom to scale.
• Freedom to choose your vendors.
• Freedom to stop paying a premium for ancient, specialized equipment that no longer provides advantages oer alternatives.

But your language is just not offering you that choice.

1. The Language Itself Is DyingSometimes, it’s not just about people or hardware—it’s about the ecosystem.

2. No updates.

3. No security patches.
4. No libraries.
5. No community.

Even if you can train new developers, you’d still be fighting a losing battle.

1. Performance Is Breaking YouMaybe your system worked great when you started. But now? You’ve grown. You’re a victim of your own success. More users, more data, more demand—and your current tech just can’t keep up.

What’s Driving You?These are the most typical reasons for migrations. If you’re migrating for something else, I’d love to hear about it—you can find my contacts on the website.

But here’s why this matters: your reason for migrating shapes your decision.

• If it’s about finding developers, cloud support, or getting away from a dying platform? The answer is usually simple: pick a popular, well-supported language and move on.
• If it’s about performance, or something more specific? You’ll need to weigh other factors.

So start here: why are you migrating? That answer will be one of the most important inputs in choosing your next language.

Factor 2 – What Kind of Organization Are You?Let’s not overthink this.

• If you do not need your language of choice to be your competitive advantage, because software is just an enabler for your main activities, then just pick something safe, stable, and boring. You don’t need clever. You need solid.
• Are you a tech-first company? Are you looking for a differentiator that could give you an edge? Do you see yourself in Beating the Averages? Then the right choice for you could be getting less mainstream language: it may have been Ruby a few years ago. Today could be Elixir, Rust, or Clojure.

Factor 3 – What Is the Domain of Your Application?Some domains naturally gravitate towards specific languages. This means that you can find talent familiar with your domain if you pick one of these languages. Also, in that ecosystem, there will be libraries and know-how that serve your domain particularly well.

• Finance, Banking, Insurance? → Java or C#Why: Stability, performance, tons of enterprise-grade libraries, and strong typing help prevent critical mistakes.
• Data Science, Machine Learning, Automation? → PythonWhy: Huge ecosystem (Pandas, NumPy, TensorFlow), easy to write, fast for prototyping.
• Enterprise Applications (ERP, CRM, HR Systems)? → Java, C#, or PythonWhy: Mature ecosystems, cloud-ready, long-term support, proven scalability.
• DevOps, Infrastructure, Tooling? → Go, Python, or Ruby Why: Go is fast and great for CLI tools; Python and Ruby are scriptable and flexible.
• Media Processing (Video, Audio, Image)? → C++, Rust, or Python (with C/C++ backends).

Why: Performance critical, often hardware-bound. * Telecommunications / Distributed Systems? → Erlang, Elixir, or GoWhy: Built for concurrency, fault tolerance, message passing.

Factor 4 – What Is the Expected Lifetime of the Application?Ok, predicting the future is always a gamble.

But just because we can’t know for sure doesn’t mean we can’t use common sense to make reasonable decisions.

There’s a huge difference between building something you’ll use for 2 years and something you plan to keep alive for 20 years.

If it is something you expect to need for the short-Term (5 years or less), then you’ve got more flexibility, you can maybe even a bit opportunistic:* You might choose a language that’s quick to develop in. * One that has plenty of available developers right now, possibly at lower rates. * You can focus on short-term costs and speed, rather than worrying about what the tech world will look like in a decade.

Basically, for short-lived projects, you optimize for the now, not the distant future.

But if you expect to keep the application long-Term (10 or more years), then stability is non-negotiable.

Now you need to think about:

• Will I still find developers for this language in 10 years?
• Will the tools, libraries, and frameworks I’m using still be supported?
• Will the language itself still be evolving, or at least maintained?

This is where it pays to play it safe.

Here’s a simple rule: don’t pick a language that isn’t already one of the top 5 most popular.

Why? Because popularity today doesn’t guarantee survival tomorrow—but it does give you odds in your favor.

• Popular languages have huge communities.
• They have momentum behind them—companies, developers, educators all invested in keeping them alive.
• If a language is thriving now, it’s far more likely to have strong support and active development for years to come.

Languages like Java, Python, C#, JavaScript, and even Go have reached a critical mass. There’s infrastructure, training, and a market built around them. That doesn’t make them future-proof, but it makes them a sensible bet. Which is the best we can do: if you and me could predict the future we would not be in this line of business.

Factor 5 – How difficult is to migrate to that languageSome migrations are just harder than others, so if time and money to get your migration done are something to keep in mind, you need to consider this factor.

Not all target languages are created equal when it comes to migration difficulty.

The bigger the gap between your current language and your new language in terms of execution model, the harder it will be to migrate.

• Procedural → Object-Oriented?Tough, but doable. This is your RPG or COBOL to Java, Python, or C# kind of move. It takes some effort to rethink things, but there are established patterns to help.
• Procedural → Functional?That’s a whole different beast. Moving from something like RPG or COBOL to Elixir, Haskell, or Clojure? Now there is a mismatch making the translation much more challenging.

It’s not impossible, but it’s harder, riskier, and more expensive.

Factor 6 – What Skills Do You Have In-House?Sometimes, you’re not starting from scratch. Sure, one application might be stuck in an old, painful language—but what about the rest of your system?

• Do you have other modules or services written in something modern?
• Do you have teams already working with Python, Java, or C# on other parts of your stack?
• Have you built any internal tools, even small ones, that rely on newer tech?

If so, let’s leverage that.

If your team already runs other services in, say, Python, then moving the legacy system into Python means:

• You’ve got people in-house who can help.
• You’ve already solved deployment issues for that stack.
• You might even have libraries, utilities, or processes built around it.

This is especially critical for small organizations, where you can’t afford to spread yourselves thin. You just don’t have the luxury of hiring teams for every language.

If you’ve already got something working in a modern language, lean into that.

For small teams, having too many languages is a recipe for: Higher costs, Slower development and Hiring nightmares.

Sometimes, unfortunately, you’re stuck: The only language your team knows is the one you want to leave behind. So, well, that means you can pick whatever language you want, because anyway you will have to retrain your team (or hire a new one).

So what?Alright, we’ve gone through six different factors. You’ve seen arguments pulling you in one direction, and others pulling you the opposite way.

| Factor | Key Considerations | Impact on Language Choice | | 1. Why Are You Migrating? | Talent scarcity, outdated hardware, dying ecosystem, poor performance. | Drives whether you need a popular, well-supported, or high-performance language. | | 2. Type of Organization | Are you tech-first or using software as a tool? | Tech-first may consider niche languages for advantage; others should pick safe, stable, mainstream options. | | 3. Domain of Application | Certain industries favor specific languages (e.g., Java in Finance, Python in Data Science). | Picking a language aligned with your domain offers better talent, tools, and community support. | | 4. Expected Lifetime of Application | Short-term (<5 years) or long-term (10+ years). | Short-term allows flexibility; long-term demands stable, popular, and future-proof languages. | | 5. Difficulty of Migration | Procedural to OOP is easier; Procedural to Functional is harder. | Choose a language closer to your current paradigm if time, cost, and risk are major concerns. | | 6. In-House Skills | Existing team experience, current technologies used elsewhere in the system. | Leveraging existing skills reduces training costs and simplifies deployment, especially for small organizations. |

So what now? What decision should you actually take?

I’m not going to pretend I can give you some magical thumb rule that works for everyone. That would be arrogant—and frankly, misleading. The truth is, you need to weigh these factors for your specific situation. Only you know your team, your problems, your goals.

If you’ve read all this, reflected, and you’re still stuck, here’s my advice:

• Pick one of the most popular languages.Popular means safe. Popular means you’ll find developers. Popular means tools, libraries, support.
• If in doubt, just go with Java.It’s not sexy, but it’s reliable, scalable, and everywhere.

But if you are not yet convinced you can always hire a consultant. Sure, you should be ready to spend some time and money. And in the end they’ll probably say: “Go with Java.”

When asking around consider that every developer has a personal bias. I personally love static typing. If it were up to me, everything would be written in Kotlin but I know that’s not the best choice for every company, every team, or every project that’s why you didn’t hear me push it much in this article (it took some will to refrain from doing so).

Hopefully this article was useful to you. Maybe you already had a gut feeling when you started reading and this article helped you cross the line and make that decision.

If it helped—great. That was the point.

If not, and you’re still thinking about it, or want to tell me what I should’ve covered instead?

Feel free to reach out. My contact info is on the website.

You can share your thoughts—or, if you really hated this, just send me a friendly insult.

Thanks for reading. Now go make the right call for you.

The post How to choose the target language for a migration appeared first on Strumenta.

View Details

In this article, we are going to present our Strumenta’s EGL parser, a battle-tested parser for IBM EGL. It is also the only parser for EGL that we are aware of. It is based on our Starlasu methodology and real-world use cases.

For instance, you can use it to analyze a project and remove code that is unused. You can use it to provide autocomplete support in an editor. It can even be the foundation of a transpiler to move from EGL to Java.

The parser is commercially licensed, but we hope this article will be useful for anybody wanting to learn what a parser is for and how to use one.

If you are looking to know what is a parser, we are going to explain it.

If you are looking to know what you can use a parser for, we are going to show you two limited examples, to give an idea of the potential and what it means to work with it:

• build documentation extracting the API of a REST service
• create a diagram from record definitions

Even if you are looking to design your own parser, we are going to help you, by showing you why you might want to follow our Chisel method to build one. So, you can do that on your own too. You should expect to take a significant effort and refinements over millions of lines of code before it is in great shape. At least that is what it took us to get there.

What is a Parser (And What is Not)We work with parsers every day, so for us, it is very clear what a parser is and what it can do. However, 99% of developers will never look too closely at what a parser is, so let’s refresh our memory on this.

In general terms, a parser is software that can understand the syntax, but not the semantics of some code.

Fundamentally, a parser is a software that creates a model of the input, so we can extract data from it.

This model should be clear, useful, and easy to work with. The model that matches all these criteria is the Abstract Syntax Tree (AST). Using an AST to represent code is a common practice, Starlasu defines a standard. Of course, how to design the AST for a particular language is open to debate.

A parser can read the code, but it cannot execute it. For example, a parser can both recognize a variable declaration and an expression.

What it cannot do is linking the two and understand where a variable used in an expression was declared. This feature is called symbol resolution and it is a functionality built on top of the parser.

Given the needs of our clients, we implemented symbol resolution functionality. This is technically outside the scope of a parser, but it was needed, so we added it.

For example, our EGL parser can resolve references to record definitions. We are going to see some examples of what our specific parser can do, later. However, it is an important point to keep in mind in case you are comparing different parsers.

Chisel MethodologyThe EGL parser is based on the Chisel methodology. It is the missing link between source code and a convenient structure for its interpretation and manipulation: an AST. When building an interpreter, transpiler, compiler, editor, static analysis tool, etc., at Strumenta we always implement the software using a pipeline. A set of reusable components that can be shared for different projects.

For example, this is a pipeline for an EGL-to-Java transpiler.

The EGL parser and the Semantic Enricher components can be re-used for, let’s say, building an interpreter. This approach increases productivity and improves the quality of the software. For instance, any improvement to a core component for one project gets automatically shared with others.

StarLasu is a method, a set of supporting tools and a collection of runtime libraries that implement this methodology to support it in Java, Kotlin, Python, Javascript, Typescript, and C#.

At its core, StarLasu permits the definition of ASTs, on which all other functionalities are built. You can navigate and transform ASTs to do everything from reading the original values to simplifying your code. With the features provided by the library, you can do anything from analyzing a codebase to building a transpiler.

Some core features shared by our StarLasu libraries are:

• Navigation: utility methods to traverse, search, and modify the AST
• Serialization: export and print the AST as XML, as JSON, as a parse tree
• LionWeb and EMF interoperability: ASTs and their metamodel can be exported to the LionWeb or EMF formats

Interoperability with LionWeb and EMF is important because these are standard formats used in language engineering. So you can mix and match different software, even from different providers.

You can read more about our methodology in a dedicated article.

What is EGL?EGL (Enterprise Generation Language) is a high-level, platform-independent programming language developed by IBM (and now open-source). It enables rapid development of multi-platform enterprise applications and is commonly used on IBM i (AS/400, System i), integrating with COBOL, Java, JavaScript, and relational databases.

For instance, you can write EGL code and have the system generate Java code, so you can run it on the Java platform.

EGL was intended to allow rapid enterprise development with less concern about platform details, like middleware and glue code to the underlying. Thus enabling business-oriented developers to write full-stack applications.

What Do You Need an EGL Parser For?EGL systems are typically large and complex. In many cases, EGL programs are older than the language itself. This is not the start of a logic puzzle: it is a language adopted by companies that wanted to keep old codebases on their IBM AS/400 platform, but develop new code using a more modern language. One consequence of this typical use case is that programs were written to work with older custom software that followed patterns of COBOL or RPG code.

You may use a parser to modernize your EGL code and cut the time your developers lose in updating old-style code.

EGL simplifies development by abstracting the underlying complexity of UI, logic, and data handling layers. This allows business-oriented developers to build software ignoring technical implementation details. The downside is that somebody must actually implement the technical details.

For example, a common complaint of Java developers is that the Java code generated by EGL has bad performance on the JVM, and it is not easy to work with from the Java side. Another example is that somebody need to implement the UI that is described using RUIHandler. Now, that development and maintenance of EGL is stopped, you might decide to take this UI description and generate a UI yourself.

To do any of these things you need to first understand the code. That is where a parser comes in.

A parser helps you analyze the structure of the code and transform it into a meaningful representation like an Abstract Syntax Tree (AST). From there, you can enable a wide range of applications: refactoring, migration, documentation, or code generation.

Why Use a Ready-to-go Parser?You need a parser that has been thoroughly tested, which is documented, and gives you someone to call in case you encounter any problems.

We are experts, and we have built tons of parsers for our clients. This means that we completely understand the importance of this component, and we have a solid methodology.

And we build parsers designed for what our users need. For example, our SAS parser is geared to support data lineage, because that is what the typical user needs.

Our EGL Parser has also been built for the needs of our clients, so it is battle-tested by us and our clients. It comes with symbol resolution, so it can also power transpilers or static analysis tools.

How to Setup the ParserThe only requirements you need to use the parser are the ast and semantics modules. They are developed in Kotlin, so they are easy to use in the JVM world. They can also be used from other languages as we can provide bindings for TypeScript, Python, and C#.

For example, for a Java maven you would write something like this.

<dependencies> <dependency> <groupId>com.strumenta.langmodules.kolasu-egl-langmodule</groupId> <artifactId>ast</artifactId> <version>1.0.1</version> </dependency> <dependency> <groupId>com.strumenta.langmodules.kolasu-egl-langmodule</groupId> <artifactId>semantics</artifactId> <version>1.0.1</version> </dependency> </dependencies> This would use the EGL ast and semantics module.

You can easily adapt this for another build system like Gradle.

dependencies { implementation "com.strumenta.langmodules.kolasu-egl-langmodule:ast:1.0.1" implementation "com.strumenta.langmodules.kolasu-egl-langmodule:semantics:1.0.1"} That is all you need to be able to use the parser in your code just as easily as any other library.

Extracting and Documenting REST Service InterfacesImagine you’re maintaining a suite of EGL programs that uses third-party REST services. Over time, new services were added, old ones were modified, and some were deprecated. There’s no centralized documentation, and nobody knows which external services the EGL code uses. This makes hard to keep track of what services your company uses or even migrating the interfaces to another language.

With the EGL parser, you can automatically:

• Identify all REST-enabled services (@GetRest, etc.)
• Extract metadata such as method, URL and path
• Generate a live catalog of used endpoints in formats like JSON or HTML

Keeping service documentation up to date manually is error-prone. By using the parser to generate this information, you ensure consistency between the code behavior and the documentation. This is quite useful for teams maintaining services or consuming APIs and for compliance purposes.

Example InterfaceLet’s take a look at some example code. This is an interface necessary to bind an external REST service to an EGL program.

Interface WeatherForecast Function GetWeatherByZipCode(zipcode string in) returns(myRecordPart) {@GetRest{uriTemplate="/GetWeatherByZipCode?zipCode={zipcode}", requestFormat = JSON, responseFormat = JSON}};end We want to extract the metadata and produces something like this.

Endpoint: GET /GetWeatherByZipCode?zipCode={zipcode}Input: [string zipcode] Create Diagrams from Record DefinitionsImagine that you have a series of records, like the following ones.

record Result id string; title string; address string; city string; state string; latitude string; longitude string; rating Rating; url string; categories Categories;endrecord Rating averageRating float; totalRatings int; totalReviews int; endrecord Categories category Category;endrecord Category id string; egl\_value string?;end You want to analyze these records and their relationships. For instance, you want to create a diagram to document the structure of the data. Or maybe you want to create corresponding data classes in Kotlin.

The first step is the same, you parse the code and gather the info about records. To build a diagram you can take advantage of PlantUML, a library that can generated diagrams based on a description (DSL) you gave.

So, to create a diagram from your EGL records all you need to do is:

1. Parse the code using our EGL parser
2. Walk the tree and get a list of records and their fields
3. Create a file written in the PlantUML format and have PlantUML generate the image

How many lines of code do you think you need to write to achieve this?

val result = EGLKolasuParser().parse(code)val result = EGLKolasuParser().parse(code)val root = result.root as EglCompilationUnitvar classesDefinitions = ""var classesLinks = ""// let's collect the classes used in this file beforehandval registeredRecords = root.eglRecords.map { it.name }// build the strings representing class descriptions and their relationshipsroot.eglRecords.forEach { classesDefinitions += "class ${it.name} {\n" it.fields.forEach { field -> classesDefinitions += "${getType(field.field.type)} ${field.field.name}\n" if(registeredRecords.contains(getType(field.field.type))) classesLinks += "${it.name} --> " + "${registeredRecords.first { r -> r == getType(field.field.type)}}\n" } classesDefinitions += "}\n"}// let's assemble the PlantUML fileval source = """@startuml |${classesDefinitions} | |${classesLinks} |@enduml""".trimMargin()val reader = SourceStringReader(source)val png: OutputStream = FileOutputStream("diagram.png")reader.outputImage(png) If you answered 30 lines of code, you were right! We have to collect the list of records used in the file beforehand, on line 7, because you could define the records in any order. So, if record Result uses record Rating you do not need to define Rating before defining Result.

At the end, you would get an image like this one.

ConclusionUsing an EGL parser unlocks the ability to understand, transform, and modernize your EGL applications. Whether you are building tools for developers, automating documentation, or preparing for a migration, having structured access to EGL source code is a game-changer.

In this article, we explored how to extract REST services interfaces, and showed diagrams that can make this process actionable.

If you are interested in learning more or trying it on your own codebase, get in touch with us at Strumenta. We will be happy to help you get started.

The post How to Use the EGL Parser appeared first on Strumenta.

View Details

Today, we are discussing the challenges of a migration project. Instead of discussing just the theory, we will take a real piece of software written in Java and we will convert it to Python.

Now, at Strumenta, we typically work on legacy migration projects, involving applications built in older languages like RPG, Visual Basic, or various 4GLs. However, those applications are proprietary and we cannot tell much about them. This is why we picked an example that was open-source (and something we actually needed to convert).

Sure, there are some differences with migrating a legacy application, but the principles remain the same. More on the differences below.

How does this migration compare to a typical legacy migration project?Typically, legacy applications we want to migrate are ERPs or MRPs which contain proprietary business logic accumulated over decades, lack unit tests, and are riddled with code clones and massive functions.

In this article instead we will share insights from our migration of LionWeb-Java to LionWeb-Python, both implementations of the LionWeb specification—a library for language engineering and model processing.

| Typical Legacy Migration Project | LionWeb-Java to LionWeb-Python | | Size of the codebase | 1M-20M lines of code | 10K lines of code | | Can we show the code? | No | Yes! | | Presence of tests | No | Yes, but we will ignore them, to simulate their absence | | Usage of libraries | Typically depending on the system library and internal library | Dependencies limited to Gson and some serialization libraries | | Quality of the code | Low: code clones, dead code, god functions | High: code developed in the open, reviewed, refined | | Migration Principles | Exactly the same |

Where do we start?For most projects, this process begins with a Migration Blueprint, during which we analyze the existing application, compute statistics, and define the target system in collaboration with the client. However, in this case, since we were both the authors of LionWebJava and the primary users of LionWebPython, this step was more informal. The main decision points were adopting modern Python practices, such as using pip for package management and type hints for type safety. Ok, we just replaced several workshops with a couple of sentences. Good.

But we also make an important decision: we will deliver the migrated code incrementally. In other words, we want to deliver a first group of files that can work independently and that the Client can start validating or even use it. On the opposite, we could deliver the entire codebase at once, but that would be a bad idea.

Why? Because a full transpiler takes time to develop, leaving the client without intermediate deliverables for validation. So, in a real project, the Client would be in the dark for quite a while, wondering if they made a good decision to trust the people they asked to build the transpiler. That is not a great scenario.

Instead, we aim to provide an incremental migration plan where initial outputs can be reviewed early. In this way the Client has something they can put their hands on and start providing feedback. That is reassuring for the Client, but also for us, because no matter how deep the initial requirements discussion are, it is always wise to check we are aligned as frequently as possible.

Defining a Migration PlanOnce we decided to proceed, the next critical task was defining the migration order of files. One cannot simply take files randomly because of interdependencies among them: a certain Python module will reference another Python module and if such module is not there the code will fail to run. Therefore, we needed to calculate dependencies between files and establish an optimal sequence for translation.

Dependency analysis requires parsing the code and resolving symbols:

• When encountering a method call, we must locate its definition.
• This requires understanding Java’s method resolution rules, including inheritance, overloading, and interfaces.

Let me illustrate this with this example:

This process can be complex, especially when dealing with inherited fields, generic types, method chaining, and lambdas.

Once we have done it, we get our migration plan, i.e., the order in which we should translate our Java files:

Our initial dependency graph showed:

1. Some files (e.g., enums, interfaces) with no dependencies.
2. Files that depend only on those fundamental files.
3. A large set of files that is interdependent with each other, forming a cluster of 27 tightly coupled files.

Handling Clusters of Interdependent FilesWhen we encounter a cluster we can decide if to translate it as a whole, or if to translate partially, using an appropriate technique that we will describe.

To make this decision we may want to consider the effort necessary to translate these files. Now, the effort is not determined by the size of the codebase. It depends instead on how many idioms and constructs we have not yet encountered in the project up to that moment. If, for example, we encounter a cluster of interfaces, and we already have created the rules to translate interfaces, there will be no new work for us to do, be the interfaces three or three-hundred. So we measure the effort in terms of idioms and constructs not yet supported by our transpiler and that we need to support to translate the cluster.

If the effort can fit into an iteration, then we do not bother performing a partial translation of the cluster and we just translate it entirely. If instead the cluster would require a lot of work to be transpiled, then we move the dependency analysis at another level of granularity.

So far, when we said that a file was dependent on another file we considered all references appearing anywhere in the file. However there are references that appear in more delicate positions than others. Consider this:

• A file Foo.java defines a class Foo, which extends Bar, defined in Bar.java. So Foo.java depends on Bar.java
• A file Zum.java defines a class Zum, which defines a method zum, which returns Baz, defined in Baz.java. So Zum.java depends on Baz.java
• A file Qux.java defines a class Qux, which defines a method qux. Inside that method we instantiate Blorp, defined in Blorp.java

These three cases illustrate three level of dependencies:

• The first case is a type-definition dependency: we cannot even define the types without satisfying the dependency
• The second case illustrates a method-signature dependency: we can define the type, but not the methods they declare, without satisfying the dependency
• The third case illustrates an implementation dependency: we can define the type and the methods, but not their body, without satisfying the dependency

So what we can do in a similar case is to translate some files partially: we would translate everything besides the bodies of the methods which have dependencies to files we have not yet translated. In those cases we would just replace the body with a statement to throw an exception indicating the file has yet to be translated.

So we could translate some methods for some classes out of those 27 files, and provide a subset of methods that work and can be executed correctly.

Similar approaches can be used for other languages, but the level of dependencies is different for different languages, so this method requires some language-specific adaptations.

Validating the Migration with Unit Tests

In the case of LionWeb-Java, we could have translated the existing Java unit tests alongside the code. This would allow the client to run tests and validate the correctness of the migrated files. However, in most legacy systems, unit tests do not exist. Given we want to show how this approach could work also for legacy applications, we pretended we did not have such unit tests.

We instead generated unit tests, and we did that based on the original Java code rather than the translated Python code. This is crucial because generating tests directly from the migrated Python classes would only verify internal consistency rather than correctness relative to the original system. By instead deriving tests from the Java source, we ensured that the migrated Python code was validated against an independently produced reference.

If you want, we have two sets of artifacts we derived from the original source code:

• The translated source code, produced by the transpiler
• The generated tests, produced by the tests-generator

These two set of artifacts are produced by two independent processes. We then verify they produce coherent results: we do that by ensuring the tests compile and work against the transpiled code. When this happens it means that either both set of artifacts are coherent, and therefore the process producing them is correct, or they happen to be wrong in an accidentally compatible way. This latter scenario is highly improbable.

This is an example of a generated test:

import unittest

from lionwebpython.utils.IdUtils import IdUtils

class IdUtilsTest(unittest.TestCase):

def test_clean_string_single_period(self):

input_string = “hello.world”

expected_output = “hello-world”

self.assertEquals(IdUtils.clean_string(input_string), expected_output)

def test_clean_string_multiple_periods(self):

input_string = “192.168.1.1”

expected_output = “192-168-1-1”

self.assertEquals(IdUtils.clean_string(input_string), expected_output)

def test_clean_string_no_periods(self):

input_string = “helloWorld”

expected_output = “helloWorld”

self.assertEquals(IdUtils.clean_string(input_string), expected_output)

def test_clean_string_empty_string(self):

input_string = “”

expected_output = “”

self.assertEquals(IdUtils.clean_string(input_string), expected_output)

def test_clean_string_only_periods(self):

input_string = “….”

expected_output = “—-“

self.assertEquals(IdUtils.clean_string(input_string), expected_output)

Ok, I will admit it is a rather simple case, but it should give you the idea.

You may wonder how we generated the tests. Well, every good chef has their little secrets, right? This is something that we will expand in a future article, but for now we can share some insights into our approach. We used a combination of algorithmic techniques to extract the appropriate context and then feed into an LLM, that was instructed to analyze the original code to identify requirements. From those, with some massaging and providing context about the migrated code, we were able to generate tests that we could incrementally refine removing compilation and stylistic errors. This allowed us to cover realistic scenarios and verify that the migrated code behaved consistently with the original.

That said, there are alternative approaches for those who cannot fully automate unit test generation in their migration projects. One option is to manually create higher level tests using Gherkin. This enables teams to describe expected behaviors in a human-readable format, making it easier for non-developers to contribute to test definitions.

Another interesting alternative is Approval Tests, which help verify code behavior by automatically comparing test output against an approved reference result. This is particularly useful in migration projects where the primary goal is to ensure that the migrated code produces the same output as the original, without necessarily having to rewrite specific test cases for every function.

In a future article, we might dive deeper into our approach and compare it with these other solutions to help those facing similar migration challenges choose the best strategy for their context.

At this point we can deliver to the Client running code with running (and passing) tests. The client could then review the translated files for stylistic and performance considerations while running the unit tests for functional verification.

Beyond validation, having unit tests for the migrated system is essential for long-term maintainability. The lack of unit tests in legacy applications is a primary reason for their difficult upkeep. By introducing tests during migration, we help clients maintain confidence in the correctness of their new system and reduce the risk of regressions in future modifications.

Additional Challenges in Java-to-Python MigrationBeyond dependency management and testing, several technical challenges arise when migrating from Java to Python:

1. Library Equivalents: Legacy applications tend to rely on the standard library and a limited set of libraries developed internally by the Client. Therefore, we can just translate those libraries. Modern applications tend instead to rely on a lot of libraries developed by others. They may have tens of dependencies and this may pose a challenge. In this case, LionWeb-Java has just a few dependencies. It uses Google’s GSON library and a few others (Protobuf, Flatbuffers). When migrating, we needed to find Python equivalents or reimplement similar functionality.
2. Overloaded Methods and Multiple Constructors: Java allows method overloading and multiple constructors, while Python does not support overloading in the same way. So we needed to merge these overloaded methods and constructors. Others may have preferred renaming the different overloaded methods and using factory methods to replace the multiple constructors. This is the kind of choice that we would discuss with the Client during the Migration Blueprint.

ConclusionMigrating every application presents some challenges: analyzing dependencies, ensuring incremental validation, and handling language differences (like support for overloaded methods or lack of it). Of course, a migration from Java to Python has some specific challenges, but the approach to deal with it remains the same: thorough analysis, structured planning, and an iterative migration process.

By generating unit tests from the original system, and gradually delivering validated components, we ensure a smooth transition while maintaining visibility into the migration’s progress. So while we demonstrate this approach on a simple, modern application, in reality, these techniques are particularly valuable for large-scale enterprise migrations, with a long of interdependencies and no test coverage.

The post Challenges in Migrating an Application from Java to Python appeared first on Strumenta.

View Details

Refactoring is part of the lifecycle of a project. When dealing with large and complex codebases this might even be the greater part of your work. Can AI help at this step? The short answer is: “eh, maybe”.

For the longer answer, you can keep reading this article. We combine our knowledge about language engineering with the one on AI to provide a good answer to this question. If you have not recently used AI-enhanced refactoring, the article might surprise you (or horrify you).

We are going to look through several examples of refactoring tasks to see where AI can help, or whether traditional methods are still better.

Our Example ProjectTo experiment with refactoring ~~our victim,~~ our test case will be a project we created in an article of ours: So Much Data, So Many Formats: a Conversion Service. This is a simple web service meant to convert files from CSV format to JSON format and vice versa. It is based on ANTLR and C#.

We are going to compare the results of:

• the existing algorithmic, deterministic methods, to accomplish a task (when available)
• a local model or, other local machine-learning based tools
• a Large Language Model (LLM), i.e., a cloud-accessible model

As the local model we are using the one currently considered the best, Codestral. For the cloud-accessible model we are using Claude-sonnet, also the state-of-the-art or close to it. For your reference, dear reader, we are writing this article in March 2025.

Renaming SymbolsContemporary IDEs offer the possibility of renaming symbols, like variables or functions with a simple click.

It is quick, reliable and it works. All you need is the ability to parse the code and resolve symbols. We need to resolve symbols because we do not want to change the name of elements with the same name or references to other elements that have the same name. For instance, in this example we want to change the name of the variable text of the constructor, but not the one of the same name of the class.

Is there any value to using AI for this? No, even a software like Cursor, an AI-based code editor, uses the basic VS Code functionality to achieve this.

This is also true for things like linting and static analysis. For the most part, the existing tools are more reliable that asking for AI to enforce a particular coding style. While AI might be easier to set up, you would be never sure if it really gets everything, It might miss or confabulating something, and you do not really want this uncertainty in a software meant to enforce standards. And we did not even mention that is slower and potentially expensive. Here, software based on language engineering techniques (i.e., parsing, symbol resolution) is still king.

The only potential value of using AI is in corner cases. For instance, if you have a project that uses cryptic names, and you want to mass rename variables to make them more meaningful.

In a small example we can see that it works quite well. Although in larger projects and variables used in different files, it is unreliable. Remember that LLMs do not perform symbol resolution, so they just act on patterns. Especially when using languages with complicated scoping rules, like Java, we would not trust the ability of AI to correctly change each variable reference. You can try it, if you are willing to move step-by-step, compiling, checking each change and repeat the process.

Making Repeated Small FixesWriting code means accepting that a lot of time you have to repeat or fix your code. You end up rewriting small bits of code. These changes are too complex for a regex, so in the past you end up doing the work by hand. Now you can speed up these small fixes.

You implement something, and then you realize you miss some details and have to rewrite some code. Making mistakes like these, does not bother me at all.

As you can see in this short video, in an IDE like Visual Studio you can activate machine learning-based suggestions, i.e., a smart autocomplete, to avoid re-typing small edits by hand.

This feature is called IntelliCode, and it is quite neat, especially since you do not need to send your data to the cloud, so there are no risks or big downsides.

Having this machine learning feature is neat. Until you need to fix 631 classes, it is helpful.

Implementing Methods If you have ever programmed in a language like C# or Java you are certainly familiar with methods used to implement object comparison. In C#, they are called Equals and GetHashCode. You probably have written a lot of implementations.

Modern IDEs can also generate basic implementations that work generally well. Sometimes you need to change them to ensure the comparison actually makes sense, but you often get lucky. Sadly, this generation is a rarity, most of the time when you ask an IDE to implement interface methods it just generates methods that return an exception.

Sure, this way at least you get the signature right and can compile the code, but it is not much.

Is there something that AI can finally help with?

In this short videos I use Cursor to generate implementations for two methods of the interface: ToFile and FromFile. The interface is a generic way to create DataItem objects, i.e., the internal representation of data in this program, and an output file.

Let’s see the ToFile method first.

And then the FromFile method.

We can compare these two implementations with the ones I have created myself.

public string ToFile(DataItem data) { JContainer container = data switch { DataArray array => ConvertToJsonArray(array), DataObject obj => ConvertToJsonObject(obj), \_ => null }; if (container == null) return string.Empty; var settings = new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore, Formatting = Formatting.Indented }; return JsonConvert.SerializeObject(container, settings); } public DataItem FromFile(Stream dataFile) { var parser = new JSONParser(); try { return parser.Parse(dataFile).Root; } catch { return new DataItem(); } } For context, keep in mind that I am using the autocomplete feature of Cursor, so not the agentic interface. So, it is the lowest level of performance and comparable to what would see a local model.

I would say that the results are mixed. The FromFile implementation is equivalent to my own, in some sense even better, given that it is checking whether the string has content before trying to parse it. So, a great result!

Instead, the ToFile method is not good. It confabulates a ToString method, that would make sense to use, if it actually existed. So, it would make sense to avoid repeating code and relying on the method that generates a string from a DataItem object. However, the method does not exist in the code. So, it is generating code that make structural sense, but it does not actually compile: it is a mediocre result, saves work, but need fixes.

Let’s see the results with the local LLM model.

public void ToFile(DataItem data, Stream outputStream){ if (data == null || !IsValid(data)) throw new ArgumentException("Invalid DataItem object"); var json = JsonConvert.SerializeObject(data); using var writer = new StreamWriter(outputStream, Encoding.UTF8); writer.Write(json);}public DataItem FromFile(Stream dataFile){ using var reader = new StreamReader(dataFile); var jsonString = reader.ReadToEnd(); return FromString(jsonString);} We would rate these results of similar quality compared to the cloud models, but with different issues. Interestingly this model does not confabulate methods. IsValid, FromString do exist in the code. The code also makes sense in both cases. However, it confabulates the interface for the FromFile method, so it loses some points there, just close to the finish line.

This is, of course, just one example. However, I am trying AI-supported refactoring for a while and these results seems to be consistent. The local model can keep its own when both local and cloud models have limited access to the code. In both cases we are just asking them to consider the local file. So, if you are privacy inclined, you do not lose much value in relying on local model for this specific task.

The problem might be the interface to access this local model, so finding an IDE that allows you to integrate a local model for autocomplete. Aside from that, it does work equally well (meh level). The situation changes the more we get close to the next use case.

Making Changes that Affect the Whole ProjectWe are now going to see refactoring that affect the overall project. These are changes that require understanding the overall structure of the program in order to be successful. They might require to write a lot of code, but not necessarily.

Some examples:

• Updating a simple dependency (i.e., a dependency that does not require rewriting much of the code)
• Separating a Web project that contains both API and client-facing interfaces
• Adding a new interface
• Adding tests to an existing project

Let’s try the first one. We are going to ask Cursor to update the outdated .NET Core 2.0 dependency to the latest version .NET 9.0. You can see the results in this video. The short version: it works, but the AI needs some prodding and help. It works better if you yourself understand the process, and can explain to it what to do, when it gets stuck.

We consider this a simple dependency since it does require any significant change to the code, only to the configuration.

This is quite illustrative of the process of working with AI, every task is a journey of discovery: of new imaginative bugs created by AI, but also of actual solutions. Jokes aside, some notes on what happened.

I had to stop the video in places to solve manually some problems. Specifically it was easier to build the ANTLR parser directly rather than let the AI figure it out. The AI was trying to mix an old version of ANTLR with the current one. You can see at around the 2 minutes mark that it was trying something that could have worked: using the MSBuild pipeline to build the ANTLR parsers automatically. I have used that in older versions of Visual Studio relying on an extension, but I do not think that is feasible anymore with the more recent Standard runtime. I do not consider this a failure: the task is unusual and there was a genuine lack of documentation in the project (it was in the original article). So, a failure was expected.

The project was also buggy from the beginning: it needs a Data directory to run correctly, which is absent in my copy of the code. The AI was able to figure it out by looking at the code and solve the problem.

The AI was also able to solve problems that it itself created, like the configuration problem requiring a redirect to an HTTPS version of the site, without setting up such version. This is a neat case to illustrate how it works: it created a standard configuration, based on its own knowledge, but that did not work for this specific case. However, it was able to figure out the issue and adapt its approach. We did end up with a working, updated project.

The AI is not omniscient, but it is able to do things by trial and error, relying on help from error messages and human suggestions.

We tried the same task a few times. It did succeed most of the time, but the code had slight differences. For instance, sometimes it used .NET 8.0 and sometimes .NET 9.0. So, it is not usable as-is in production projects: you will get with inconsistencies. You need to actually check every line of code added by AI.

We did try to do the same with a local model and the result was just failure. Partly it is an issue of lack of context. There is not a ready to use interface to provide access to a whole project using a local model. In part, it is that the buzzword of the time, agentic, in this case is actually meaningful: there are no good and effective open-source models able to act meaningfully in managing projects (i.e., compiling and react to compiler feedback). So, local models do work for solving small problems, but not changing projects.

What About Other Examples?We also tried all the other examples listed at the beginning of this section, like adding tests. We did not show a video of those attempts because the results are similar. It works better on a clean slate and with less complicated things than others. For example, it managed to create a Vue-based web interface. Even adding, completely unprompted, an about page with some interesting ideas to improve the project.

Notice that the project did effectively just support CSV and JSON, and adding other formats make sense. So, it was really a good surprise.

However, it failed in creating a Blazor-based interface. To be fair, even manually I was unable to find what was wrong exactly. Even comparing it with a Blazor project created by Visual Studio, I was not able to figure out what was wrong. So, maybe it is just Blazor that is error-prone. There have also been drastic changes in the use of the Blazor technology in the last few years, so the examples available in its training were probably confusing.

In any case, generally it does ultimately produce working code, especially if it can rely on hard, reliable information: standard error messages from compiler and other tools.

This is the best source of feedback probably because it is consistent, so the AI had seen it as-is during training. Sometimes when it got stuck we tried other solutions, such as pointing it to an article with the right approach or carefully instruct it on what to do. These solutions are less reliable since it is not always able to interpret it correctly.

Changing Foundational DependenciesWe are at the last big category of refactoring: changing foundational dependencies that requires refactoring large parts of the code. An example could be changing Object-relational mapping (ORM) libraries or large scale refactoring. Another one is migrating from Mulesoft to Spring Boot. The linked article is about this kind of refactoring.

The current best solution is to use tools like Spring Boot Migrator or OpenRewrite that implement a recipe: a sequence of rules. The rules are deterministic changes to the project, like adding some configuration setting or changing function calls. These rules are written by developers expert in both platforms.

Does AI change that? It seems that the answer is, yes but not in a straight way. Moderne, is the company that maintains OpenRewrite. So they are an interested party in this debate. However, they performed an analysis that confirm our experiments: How to use generative AI for automating code remediation at scale.

The asked some AI models to apply one of the rules you would use in these migrations. In their case is a static analysis rule, but the principle is the same: find and modify certain lines of code. The automated tool is reliable and effective. The AI sometimes get it right, especially if guided and when the human tells it where to focus:

One thing of note that this experiment confirmed for us is that AI is more successful (i.e., accurate) if the issues are identified first then provided to the AI to focus the work. As an anecdote, the AI performed better when only given the code snippet that needed to be changed.

So, this confirm what we have seen previously in the article. The AI works better when asked to perform some limited task and when provided with clear feedback. The problem is that this does not fit well with the task we are currently focusing on. Sure, changing the code correctly is important, since it requires specialized knowledge in both platforms. It is also boring and repetitive. However, finding the code to change is also a big part of the job.

To solve this problem, then, of course, Moderne offers its own AI. They trained a custom model to be better at this kind of refactoring. Their main claim is that their model is better at creating a meaningful representation of code, therefore it can work better on manipulating it. We are not in a position to evaluate this, but we agree on a specific use case they suggest:

Second, while auto-remediation with OpenRewrite recipes is the most effective way to fix source code, recipes must initially be created by developers who could use an assist from deep learning. The Moderne platform can use AI to enhance and speed recipe authorship, resulting in 100% accurate code changes. In fact, recipe authorship is a perfect use case for generative AI—with well-defined parameters and well-tested results before put into mass use.

The emphasis is ours. We would not ask an AI model to perform mass refactoring directly. However, it is a great idea to use it to create rules that govern industry-standard tools for refactoring. It is a great fit for AI since you have a well-defined output (i.e., the recipe) that a human can inspect and evaluate. Then you rely on an automated, deterministic and reliable tool to actually implement the change.

Using this approach you can improve productivity on the creation of rules without risking disaster modifying the code at scale.

The Many Kinds of RefactoringsThere are many kinds of automatic refactoring, that have been invented or used during the years. We have seen a few examples in this article. We categorize them to see if, and when, AI can be useful.

• Renaming variables and functions. Such as changing a function from getHelp to helpMe.
  • AI is not needed here.
• Making Repeated Small Fixes. For example, you want to replace a long series of nested calls to a call to one extension method.
  • Local machine learning methods work well, when available.
• Implementing Methods. Like creating the methods of an interface.
  • Both local models and large cloud models can be effective. You need to find good tools to use local models in your IDE
• Making Changes that Affect the Whole Project. Such as adding a new web interface. Or changing from using directly a library like ANTLR to build an AST with a support library like Sharplasu.
  • Your only option is to use cloud models. They work, eventually.
• Changing foundational dependencies. For instance, migrating from an outdated framework to a new one in a production application.
  • Local cloud models used directly are unreliable. They can be used to instruct automated refactoring tools that can do these changes well, though.

Refactoring Projects Written in Legacy LanguagesIn this article we have not talked about the specific case of code refactoring in legacy languages. There are two issues to discuss:

• Availability of LLMs that are trained for legacy languages
• Specific kinds of refactoring needed for legacy code

Quite simply if there are no LLMs trained on legacy languages, then no AI code refactoring is possible. This is especially true because there are often few open-source projects written in legacy languages, so you cannot do it yourself, even if you had the budget to spend on building your own.

LLMs for Legacy LanguagesThe good news is that some LLMs are somewhat trained on many legacy languages: you can find models trained on SAS or COBOL. However, we should understand what it means: these LLMs have some familiarity with the syntax of the language, but that is all.

Now, we should consider that LLMs attempt to generalize from the information they are trained on, but there is a common misconception that everything is available to these models—whether legally obtained or otherwise. In practice, training data coverage is uneven, particularly for legacy technologies.

Modern languages like Java, Python, and JavaScript are heavily overrepresented, as millions of public repositories exist for them. For legacy languages such as SAS, COBOL, or RPG, the situation is very different: there are not nearly as many publicly available codebases. Not only that: the publicly available code it is often not representative of real-world enterprise applications. We experienced that ourselves: for years we have looked for an industrial-grade ERP written in RPG having its code publicly available, so that we could use it in public demos. We did not find a single one. We could find a few toy examples or small utilities, but that was all.

This is not just the case for RPG. It is also true, for example, for SAS, where most production systems live in closed, proprietary environments. For instance, a reader familiar with the StarCoder dataset pointed out that although SAS is nominally present in the training data, it is represented by around 9,000 programs—mostly from clinical trials or tabular report generation. These do not reflect the core analytics workflows that companies actually use SAS for. So while technically “included,” such datasets are of limited value for serious SAS-related LLM tasks. The consequence is that the conception that the LLM would have of SAS code, would be one derived from looking at code that is very different from the one used in most companies. To build a proper solution one would need to have access to tens or even hundreds of millions of lines of real-world code.

If your legacy language of interest is IBM RPG, the situation is even worse, as there are no known good models available with at least some cursory knowledge of the language. Even IBM, with Granite, its open source LLM, does not support RPG. You can see in the appendix of the linked paper that SAS and COBOL are there (with the limitations discussed), but not RPG. So, you will have try cloud LLMs and see how it goes.

Personally, we tried them, with decent results on simple refactoring. The problem for large code refactorings, aside from the quality of the training, is that even with agentic LLMs the feedback cycle is broken. We have seen that AI gets things wrong, but it can fix them based on hard feedback from compilers and other tools. However, this is not possible when trying to compile things for platform like the IBM AS/400, so the results are not going to be great.

We did ask an LLM to create a program a small IBM RPG program, but the code did not compile. And we did not receive any usable feedback from the IBM tools, so we stopped there.

Specific Refactoring Needs of Legacy ProjectsThe second issue relate to specific refactorings that are more useful for legacy projects. We do not need to repeat the reasons here, but legacy program often follow poor practices like:

• cloning existing programs and make small changes
• having a lot of dead code lying around that nobody is using

Old languages have no support for project management, so it happens frequently. We think that the results are a mixed bag here. AI could probably find cloned code and even refactoring it. Assuming it clears the first hurdle of having access to the feedback of compiler tools.

We do not think that there is a need for using AI to find dead code. Understanding what is dead code (i.e., code not used by any program) requires a deterministic dependency analysis, that is better done programmatically. This is a work for your language-engineering experts, that can build and use parsers and symbol solvers to navigate the code and safely identify what is used or not.

ConclusionsAI promises to improve productivity and revolutionize coding. We agree on the first part, but we are only half there for the second.

In our experience, it can dramatically improve the creation of prototypes or adding stuff like a brand-new Web interfaces. It does not work great for refactoring, except on the small scale. And it is too unreliable for mass changes in production code.

So, for large scale changes you should use automated tool create for that specific purpose. You can use AI to instruct such tools, though. After all, configuring such tools only requires a small amount of code.

The post Automatic Code Refactoring with AI appeared first on Strumenta.

View Details

One of the challenges of working with IBM AS/400 (also known as System i) platform is that is designed following completely different principles from other contemporary systems. For one, they use a text-based interface, which now feels ancient and unfamiliar to most. These systems can be powerful, but they are certainly complex. There is no tutorial, and not easy way to discover information. You need to read a manual to use them, but even that are hard to find. In a word, they are esoteric: they work for people trained in them, but they are hard to understand for newcomers.

So, even relatively simple things, like extracting some files are hard to do: it is difficult to find the files are and to actually perform the commands to extract them. This is where this guide comes in.

The GuideNavigating an AS/400 system can be challenging for inexperienced users, and extracting files is not a common task. Therefore in this guide, we will show you how to extract files using standard tools whenever possible—tools that should be familiar to users across different platforms.

The first step is to ensure that the SSH server in the system is running.

Ensure SSH Server is Running1. Log in the system, you should see a command prompt similar to this.

1. Ensure the cursor is under Selection or command, next to the arrow and type

STRTCPSVR SERVER(*SSHD)

• If SSH is already running, you’ll see a message saying it’s already active.
• If SSH is not started, this command will attempt to start it.

Locate RPG Files in IFSRPG files can be stored in the Integrated File System (IFS) orinside the system database, but they can also be stored. Let’s start with this first case, since it is the easier one.

1. Access the IBM system with SSH. Use SSH to access the system, where is the name of your user on the IBM system and is the IP address or host name of the IBM system.

ssh <user>@<ibm\_system> 2. Locate the files. At this point you can locate the files using standard POSIX tools, like find.

find . -type f -name "*.rpg*" The previous command will just find RPG source files, but you will need also the DDS files, that describes the format of the data used in RPG programs. Look whether any .dds/.dspf files present in the directory containing RPG files. Once you have a list of these directories, you can proceed to the next step.

1. Copy the files. Exit the SSH connection or open another terminal and copy the files using SCP. You need to run the following command from your local computer, not from inside the IBM system.

scp -r <user>@<ibm\_system>:/path/to/rpg\_files /local/destination This should take care of the case in which RPG files are available in IFS.

Locate RPG Files in the DatabaseGiven that the files are located in the system database, in theory you could connect to the DB2 database using any Database Tool that supports DB2. The problem is that the files are stored line by line, similar to this.

If you have a DB administrator or developer available to create a script, you can go this route: loop through the tables, extract the lines and reconstruct the file. However, this is not convenient, so we suggest using an external tool.

Install External Tools for Easy Access and LoginNow, that you have enabled SSH, you can access your IBM system from a computer using a more familiar system (i.e., Windows, macOS, Linux).

1. Install VS Code. Install Visual Studio Code, a widely used development tool (IDE) designed by Microsoft.
2. Install an extension to support IBM i. Add the Extension Code for IBM i to VS Code. This is an extension to work with RPG, COBOL, etc. code from within VS Code.

The extension allows you to connect to an IBM i system and comes with several browsers (IFS, Object file system, etc.) to explore the files present in the system and work with them. It is mainly designed to develop code on an IBM i system, but it also allows accessing the system in a way that would be familiar to any sysadmin or developer.

Download the Source FilesWe can now proceed to download the RPG source files stored in the database.

1. Login to your AS/400 system. Follow the official documentation to fill your connection details to the system.
2. Create a SQL file. Right-click on the directory inside the IFS Browser.

Click on New File… and pick a name, for instance, hello.sql.

1. Use SQL to list all libraries. Add a SQL statement to list all libraries on the system inside the newly created file.

SELECT SCHEMA\_NAME AS LIBRARY\_NAMEFROM QSYS2.SYSSCHEMASORDER BY SCHEMA\_NAME; The select the statement and click the button (Run SQL statement) in the top right corner to execute the SQL statement.

A list of libraries will appear in the bottom panel.

1. Select each library. Now go in the Object browser panel and click on Create new filter…

A form will appear. Input the name of the first library you have found. In our example is STRUMENTA1.

1. Select the files. Clicking on the filter will update the list of files in the library. In the Object Browser you will see a list of folders for each kind of file, such as qrpglesrc for RPG files. The terminology might seem confusing: we are using the Object Browser, and what we would call source code files (i.e., RPG files) are referred to as objects or members. That is because in IBM parlance a source file the group of files inside a library. The actual files are members or objects.

2. Download the files. Right-click on the qrpglesrc (RPG source files) and click Download.

You will be prompted for a location where to save the files. Repeat the process for qddsrc (DDS source files).

1. Repeat steps 4-6 for each library found on step 3. You should now have all RPG and DDS files on your local system.

Access the TerminalYou should now verify that you have access to the Bash terminal on your IBM system.

1. Login to your AS/400 system. Follow the official documentation to fill your connection details to the system.
2. Launch the terminal. Hover your mouse on the name of your system that should appear in the bottom left corner of your VS Code.

In the previous image the system is called pub400. A menu will appear, which will include the item Terminals. Click on it and a new menu will appear on the top center of VS Code.

Select the PASE item. This would make appear the bash terminal on the lower part of VS Code. If there is no PASE item, you will have to install the IBM Portable Application Solutions Environment for i (PASE for i) on your IBM system.

ResourcesThis article provides insight into the Strumenta approach in migrating an ERP system from IBM RPG to Python based on our experience with a real project in the manufacturing industry.

This article explores the challenge will be transforming RPG code into Clear Sequence Diagrams

If you are looking for a Ready-to-go parser here you can find a tutorial.

The post Extract RPG files from AS/400 or System i appeared first on Strumenta.

View Details

Let’s be clear: Changing technology for the sake of change is pointless—it’s just a status game. If that’s your motivation, you’d be better off spending your money on a nice car. However, there are situations where outdated technology actively holds back your business, and in those cases, modernization is not an IT decision—it’s a business decision.

You should consider modernizing when your ERP or other core systems slow down your ability to act. If every time you have a new idea, your software team tells you it can’t be done, or that it’s too risky, or that it will take six months while your market opportunity lasts only a few weeks, you have a problem. Eventually, you’ll stop trying, and when that happens, the writing is on the wall for your business.

On the other hand, if your technology is ridiculously outdated but isn’t limiting your ability to operate or innovate, who cares? You can ignore it for now. But the moment it starts restricting your business, it becomes a business problem—not just a technical one. And depending on how much you care about the survival and competitiveness of your company, you should take action.

In this article, we explain how you can do that.

What is the situation out there?Being a consultant is like being a priest: you end up knowing a lot of the sins committed in your village. So I can tell you: there are a number of mid-sized companies (50–500 employees) that have an RPG system at the heart of their operations. And while it was great back in the day, now the number is going down: some companies close shop, others bite the bullet and move to a new system (more or less painfully). It may feel like being among the last survivors: you turn around and notice there is one less of us left.

While you would like to think about something else, the fact is that those systems are hard to change, and finding people who know how to work on them is a nightmare. They also happen to be expensive, but that is just the cherry on top. You keep patching things up, hoping to see another day. One day, you are sure, we will get the time to think about a more permanent solution.

A simple metric (that you may not like)Modernization shouldn’t be “IT for IT’s sake” – it’s not about chasing trends or keeping up appearances. It has to serve your business. The real question: is your tech helping you move forward, or is it just something you keep pouring money into because you don’t know how to escape it?

If you’re running on an old COBOL or RPG system, chances are 60–80% of your IT budget is just keeping the lights on. That’s like spending most of your paycheck every month just to cover interest on old debt—you’re not getting ahead, you’re just stuck. And at some point, skipping town seems a tempting option. That’s a dangerous place to be.

Tech should be an advantage, not a financial black hole. If you’re only maintaining what you already have, you’re playing defense instead of offense. The IT budget should be about helping you move faster, making you more agile than competitors, and giving you the edge that made your business successful in the first place. If every time you need a new feature, your team tells you it’s too risky, takes six months, or just plain can’t be done, then your technology isn’t just old—it’s holding you back.

Modernization, when done right, frees up budget and brainpower for things that actually push the business forward. Whether it’s integrating a cloud CRM, giving sales teams better tools, or making operations run smoother, every IT move should support a real business goal—scaling up, responding faster, or staying ahead of the market. And when leadership sees modernization actually cutting costs and opening new opportunities, they’ll get behind it.

Bottom line? Tech should be your secret weapon, not your ball and chain. If your IT spending is all about survival instead of growth, I would humbly suggest that you’re doing it wrong (I know, easy for me to say).

The Real Problems with Legacy SystemsOkay, we’ve painted the big picture—modernization isn’t about chasing trends, it’s about keeping your business agile and competitive. But let’s get specific. What are the actual problems you might be dealing with your legacy system?

We will discuss it, making explicitly reference to the classical situations of RPG-based EPR or MPR systems, but the situation is very similar if you are using EGL, some long-forgotten 4GL, or maybe Visual Basic 6.

1. The Code is a Mess

Nobody really knows how it all works anymore. Over the years, these systems have been patched, extended, and customized—often by developers who are long gone. Documentation? If it exists, it’s outdated or incomplete. Yes, there are low-level comments on single functions (not always aligned with the code), but the overall picture is not captured anywhere. That makes every change a high-risk move—because when one part of the system depends on another in undocumented ways, a small tweak can trigger unpredictable failures. So the default answer of the technical team becomes “It cannot be done,” and if you keep pushing, they shoot you a crazy estimate, so that you will just leave them alone.

2. Finding RPG Devs is Like Finding a Unicorn

Where are all the RPG developers? Most of them are retiring or have already left the workforce, and almost nobody is learning RPG today. That leaves you dependent on a shrinking pool of experts, driving up costs and increasing the risk of losing critical business knowledge.

The reality is that the average RPG developer is over 55 years old, and most young developers have never touched RPG. That means every time a key RPG expert leaves, maintaining your system gets even harder. Companies running RPG-based systems consistently report hiring challenges as their #1 modernization blocker—not because they don’t want to update their systems, but because they don’t have the people to do it.

Sure, you could convince people to join your company without knowing RPG and then teach them. But does it seem an efficient and sustainable thing to do for years to come? Are the brilliant young developers out there dreaming of learning and working with RPG?

3. They Don’t Play Well With Others

Trying to connect an RPG-based ERP to modern cloud platforms, mobile apps, or external APIs? It’s a constant battle. These systems weren’t designed for integration, meaning every connection requires custom-built middleware, brittle workarounds, or slow batch processes. It made sense at that time to have “one system that just worked”, but today software is more pervasive, and you want to add more “stuff” and plug or replace components over the years. That was something legacy systems were not designed to support.

Many companies still using RPG struggle with green-screen interfaces and outdated data structures, making automation and real-time integration nearly impossible. And in today’s business world—where APIs and cloud services are the norm—running an isolated system is a major disadvantage.

4. They’re Not Flexible

Want to add a new feature? It’s going to be a struggle. The codebase is a mess: it’s all connected, and no one knows how it works. Even small updates take ages, or get put on hold because they’re too risky. This means the business slows down, all new ideas are just deemed “impossible to implement”, and the competition gets ahead.

A survey of IT leaders found that 70% say their legacy applications prevent them from quickly adapting to market changes. And when making a change takes six months while your competitors can try out things in weeks, learn and refine, you’re no longer playing on a level field.

5. Waiting Just Makes It Harder

If you don’t act now, the problem will only get worse. Every year, RPG knowledge becomes harder to find, system complexity increases, and the cost of migration goes up. This would be a good point to add a sentence to soften the news and paint a more rosy picture, but I cannot come up with anything. Sorry.

Ways to Actually Fix These SystemsThere’s no magic wand that instantly modernizes an RPG-based system, but there are strategies that have been applied and worked well. Let’s take a look, and hopefully they will be of inspiration.

Strategy 1 – Automated Code ConversionFor companies looking for a more direct transition away from RPG, automated code conversion tools can provide a head start. These tools translate RPG code into modern languages like Java, C#, or Python while preserving the original business logic. While this doesn’t create fully optimized modern applications, it removes the manual effort required for migration. After the translation, you can refactor the solution over time, if you wish.

The biggest advantage of this approach is speed—instead of rewriting millions of lines of RPG code from scratch, companies can use automated tools to generate an initial version of the new system, which can then be refined over time. This can reduce migration timelines from several years to just 6–18 months, depending on the complexity of the system.

However, automated conversions are not perfect. The resulting code is often functional but not optimized, meaning that developers will still need to refactor, clean up, and validate. The good news is that now that the system is in a more modern language, you can find talent, and you have access to better development tools.

Strategy 2 – Make it more modular through APIsOne of the best ways to make an RPG-based system more useful without rewriting it is to wrap it in APIs.

In practice, we mean taking your existing RPG system and breaking it up into modules that communicate between them through clearly defined (and documented) APIs. This will permit to build new components that replace some of these “RPG silos” or that provide additional features.

This will not completely solve the problem: you will still have a lot of RPG around, but over time you could replace those components, and in the meantime, you can build new applications that integrate with RPG through the APIs.

While APIs can extend the lifespan of a legacy system, they don’t eliminate the underlying maintenance burden—at some point, core modernization will still be necessary.

Strategy 3 – Off-the-Shelf Replacement (e.g., SAP Business One) with Business Logic ExtractionAnother option is to throw everything away and adopt an off-the-shelf solution like SAP Business One, then recover the business logic from the existing system and use it to configure the new one. This is a radical approach, and while it may seem like a clean break, it comes with two massive challenges that make it both time-consuming and expensive.

The first challenge is recovering the business logic. Your RPG-based ERP has likely evolved over decades, with business rules deeply embedded in its code, intertwined with system-specific logic, and lacking clear documentation. Unlike modern systems where processes are neatly mapped out in workflows, many RPG systems operate on implicit rules that only exist inside the code or in the heads of long-time employees. Extracting and formalizing this logic is a significant undertaking. This is something we’ve been working on recently, and we may be able to help—but for now, let’s be honest: it’s a challenge.

The second challenge is translating that business logic into the new ERP system. Once you’ve successfully extracted and documented the processes, you then need to work with SAP (or other ERP) consultants to implement them correctly. This means extensive configuration, customization, and testing. And, as anyone who has gone through a major ERP migration can tell you, these projects are long, expensive, and difficult to get right.

Realistically, you should expect at least 12 months, and more likely between 18 and 36 months, to complete this process. The time required to extract the business logic can be significantly reduced with the right techniques, but as far as we know, the implementation and configuration of a new off-the-shelf ERP cannot be compressed much—it’s inherently a slow and methodical process.

This approach makes sense if your current system is truly unsalvageable and you are already committed to transitioning to a modern ERP platform. However, it’s also the most disruptive and expensive route. For many companies, it’s worth exploring whether incremental modernization (via APIs, cloud migration, or modular replacements) can achieve the same goals with less risk and cost.

Strategy 4 – Aggressive Modernization While Keeping the Legacy SystemIf moving away from RPG entirely isn’t an option, you can still modernize aggressively within the constraints of your legacy system. This means not just keeping the system running but improving how it is structured, maintained, and extended. While this won’t solve all the fundamental problems of legacy technology, it can significantly reduce maintenance headaches and make the system more appealing for new RPG developers to work on.

There are several steps you can take:

1. First, if you’re still using fixed-format RPG (or even RPG II or RPG III), you can migrate to RPG IV in free format, making the code more readable and maintainable. This alone can make a huge difference in how easily new developers can understand and modify the system.
2. Second, instead of relying on physical and logical files for data storage, transitioning to a modern SQL database brings better query capabilities, reporting, and integration potential.
3. You can also eliminate dead code and reduce code duplication, both of which tend to accumulate over decades, making maintenance much harder than it needs to be.
4. Another key improvement is introducing unit tests, which are rarely found in legacy RPG systems. Unit testing makes it safer to refactor and improve code over time without introducing unintended bugs. It also helps document expected behavior, which is particularly valuable when dealing with old, undocumented business logic.

This approach has two major advantages. First, it makes the system more maintainable—cleaner code, structured data, and automated tests reduce the risk of breaking things every time you make a change. Second, it helps with hiring and retaining RPG developers. A modernized RPG system—one that’s easier to read, structured properly, and equipped with testing—is simply more pleasant to work with. Given the difficulty of finding RPG developers, anything that makes the system less painful and more attractive to work on will make recruitment and retention easier.

This isn’t a cheap solution, nor does it solve the bigger issues of RPG developer scarcity or IBM i licensing costs. However, it can be done incrementally, meaning you can modernize parts of the system over time with limited risk. While it won’t make RPG a cutting-edge technology, it will at least make your legacy system a modernized, structured, and more manageable version of itself, buying you time and reducing operational pain. Also, if at some point you decide to migrate, these improvements will also make an automated conversion easier to implement.

Common Challenges During Modernization (and How to Mitigate Them)Modernization projects always hit roadblocks. Here’s how to handle the common ones.

Legacy Code Complexity & Lack of DocumentationAs noted, many legacy systems have sparse documentation – or, let’s be honest, just no documentation at all. And code written in the 80s and 90s. It can be hard to understand the exact business rules buried in legacy programs.

Mitigation: Start with a thorough discovery and analysis phase. Use automated code analysis tools to map out program flows, dependencies, and even automatically document the existing codebase. Engaging veteran RPG developers (if available) to knowledge-transfer is vital as well. By creating a detailed picture of what your system does, you reduce the risk of surprises during migration. By mapping out every interaction and dependency to plan legacy modernization precisely you can reduce your risks. In short, invest time up front to understand your legacy system before you change it. If you want help in organizing this discovery and analysis phase, this is something we have experience with.

Migration Risks (Downtime, Data Loss, and Project Failure) Modernizing a core business system is akin to open-heart surgery – there are risks of downtime or even failure if things go wrong. If you do a full replacement, the business could be disrupted during cutover. Moreover, as cited earlier, large-scale rewrite or replacement projects have high failure rates.

Mitigation: Favor incremental and phased migration over “big bang” go-lives. For example, run the old and new systems in parallel for a period, or replace one module at a time (as in a microservices approach). Have robust backup and rollback plans so if a deployment goes awry, you can revert to the legacy system. Rigorous testing is non-negotiable – not just unit tests, but integration tests and user acceptance tests with real scenarios. One best practice is to create a comprehensive test suite against the legacy system’s known outputs and run it against the new system to catch any deviations. Not trivial to pull out, but it helps to sleep better at night.

Cost Overruns and Scope CreepModernization projects get expensive fast, and the quickest way to blow your budget is by trying to do everything at once. If you go into a project thinking, “Let’s rewrite the system, clean up the code, rethink the business logic, modernize the UI, and improve performance”, you’re setting yourself up for a never-ending project.

To avoid this, set a clear goal from the start and stick to it. If the goal is getting off RPG and onto a modern architecture, focus on that. If it’s replacing the UI while keeping the business logic, don’t suddenly decide mid-project to refactor everything. Every additional change—rewriting logic, reorganizing databases, optimizing performance—should be done later, in a separate phase. Trying to cram everything in at once is how projects stall indefinitely.

You also need a real way to measure progress. Don’t trust anyone who promises a fixed timeline—those are usually best-case scenarios that never hold up in reality. Especially if the consultants you are working with are paid by time and material: they get financially compensated by getting late. Who ever thought that was a good idea? In our case, for many of our engagements (including the Migration Blueprint and Migration CodeCraft) we offer a fixed price. This ensures it is in both our interest and the Client’s interest that we complete the project as soon as possible.

Instead, track actual work completed—how many modules have been migrated, how many features are done, and how much legacy code is still in use. The sooner you notice a slowdown, the sooner you can fix the problem before it derails everything.

The best way to keep things on track? Break the project into small, manageable phases. First, migrate the database and core logic. Then, work on the UI. Then, clean up and refactor. Each phase should have its own budget, timeline, and deliverables, and if something starts taking longer than expected, adjust before it snowballs into a disaster.

In short: don’t try to modernize everything at once. Pick a goal, track real progress, and keep the project manageable. That’s the difference between a successful modernization and one that drags on forever. Or at least until you run out of money or patience.

Cultural Resistance and Change ManagementPeople-issues can make or break modernization. Employees who have used the RPG system for decades might resist new tools or fear job loss. IT staff might be defensive about the old system (after all, it’s been their life’s work).

Mitigation: Change management should run in parallel with technical work. Communicate the “why” of the project clearly to all levels – emphasize that modernization is about securing the company’s future (and their jobs), not nullifying past work. Acknowledge the value the legacy system provided, but highlight how business needs have evolved. Involve end-users early by soliciting feedback on pain points and wishlists for the new system – this inclusion helps gain buy-in because people feel heard. Provide training and support as new tools roll out, so users feel confident, not frustrated. It also helps to identify internal champions – respected employees excited about the change who can evangelize to their peers. Lastly, celebrate small wins: when a new feature goes live, or a milestone is hit, publicize it and recognize the teams involved. That builds momentum and turns skeptics into supporters over time.

Conclusion: Practical Recommendations and Next StepsEmbarking on a modernization of an RPG-based ERP/MRP system is undeniably complex, but it is also highly rewarding when aligned with your business’s long-term success. By addressing both the technical and strategic facets, you turn an aging system from a liability into a competitive asset. Let’s recap and outline some actionable next steps:

Step 1 – Assess Your Legacy Landscape

Start with a frank assessment of your current RPG system. Inventory all modules and their business importance, analyze the code quality (possibly using automated scanners), and identify critical pain points and how well the system can support your future business needs. This step often reveals quick wins and informs the priorities of larger initiatives.

Step 2 – Define a Modernization Roadmap

Map out a multi-phase roadmap based on the assessment. Prioritize projects that address the biggest risks and offer the highest business value. Ensure each phase has clear business-aligned goals. This roadmap translates business strategy into tech milestones. Keep it flexible; it can evolve as you learn from each phase and as business conditions change.

Step 3 – Secure Buy-In and Budget

Use the roadmap and business case to get official buy-in and budget. Articulate the cost of doing nothing and the benefits and ROI of modernization at each step. With executive sponsorship and allocated budget, you have the mandate to execute the plan with authority and adequate resources.

Step 4 – Build the Right Team and Partnerships

Put together a modernization team that blends institutional knowledge with fresh skills. This could mean upskilling internal staff, hiring new talent, and engaging external experts. Assign a strong project manager to coordinate between IT and business stakeholders.

Step 5 – Execute in Iterative, Value-Driven Increments

Approach modernization projects in an agile, iterative manner. Deliver smaller increments every few weeks if possible, every few months otherwise, and gather feedback. Each increment should deliver tangible value or learning, which reduces risk and allows the business to start reaping benefits early.

Step 6 – Focus on Change Management Throughout

Parallel to technical work, keep executing your change management plan. Continue updating everyone on progress, training users for upcoming changes, and capturing feedback. Monitor adoption of new tools and be ready to adjust.

Step 7 – Measure, Adapt, and Evolve

Establish metrics to track the success of modernization against your business goals. Report these wins and be candid about any shortfalls. Use a continuous improvement mindset. A tech strategy aligned with business goals is not a one-and-done project; it’s an evolving program. As new technologies emerge, incorporate them into your roadmap where relevant to the business.

By following these steps, a company with a legacy RPG-based system can methodically transition to a modern, agile technology footing. It’s a journey that addresses immediate risks (like legacy maintenance costs and developer shortages) while setting the foundation for long-term growth and innovation.

I am not going to sugarcoat it: the challenges of legacy modernization – from decoding old code to managing change – are significant, but they are far outweighed by the risks of standing still. Clinging to outdated systems eventually holds back business initiatives, whereas application modernization removes obstacles to progress. In plain terms: modernizing your tech is investing in your business’s future.

What you want to do is to build Long-Term Success through Alignment Between your Technology and your Business Needs. Doing business is difficult under the best circumstances, and your technology should be a catalyst, not a constraint. Legacy RPG systems served well in their day, but the future belongs to flexible, integrated, and intelligent systems. By undertaking a thoughtful modernization – and openly addressing challenges like legacy code maintenance, the RPG developer shortage, and migration risks at each step – you position your company to thrive for the next decade or two. The road isn’t easy, but, is the alternative any better?

ResourcesYou can read this article, to discover more about The Migration Blueprint and this article to discover more about The Migration CodeCraft.

You can discover more about migrations here and get in touch with our team of experts!

The post Aligning Tech Strategy with Business Goals: Modernization for Long-Term Success appeared first on Strumenta.

View Details

Do you want to know more about an incremental parsing library used at GitHub and that was born out of a failed software? Then, keep reading because we are going to talk about Tree-sitter.

Tree-sitter is an incremental parsing library, which means that it is designed to efficiently update the tree, without throwing away the work already done. This ideal for parsing a file that is being written in a code editor. If you are interested in knowing more about GLR, the kind of algorithm that powers Tree-sitter, you can read about our introduction to parsing algorithms. That explains what incremental parsing is.

Tree-sitter was created as part of Atom, the editor by GitHub that battled with VS Code and Brackets, a few years ago, for the title of best lightweight, cross-platform editor. As you know, Atom lost and VS Code won. However, Tree-sitter survived and it is still used at GitHub.

Tree-sitter is used at GitHub and creates efficient parsers. It also comes with support for syntax highlighting. So, it looks like a great fit for an editor. Here at Strumenta, we are fan of ANTLR, a robust and flexible parser generator. However, we found that Tree-sitter has its merits and in this article we are going to show you where it shines and how to use it.

For instance, Tree-sitter:

• has excellent performance
• comes with included support for syntax highlighting
• there are parsers available ready-to-use in many languages

In this tutorial, we are going to see:

• how to create a parser in Tree-sitter
• how to define rules to support syntax highlighting
• how to identify elements and tag them
• how to use the generated parser in Python and do things like walking the tree

You can find the repository with the code for this example at tree-sitter-story.

What is Tree-sitter designed for?This is how the official documentation introduces it:

• General enough to parse any programming language
• Fast enough to parse on every keystroke in a text editor
• Robust enough to provide useful results even in the presence of syntax errors
• Dependency-free so that the runtime library (which is written in pure C11) can be embedded in any application

Tree-sitter was tailored for GitHub needs in supporting an editor. So it is a great software, worthy of such a company, but it has some peculiarities. It has great performance, but a bit of a complex setup. Tree-sitter is written in Rust, but the parsers are generated in C. Do not worry, you do not actually have to write C to use it, because you are supposed to use JavaScript to write grammars.

It is a parser generator tool: you write a grammar (in JavaScript) describing a language and then Tree-sitter (written in Rust) creates a parser (in C or WebAssembly) to handle that language.

You then use the parser in your favorite language, as long as there has bindings available for this language. There are many bindings available, official (Go, C#, Java, Kotlin, etc.) and community-supported. Bindings are essentially a bridge between two programming languages. You end up in an interface (for instance in Kotlin) allowing you to use a library or code written in one language (like C) naturally.

| What | Language Used | When You Should Care | | --- | --- | --- | | Tree-sitter itself | Rust | If you want to contribute to the project | | Grammar | JavaScript | If you want to create a new parser | | Generated Parser | C compiled to native library/WebAssembly | If you want to run the parser | | Client libraries | All that have bindings available | If you want to use the parser |

You can see look at the table for getting a summary of what you should care about for what you want to accomplish. The most important parts are relative to writing the grammar and running the parser.

You need to use JavaScript to write the grammar. Tree-sitter will generate C code and compile into a native library. If you want to use a parser, you do not need to care about JavaScript, you need to take care that the running environment can run the generated parser. If you are compiling the generated C code to a native library, you need to do that for the platform you care about. For instance, if you want to use the parser on macOS, you need to compile the code on a macOS system in order to generate a .dylib library.

You could also generate a WebAssembly (WASM) module to run in any platform that can handle WebAssembly. Usually these platforms are browsers, but there are other options.

It is written in Rust because that is a productive language. It outputs C, because that is a widely supported language. This allows to embed it in many software and define bindings for a variety of languages. It is also possible to consume WebAssembly, that allows to easily connect JavaScript and C code.

So, it is complex software that works well if you want to build editors. It has the performance necessary to be used in compilers and interpreters.

It is also used in static analysis: it provides parsing to implement partial symbol resolution in the online code viewer on GitHub. For instance, if you open a Java file on GitHub and select a property you can see the references and potential definitions on the sidebar. This system uses Tree-sitter based parsers.

This makes sense for GitHub because they are a company providing web software (hence the JavaScript) which also needs the performance of system software (hence the Rust/C).

Using Tree-sitterNow that we understand what is Tree-sitter for, we can start using it. There are actually several ways to take advantage of it. If you are interested in just using an existing parser, you can use a library in many of the languages for which there are bindings. The tree-sitter organization on GitHub has parsers for many languages. As a side note, they are both official languages and community-provided ones.

This is a good reason to use Tree-sitter:

• there are parsers available for many languages
• if you write a parser in Tree-sitter, you can process your parser and all other languages supported by Tree-sitter in one pipeline

However, take notice the quality and integrations with the respective ecosystems varies, for instance the Python binding works, but the Kotlin one it does not work well. When we tried it, they were not generated.

If instead you want to build parsers, you need to use JavaScript.

So, let’s start by installing the tree-sitter-cli globally via NPM.

npm install -g tree-sitter-cli There are other options, including downloading the binary from GitHub or using a Rust crate, but this is the easiest way to install the software.

You can now:

1. create a folder for your project, like tree-sitter-story for this tutorial
2. create the project with tree-sitter init. The first time you run it on an empty directory it will ask you some questions to create the project. Subsequent times will do nothing because it detects that a project is already there

Running the command you will see something like this.

Once you run the command, you will end up with something like this.

A folder with all the files to run, test and compile the parser.

Exploring a Tree-sitter ProjectIt is useful to look around this newly created project, since the CLI relies on this project structure to fulfill its role. One thing that is missing from the generated project is a test/corpus folder. This should contain example files to validate your parser.

You can see a lot of language-specific build files, like CMakeLists.txt for C/CMake or pyproject.toml for Python. The C-related ones will build the main parser, while the other ones handle the bindings. You can control which bindings are created by going into the file tree-sitter.json and changing the bindings values.

"bindings": { "c": true, "go": true, "node": true, "python": true, "rust": true, "swift": false, "zig": false, "kotlin": false } For instance, if you set swift to false and delete the Swift-related files, they will not be there when you run the command tree-sitter init --update. This does not seem to work perfectly though, because we tried to set kotlin to true, but nothing happened. You also cannot remove Rust or C.

The main project however is the JavaScript project, that relies on package.json. The file sets up the running of the tests and even to run the parser in sample editor (the command start).

You can do directly with the CLI, but the JavaScript project makes it easier. The main file you care about is grammar.js, which will contain your grammar. The default project has a grammar for a hello language, literally a language that accepts hello.

All you need to do now is to install the necessary dependencies with npm install.

The Tree-sitter WorkflowThe CLI creates a working setup with a usable language, this means that you can already experience the basic Tree-sitter workflow:

1. run tree-sitter generate to create the C parser under the src/ folder.
2. run npm test
3. run npm start to launch a web editor to test your parser

Running the test and the web editor requires a C compiler and Emscripten (emcc) to compile the code to WebAssembly. If you have installed Docker on your system, running tree-sitter build --wasm will automatically pull and create a docker image that will contain Emscripten and will generate a WebAssembly module. If you are using Windows, it is better to use Windows Subsystem for Linux (WSL) to work with tree-sitter projects.

This is quite a simple workflow for such a powerful system. So, that is good. You could also run the command tree-sitter parse example.story to test the parser on a file. This would produce a parse tree, even if errors are present in the example file. This is useful to check that there are no errors, and also save you from the need to have Emscripten on your main development computer. However, it is complicated to analyze parse trees in terminal.

The playground web editor requires you to compile the generated parser in a WebAssembly module. So, if you cannot compile to WebAssembly in your development machine you will be forced to debug the grammar and explore the parser tree looking at the terminal.

Compare this view (from the terminal).

With this one (from the editor).

It is much more readable, and you can also click on the parse tree elements to highlight the source code that corresponds to them. In case you are wondering, the editor used for the playground is codemirror. You can use this editor by running tree-sitter playground, but you need to have a working grammar and generate a WebAssembly module.

Writing a GrammarWe now know how to use the software, how can you write a grammar? The short version is like in a parser combinator, but with a better organization. The longer version is the following section.

You write rules similarly to what you would do using a parser combinator, which is a software that provides parsing functions that you combine to parse things.

module.exports = grammar({ name: "story", rules: { source\_file: $ => seq( $.title, $.characters, $.inputs, $.constraints, $.text, $.question ), For instance, the start rule is source_file, which is also the first one you define. You can see that in our example the rule defines a seq(uence) of rules. This means that a file in our language must have all these elements in the correct order to be valid. The elements prepended by $. like $.title are a reference to other rules. What happens if the file is not valid?

You get an error node, but the parsing continues unabated.

Parsing ListsYou can combine functions like seq, optional and repeat to parse something like a list of identifiers:

one, two, the_unbeareable_sadness_of_using_c

This is what they do individually:

• repeat parses zero or more of its arguments. If you want to parse at least one or more, you need to use repeat1.
• optional indicates that that part of the rule is optional

You can combine them with seq to parse a list in the rule. We do just that in _list, that we use when parsing the list of characters in our story and other rules.

characters: $ => seq( 'characters', ':', $.\_list ), \_list: $ => seq( "(", optional( seq( $.expression, repeat( seq(",", $.expression) ) ) ), ")" ), The rule _list shows how it is done:

• we start with seq, because in tree-sitter a rule cannot match an empty string, so we start the sequence with delimiters like square brackets or parentheses. If you do not want delimiters you can make the list always contains at least one identifier. Basically, just the part between lines 8-12
  • this parses
    • (
• we continue with optional, because the list could be empty
• we then use seq
• we continue with $.identifier, on line 8, because we need to match the first element
  • this parses
    • one
• then we use repeat because the first element can, but it does not have to, be followed by other elements
• we use seq, on line 10, because these other elements are a sequence of identifiers, each separated by a comma
  • this parses
    • , two, the_unbeareable_sadness_of_using_c
• we finally match the ending parenthesis
  • this parses
    • )

It seems complicated at first, but once you get the hang of it, combining rules became easy. And the availability of many ready-to-use grammars gives you the chance to see how to parse real programming languages.

One interesting think that we noticed is that Tree-sitter is stringent on forbidding to write rules that can match an empty string. When using a parser generator like ANTLR you may receive a warning about such rules, in Tree-sitter you cannot write rules like that all.

Parsing ExpressionsAnother common feature that you want to parse are expressions.

\_expression: $ => choice( prec(1, seq('(', $.\_expression, ')')), $.identifier, $.number, $.binary\_expression ), binary\_expression: $ => choice( prec.left(2, seq($.\_expression, ">", $.\_expression)), prec.left(2, seq($.\_expression, "<", $.\_expression)), prec.left(3, seq($.\_expression, "/", $.\_expression)), prec.left(3, seq($.\_expression, "*", $.\_expression)), prec.left(4, seq($.\_expression, "+", $.\_expression)), prec.left(4, seq($.\_expression, "-", $.\_expression)), ), The main rule to accomplish that is _expression. It starts with the function choice, which means that the rule matches one of the options available. The first option is for parsing an expression between parentheses. That is what the seq('(', $._expression, ')' is for.

You can parse combined expressions in different ways. Imagine you want to evaluate an expression like:

1 + 2 * 3 Assuming you knew nothing about mathematics you could try to evaluate the expressions in the order that they appear, so 1 + 2 and then multiply the results for 3. However, we know that we must something different: you first need to evaluate the multiplication and then the addition, so you multiply 2 * 3 and then add 1. In other words, there is a specific order in which the expressions need to be evaluated, and your parser must be able to handle that.

This rule shows how you can do just that: it is wrapped in a prec method. There are also a prec.right and a prec.left method that could be used. The prec method indicates a precedence between expressions, the lower the first argument, the higher the precedence is. Higher precedence means that is picked before options with lower precedence.

If you do not use this method, the precedence for all options is set to 0. So, the way we have written the rules, it means that parentheses would have a lower precedence than other alternatives. This has no effect on this particular case, since the option with parentheses is not ambiguous with respect to any other option.

The binary_expression rules employs the method prec.left, which combines precedence with associativity. Associativity determines in which order expressions using the same operators are evaluated. For instance:

1 * 2 * 3 Do we mean (1 * 2) * 3 or 1 * (2 * 3)? By picking left associativity we pick the first interpretation. It is a similar concept to order of precedence but determine the order for expressions of the same type.

Most common mathematical operators are left-associative, but exponentiation is right-associative.

Lexing RulesBefore moving on to see the last interesting feature of our grammar, we need to see how Tree-sitter implements lexing: it does it with regular expressions, the end.

Ok, we need to tell something more. A parser rule tells the parser how the input is organized. For instance, what sequence of elements defines a list. However, the parser does not handle the source code directly. The parser handles tokens. The lexer transforms the source code into tokens using its own rules, the tokens are then evaluated using the parse rules.

Imagine this process applied to a natural language such as English. You are reading the single characters, putting them together until they make a word, and then you combine the different words to form a sentence. Words in English are like the tokens that are analyze to form the sentence (parser rule).

For instance, the sequence of character 91 becomes the token number, the sequence of characters if becomes a token if.

This is what you need to know about the difference between lexing and parsing to write Tree-sitter parsers, if you want to go deeper, we recommend reading our introductory article about parsing.

The way Tree-sitter defines tokens is with strings or regular expressions. We have already seen how you can use strings, like parsing expressions. We have indicated the tokens for operators (like ">" or “+”) directly into the parser rules.

This works well for tokens that you are going to use just once. For frequent tokens you want to define them separately.

For instance, this is how you create tokens for identifies and numbers.

identifier: $ => /[a-zA-Z\_]+/, number: $ => /\d+/ You can use regular expressions, as you would in regular Rust. Yes, this is not a typo. You are defining using the JavaScript syntax, so they are wrapped in slashes, but they are actually powered by the Rust regular expression engine.

This is relevant, because there the Rust engine has some limitations compared to that of JavaScript. In addition to that, they are far more stringent limitations related to performance needs.

You can only use regular expressions in the computer science sense. By that we mean that they follow the computer science definition of a regular language, so they do not support features like lookahead and lookaround. The short version is that you cannot define regular expressions that select elements based on other parts of the string, such as “find the second comma in this string”. We point you to the same introductory article we mentioned before, if you want to know more about what regular language means in computer science.

This is a design choice by the Tree-sitter designers to guarantee a certain level of performance. In fact, Tree-sitter grammars are LR(1) grammars, which means that the parser should be able to decide which rule to match only looking one symbol ahead of the current one that the parser is analyzing. This puts some restrictions on the rules you can write and this is one of them.

Another peculiarity of Tree-sitter lexing is that whitespace, like spaces or newlines characters, are already handled by the engine. So, you do not need to create specific rules to handle them. And what about elements where handling whitespace is necessary, such as indentation in Python? You need to define an external lexer (also called scanner) in C, using the Tree-sitter API.

This is quite different from a parser generator like ANTLR, where you can add custom code in any target language supported by ANTLR. Both methods have advantages and disadvantages:

• with ANTLR, you can write custom code in any language, but this also means that the whole grammar is not language-independent anymore. So, if you write custom code in Java in your grammar, you can only generate a parser in Java for your language
• with Tree-sitter, you must use C, which requires you to be proficient in it. On the other hand, the parsers are always generated in C, and interoperability with other language is achieved through bindings

Effectively this means that Tree-sitter is a perfect fit, only if your organization is already using C or it can handle it. For such users, Tree-sitter is a quicker and more efficient way of creating parsers usable in many languages.

Now, we can go back to looking at the parser.

A Simple Way to Parse Interpolated Strings \_inner: $ => choice( field("text", token(/[ \r\n\.\,\?a-zA-Z]+/)), field("vars", $.interpolation) ), string: $ => seq( '"', repeat1( $.\_inner ) , '"' ), interpolation: $ => seq( '{', $.identifier, '}' ), Our language also enables string interpolation. We use the rules string, _inner and interpolation to achieve that. The rule string does not use any new parsing function. We are parsing a sequence of elements starting with the double quotes and ending also with double quotes. In the middle there are one or more elements captured by the rule _inner.

This rule starts with a choice, so it matches one of two alternatives. The function field does not affect parsing per se, but the concrete syntax tree created by a tree-sitter parser. It allows us to give a specific name to each element matched by the subrule. In this case we name text the free text between the double quotes and vars the interpolated element.

There is another interesting part of this rule: we use the function token. Usually Tree-sitter create a separate token for each match of a token rule. For instance, the regular expression that is the argument of the function token matches all text, including words and symbol. If you put into a separate rule Tree-sitter will create separate text tokens between each interpolation.

By using the token method, instead we create just one text token.

The token method accepts only terminal rules, so you cannot reference other rules. You need to define everything you need inside the token method.

Finally, the rule interpolation is just a sequence of one identifier wrapped between two angle brackets.

The Word TokenTree-sitter uses context-aware lexing, which means that it performs lexing on-demand, during the parsing process. It tries to match the current input only with tokens that are valid in the particular context is parsing at the moment. That makes it more robust than your typical lexer technology, but this is not necessarily a good thing.

The problem becomes evident when you think about the following example. The language C# has an is operator, which checks whether an expression is compatible with the specified type. For example, the following code checks whether the variable c is a string.

c is string So, writing this code is invalid.

c isstring That is because a traditional lexer would match isstring as one identifier. Thanks to its context-aware approach, instead, Tree-sitter would correctly match both is and string. This makes some invalid code looks valid.

You can avoid this issue using a feature called keyword extraction.

You would first use the property wordto identify a specific token before defining the rules.

module.exports = grammar({ name: "story", word: $ => $.identifier, rules: { // .. identifier: $ => /[a-zA-Z]+/, } In this case, we use word to identify the rule identifier. This will tell Tree-sitter that it should consider all literal tokens as keywords. For instance, given this rule:

question: $ => seq( 'question', ':', $.string, ':', $.\_expression ), The literal question would become a keyword. During lexing, Tree-sitter will try to scan for identifier before trying to match the keyword and therefore would match isstring rather than is. This would allow it to correctly recognize the code as invalid.

Parsers Built For EditorsAn interesting thing to note is that every rule that starts with an underscore will be invisible in the parse tree. We have seen examples like_list and _expressions. This allows you to both create a well organized grammar, but also make sure that the parse tree is still readable. This is a small feature, but it is a nice one, if you are building an editor.

You typically want to use this feature for repeatable fragments, like defining a list, or for nodes that capture a single choice among many options, like an expression, which can be one of many kinds of expressions.

This is not the only feature designed for editors. Tree-sitter comes with support for syntax highlighting and tagging elements. Both of these rely on defining rules inside specific files using query patterns.

You can think about tagging elements as a lightweight form of symbol resolution: you are going to define patterns to power a feature such as Go To Definition.

You are going to define these patterns using the query language: you are going to describe the shape of a tree that matches what you want. For instance, you are going to say that all identifiers inside an interpolated string are references.

Highlighting is IncludedAll you need to implement syntax highlighting is defining patterns and add a setting, Tree-sitter will do the rest. Let’s start with the setting to be added in tree-sitter.json, in the grammars section, next to injection-regex property.

"highlights": "queries/highlights.scm" The path, name and extension of the file are the standard values for a Tree-sitter grammar. You can use whatever you want, but published Tree-sitter grammar will use this value.

You can now create the directory queries and the file highlights.scm and put there this content.

(identifier) @variable[ "title" "characters" "inputs" "constraints" "story" "question"] @keyword; Identifiers; Literals[ (string)] @string[ (number)] @number(interpolation "{" @punctuation.special "}" @punctuation.special) @embedded; Operators[ "," ":"] @punctuation.delimiter[ "(" ")"] @punctuation.bracket[ ">" "<" "+" "*"] @operator As you can see, it is all quite simple:

• you define a query to identify the nodes captured (eg., (identifier))
• you assign it a highlight name (e.g., @variable)

A query uses a Lisp-like syntax (i.e, an S-expression) to define a pattern that, if matched, will make Tree-sitter assign the highlight name to the nodes. To indicate a node, you put its name between parentheses. In the case of a literal token, you can just type the literal (e.g., “{“).

While you can make queries as complicated as you like, queries for syntax highlighting are fairly simple. You can see for yourself the queries for a real language like Ruby, are not much more complicated than what you see here.

(interpolation "{" @punctuation.special "}" @punctuation.special) @embedded The query to highlight the interpolated element is the most complicated in our example. And even that is fairly simple. We are saying that inside an interpolation node there are curly brackets that must both be assigned the punctuation.special highlight name. The whole interpolation node should have the highlight for embedded.

We do not need to highlight here the identifier inside the interpolation node, since that is already highlighted by our first rule in the file, which captures all identifier nodes.

You can try to modify the query to give a specific name to the identifiers inside interpolation.

(interpolation "{" @punctuation.special(identifier) @number "}" @punctuation.special) @embedded We are giving it the name number just to make it easily distinguishable in the following screenshot.

The interpolated identifiers are colored a light orange/brown. You can see the results of highlighting using the command tree-sitter highlight example-file.story. Assuming you have created an example-file.story with some code on your system.

If you restore the previous query, instead you get this result for the whole file.

All identifiers, inside or outside an interpolation are colored white.

What about the highlight names?

Are they arbitrary?

Who picked the colors?

We are going to answer all of your questions. If you try to run tree-sitter highlight you will be prompted to run

tree-sitter init-config

Which will create a standard configuration file with some default colors for standard highlight names. You can actually use any arbitrary name, but if you use standard ones you will not have to define colors for yourself or your users. So, there is a strong incentive to use standard names. You can find a list of standard name in the repository of tree-sitter highlight.

Tagging ElementsQueries also power another features called tags. You should start the same way of syntax highlighting, adding a property inside tree-sitter.json.

"tags": "queries/tags.scm" You can then create the file queries/tags.scm and start writing your queries.

; Variable definition(characterspeople: [ (identifier) @name]) @definition.module; Calls(text(string (interpolation (identifier) @name ))) @reference.call The principle is the same as for syntax highlighting: you define tree patterns and assign names along the way. You can see the list of standard names in the documentation. The patterns tend to me more complicated because the scope is different.

While it is clear what syntax highlighting does, it is not obvious what is tagging. We think the easiest way to explain is that it powers features like Go To definition in your editor.

It is not technically a form of symbol resolution, because you are not trying to solve the symbols. You are just trying to look at the tree in each file and collecting things that can be referenced, like modules or classes, and references, like method calls. An editor that supports Tree-sitter grammars can then look through these lists when you click the button Go To. It is the best you can do, with a standard approach and without having to actually perform language-specific symbol resolution. You can think of it as syntax resolution: gaining all the information you can just by looking at the tree itself.

TestingTree-sitter also comes with specific support for testing. There is testing for syntax highlighting and tagging, but we are going to take a look to testing for the grammar itself.

Using the magic of default directories, you can add test inside the directory test/corpus. The extension is not relevant, typically you use .txt. You do not want to use the extension for your language, because the file will not be a valid one for your language.

So, you create a file like test/corpus/structure.txt, the name should be meaningful to your test.

==================Structure==================title:"Hello story"---(source\_file (title(string) )) Then you add three sections:

• name of the test, between the lines of =
• the example source code
• the tree that the parser should create when given the source code. The tree is defined like a query, but it should match the whole example

In case you are wondering, for this example we changed the grammar to have most elements as optional. We did this in order to make the example easy to see at once.

Setting Up the Parser in PythonWe have seen how to define a grammar, including syntax highlighting and tagging elements. All that remains is to show how to use a parser.

Tree-sitter provides various binding, so we are going to test Python. A word of warning, not all Tree-sitter software (like the CLI) and libraries are released at the same time. For instance, if you try to run a parser generated with the latest CLI (version 0.25.2) and the latest tree-sitter library in Python, you would receive a warning like this one:

ValueError: Incompatible Language version 15. Must be between 13 and 14 Therefore, if you want to run our example, you should use the 0.24.0 version of Tree-sitter.

Assuming you have the right versions of the libraries, you can build the parser from the grammar with the command tree-sitter build. You should have already generated the grammar with tree-sitter generate.

You can now install your parser as a local library with:

python .\setup.py install You would also need to install other requirements, if they are missing.

pip install setuptools pip install tree-sitter Finally ParsingOnce you have set up everything, parsing is easy.

import tree\_sitter\_storyfrom tree\_sitter import Parser, Languagedef parse(): language = Language(tree\_sitter\_story.language()) parser = Parser(language=language) # Example source code with open("./example-file.story", "rb") as file: source\_code = file.read() # Parse the source code into a tree tree = parser.parse(source\_code) # Walk the tree from the root walk\_tree(tree.root\_node) All you need to do is to set your language and pass it to the standard Parser class of tree_sitter. Using the parse method, you obtain a CST (Concrete Syntax Tree) that you can walk.

The method walk_tree is not a standard method of Tree sitter, but it uses the three standard walking methods to walk the tree.

def walk\_tree(node): cursor = node.walk() visited = set() depth = 0 while True: if cursor.node.id not in visited: visited.add(cursor.node.id) print(f"{depth * ' '}({cursor.node.type} " f"Start: {cursor.node.start\_point}, " f"End: {cursor.node.end\_point}" f"Text: {cursor.node.text.decode()[0:12]} {'...' if len(cursor.node.text.decode()) > 12 else '' }" f")") # Traverse the tree if cursor.goto\_first\_child(): depth += 1 continue while not cursor.goto\_next\_sibling(): if cursor.goto\_parent(): depth -= 1 else: return These methods are:

• goto_first_child
• goto_next_sibling
• goto_parent

Their names are self-explanatory, using them you can navigate any tree. That is simply because every node can have some children and at most one parent node. The root node, of course, has no parent.

These methods return false if they are not successful, for example goto_first_child return false if the current node has no children. If the node has no children the cursor cannot move to the first child of the current node.

The cursor is nothing else that the current node: when you start the walk the current node is the root node. Then the current node is changed by walking the tree.

Navigating with QueriesUsually you walk the tree to find some node you are looking for. Tree-sitter offers another way to accomplish that: using queries.

You can define a query and then ask Tree-sitter to find the matching nodes given a root node.

query = """(interpolation "{" (identifier) @interpolated "}" )"""query\_obj = language.query(query)captures = query\_obj.captures(tree.root\_node)for name in captures.keys(): for node in captures[name]: print(node.text.decode()) You define a query just like you did when defining rules for syntax highlighting or tagging.

Next, you build the query object and obtain the results, for the given tree.root_node. We assume that you are re-using the root_node we obtained from parsing in the previous section.

You obtain a dictionary having as keys the name of the query elements (e.g. interpolated) and as values the list of nodes that matches the query.

If you run this code, it will print the list of identifiers inside the text string.

maingivenapplesdogs SummaryWe have explored Tree-sitter, a great parser generator library. We have seen how to create a parser, add support for syntax highlighting and tagging.

Finally, we have seen how to use the generated parser from Python; how to walk the tree and find specific nodes using queries.

That is not all there is to Tree-sitter, you can find more reading the official documentation. For example, one thing we did not show is how to add tokens using an external scanner, because that requires defining the scanner in C.

Tree-sitter is a good software, with excellent performance. It also comes with support for highlighting and tagging elements. This makes it great for its intended use: providing language support in editors or similar tools to navigate code. For instance, it powers the GitHub tool that allow you to find symbols in the online code viewer: semantic. They explained why they picked tree-sitter. They list many advantages, such as:

Secure and well-tested. We’ve actively invested in fuzzing and other security measures to properly deal with corner cases.

And some drawbacks:

Support for unicode is currently lagging.

Of course, not everything listed there could be applicable to your use case, but it is a useful expert opinion on Tree-sitter strong and weak points.

In short, it is designed to quickly create parsers that you can use in a pipeline of software to handle generic concrete syntax trees. In theory, it would be great to use for things like providing code completion. However, unlike ANTLR, nobody created a generic code completion engine to get code completion for free from any grammar. So, we would probably use ANTLR for that. It is a bit slower and less robust to errors, but the advantage of having a ready-to-use code completion engine is hard to beat.

Another issue we encountered with tree-sitter it seems that is used mainly by GitHub. So, it is less tested in environments that they do not use. It is actively developed, which means that community supported parsers sometimes broke and nobody is tending to them. This is not an issue if you just care about your parser, but that means that the ecosystem is less valuable to you.

You can find the repository with the code for this example at tree-sitter-story.

The post Incremental Parsing Using Tree-sitter appeared first on Strumenta.

View Details

There is one thing that studying English and French (and failing at German) taught me about coding — especially when dealing with unfamiliar codebases.

I’ve learned through firsthand experience that you can’t truly learn a language by relying on books or your high school English teacher. Maybe you’ve gone abroad and enthusiastically wished someone good luck by translating literally, “break a leg,” only to be met with blank stares and a bemused “Oh, les Américains…”. In Italy, for example, the equivalent phrase is “in bocca al lupo”—literally, “into the wolf’s mouth.” An Italian might respond “crepi!” (may it die!), which would sound just as strange to an English speaker. If you’re unfamiliar with idioms, communications can be difficult and confusing.

Ok, but what idioms have to do with understanding code?

Idioms, are shortcuts to communicate something complex, very quickly. In code, idioms provide concise, elegant ways to express ideas, transforming complex instructions into something instantly recognizable to fellow developers.

In this article, we’ll explore how to leverage idioms to define idiomatic migrations. We’ll learn:

• The science behind idioms;
• What are code idioms useful for;
• How to identify idioms that are specific to a codebase;
• How to used to translate that meaning—not just the literal syntax—into another language or platform.

This is the difference between wishing someone break a leg in Italian as “rompiti una gamba” (which would just sound like a threat) and saying “in bocca al lupo” like a true Italian. And, of course, if you avoid asking for pineapple on your pizza and gesture with your hands a little, people might even mistake you for a native.

For the impatient, this is a video given a concrete idea of how idioms look like:

The science behind idiomsIdioms are more than just handy shortcuts—they tap into how our brains work. In The Programmer’s Brain by Felienne Hermans, the author builds on De Groot’s theory to explain why idioms are so powerful. Here’s the gist: your working memory can juggle only 5 to 9 things at once. That’s it. To do more, you need to pack more into each slot, and this trick is referred to as chunking—grouping a sequence of smaller elements into a single, meaningful unit.

Think about it this way: someone learning a programming language for the first time might need all their focus to keep track of the pieces of a for-loop. They’re thinking about initialization, the loop condition, the increment operation, and the loop body—juggling all these elements takes effort. Now imagine an experienced developer: they don’t think about those details anymore. They just see the for-loop and instantly think, “iterate over the list.”

Not only does this make writing code easier, but it also streamlines communication with others. An experienced developer doesn’t need to explain every detail of a loop—they can say, “iterate over the list,” and the other person understands. What once required a lot of mental energy now takes almost none, leaving room to focus on higher-level problems.

Idioms compress complexity into a single, digestible concept. This doesn’t just save effort—it enables you to operate at a higher level of abstraction. By expanding your mental vocabulary with idioms, you reduce cognitive load, write cleaner code, and collaborate more effectively.

Not a small feat, eh?

What are code idioms useful forCode idioms are the glue that holds consistency and clarity in a codebase. They’re more than just shortcuts—they also reflect the intent of the developer, showing not just what the code does, but often why.

Every community of developers—whether it’s the broader users of a programming language, the team at a specific company, or even contributors to a single codebase—tends to develop its own set of idioms. These are the “unwritten rules” of how things are done, shaping the style and logic of the code. And the more consistent these idioms are, the better. In fact, “boring” and predictable code is exactly what you want: excitement might be great in life, but not so much in code.

At Strumenta, code idioms are central to two areas of our work:

1. When Designing DSLs

As we develop Domain-Specific Languages (DSLs), we often notice that users express themselves using certain recurring structures—essentially, their own idioms. This can indicate a missing abstraction, something that could make the DSL more expressive or natural. This is an important part of refining a DSL, and we may want to explore in depth in a future article.

1. When Dealing with Legacy Code

Code idioms in legacy codebases are essential for two main reasons: helping developers familiarize themselves with unfamiliar systems and enabling high-quality migrations to modern languages or platforms. This is the use case we want to investigate further in this section.

How Idioms learn with legacy codeWhen we have a legacy codebase to deal with, there are essentially two situations we find ourselves in:

1. If we decide to keep it as is, we need to help people navigate it and learn about it
2. If we decide it is time to move to another language, we need to translate the knowledge trapped in this codebase into another language

In both situations, Idioms can help us. Let’s see how.

Using Code Idioms to Familiarize with a Legacy CodebaseImagine joining a team tasked with maintaining a legacy system, perhaps one developed over decades, long before you arrived. The codebase might span millions of lines of code, written by different developers using various styles, reflecting the practices of their time. Getting up to speed with it will not be an easy task.

Here’s where code idioms come in. By identifying idioms across the codebase, we provide an interpretation key—a way to understand the underlying recurring structures quickly. Automating this process is crucial because manual exploration of such a vast and complex system would be impractical. When idioms are identified and presented to the developer, they offer an efficient way to grasp the logic and conventions of the system, making onboarding and ongoing work significantly easier.

In practice, we can:

• Beforehand we can build a catalog of common idioms, providing for each one a description, an explanation of its meaning, and a list of examples.
• When navigating existing code we can show inline if they are instances of those idioms and if so let the user navigate to the definition of the idiom. We can do that through a dedicated code navigation tool or by creating a plugin for the IDE of choice.

Using Code Idioms to Avoiding Frankenstein Code During MigrationsIdioms also play a critical role in migrations. Without recognizing and leveraging idioms, a migration becomes a low-level translation exercise: replicating the original code, construct by construct, in the target language. This approach often leads to what some call “Frankenstein code,” where the target language is filled with patterns and constructs that clearly don’t belong.

Take, for example, the term JOBOL—a nickname for COBOL code translated into Java. While the code is technically Java, it just means that the Java compiler would be willing to accept it. It does not mean that a self-respecting Java developer would want to touch. Each line reeks of COBOL from three miles away, with every line screaming that it was written by someone who has never heard about the practices of the Java community.

By identifying idioms, we can find a solution to this problem. Instead of copying the original structure directly, we pause to understand the intent behind the idiom. What question was the idiom answering? What purpose was it fulfilling? Then, we re-express that purpose in the target language in the most idiomatic way possible.

For instance, imagine a complex structure in the original code that represents iterating over a list. In the target language, we wouldn’t replicate every low-level detail of this structure. Instead, we’d translate the intent—iterating over a collection—into the cleanest, most idiomatic construct available in the target language. The result is not just functional code but code that feels natural and at home in its new environment.

Does this sound very abstract? Let’s see an example then.

Suppose you have this COBOL code. It searches over a record in a sequential file:

SET NOT-FOUND TO TRUE.PERFORM UNTIL EOF-FLAG = "Y" OR NOT-FOUND = FALSE READ FILE-IN INTO RECORD-AREA AT END MOVE "Y" TO EOF-FLAG NOT AT END IF RECORD-ID = SEARCH-ID MOVE FALSE TO NOT-FOUND END-IF.END-PERFORM. If we can recognize this idiom we can define the corresponding idiom in Java, and translate each instance of the COBOL idiom in an instance of the corresponding Java idiom. For example, in this way:

List records = readRecordsFromFile(“file.txt”);

Optional<Record> foundRecord = records.stream() .filter(record -> record.getId().equals(searchId)) .findFirst();if (foundRecord.isPresent()) { System.out.println("Record found: " + foundRecord.get());} else { System.out.println("Record not found.");} If we instead had translated the code a-la-JOBOL, the result may look like this:

boolean notFound = true;boolean eofFlag = false;RandomAccessFile file = new RandomAccessFile("file.dat", "r");while (!eofFlag && notFound) { byte[] recordBytes = new byte[50]; // Assuming each record is exactly 50 bytes int bytesRead = file.read(recordBytes); if (bytesRead == -1) { eofFlag = true; } else { String record = new String(recordBytes); String recordId = record.substring(0, 10).trim(); // Assuming ID is in the first 10 characters if (recordId.equals(searchId)) { notFound = false; } }}file.close(); Which one would you want to maintain?

Recognizing idioms (or mining code idioms)

If you want to use idioms effectively in a migration—or any coding task—you need to start by building a catalog of idioms. Think of it as your pocket phrasebook.

Building this catalog is the prerequisite. Once you’ve identified the idioms, you can look for instances of them in the codebase, understand their purpose, and ensure they’re translated idiomatically into the target environment during migration. But how do you go about finding these idioms?

We can ask humans or machines to find them. Let’s see how these approaches compare.

Human-Centric Approaches to Finding IdiomsIf we want to use humans to build the catalog of idioms our first choice may be an expert in the language under consideration (e.g., COBOL). So let’s start with that and then see what it looks like to use someone who is not an expert instead.

Asking an Expert in the Language to Find IdiomsThe most obvious place to start is by asking a developer familiar with the codebase or language. This is like asking a native speaker to teach you idioms in their language. They might rattle off a few classics—the “break a leg” or “in bocca al lupo” of the codebase.

But there is one problem: just as native speakers don’t consciously think about every idiom they use, expert developers can be “blind” to idioms in the code they work with every day.

Idioms tend to surface naturally in context. So, a developer reviewing the codebase with you might point them out as they encounter them, but they’re unlikely to remember everything in one sitting. So just asking an expert for idioms is a good starting point but won’t give you a comprehensive catalog.

Asking a Fresh Pair of Eyes to Find IdiomsHere’s where things get interesting: someone who’s not familiar with the codebase—or even with the language—can conter-intuitively be a good idea.

For example, imagine a junior developer learning Java for the first time. They might look at the code and say, “Why do all these methods start with get or set? Why does every file operation involve a try block?” These questions can reveal idioms that a seasoned Java developer takes for granted.

This outsider perspective can be particularly useful for spotting surprising or non-obvious idioms.

We can turn these things he noticed into questions and get the answers from an expert.

Machine Centric Approaches to Finding IdiomsHuman intuition is invaluable, but it has limits and costs—especially with large or complex codebases. This is where technology steps in.

Mining Code Idioms AlgorithmicallyThe fundamental idea is simple: look for constructs—statements, expressions, methods, etc.—that frequently appear together or in specific contexts more often than expected.

For example:

• Calls to close() are far more likely to occur within a finally block than in other contexts.

• Methods prefixed with set typically have one parameter and return void, which is statistically significant compared to the general occurrence of single-parameter methods.

By analyzing Abstract Syntax Trees (ASTs) or control flow graphs, one can detect recurring combinations and flag them as potential idioms.

This process, however, isn’t trivial. While tools for frequent subtree mining, graph-based analysis, or statistical correlation can help, distinguishing meaningful idioms from noise is a major challenge. You might end up with thousands of irrelevant patterns unless you carefully filter and validate the results.

Scientific FoundationsIf you are interested in this topic, there is some material you can dive into. The term “mining code idioms” was introduced by Allamanis et al. in their paper Mining Idioms from Source Code (2014). This work pioneered a method for extracting idioms by analyzing large code repositories, aiming to capture recurring syntactic fragments that reflect a community’s coding practices. Their probabilistic model was specifically designed to extract idioms that convey meaning and intent beyond surface-level patterns.

Other research has contributed to this field, exploring different aspects of code idioms:

• Kim et al. (Automatically Inferring Code Idioms, 2018) focused on identifying idioms localized to specific domains or projects, recognizing that idioms often reflect contextual best practices.

• Allamanis and Sutton (Mining Idioms for Code Comprehension and Synthesis, 2013) demonstrated how idioms could aid code comprehension and automated synthesis, reinforcing the idea that idioms carry significant semantic value.

• Nguyen et al. (Graph-Based Mining of Frequently-Used Code Idioms, 2012) emphasized the importance of graph-based analysis to uncover idioms tied to control flow and structural relationships, particularly in large-scale systems.

These studies highlight that idioms are more than recurring patterns—they encapsulate meaning, intent, and best practices critical for understanding and working with a codebase.

The Challenges of Idiom MiningWhile tools for frequent subtree mining, graph-based analysis, or statistical correlation can help, the process is far from straightforward.

• Noise and Irrelevance: Mining tools often generate thousands of patterns, many of which are statistically significant but semantically meaningless. Filtering out this noise requires careful validation and domain-specific heuristics.
• Semantic Validation: Identifying patterns that are not just frequent but also meaningful to developers requires incorporating type information, control flow, and developer intent.
• Scalability: Analyzing large codebases efficiently demands algorithms that can handle the complexity of real-world systems while maintaining performance.

The Temptation of LLMsWe have started by looking at algorithmic approaches, but in these times and days one start to wonder, why going through the effort of designing or implementing algorithms when an LLM can give me the solution?

It’s an appealing idea—low-cost, low-effort.

We tried this approach. And the results were less than impressive.

The problem is that LLMs struggle to differentiate between design patterns, code clones, and code idioms—subtle distinctions that require deep understanding. Additionally, the lack of extensive literature on idiom mining makes it harder for LLMs to provide accurate answers. While LLMs can occasionally generate interesting insights, they’re unlikely to offer a reliable or systematic solution to idiom discovery.

That said, if you don’t have the resources to develop your own algorithms, experimenting with LLMs might still be worth a shot. Just manage your expectations—they’re no substitute for a robust, purpose-built tool.

FactsVector: the algorithm we developed for mining code idioms

At Strumenta, we designed an algorithm called FactsVector to identify code idioms in any codebase, regardless of the programming language. The magic lies in how we analyze Abstract Syntax Trees (ASTs) built using our StarLasu approach, which provides a consistent and unified API for analyzing code constructs. This flexibility is what allows the algorithm to be language-agnostic

Here’s how FactsVector works:

• The first thing we do is gather a list of “facts” about each AST node type. These facts are simple yes-or-no statements that can be checked for any given node. For example, if we’re analyzing a method declaration, here are some facts we might look at:
  • The method name starts with get.
  • The method name starts with set.
  • The method name starts with IS.
  • The method name is equals.
  • The method name is hashCode.
  • The method name is toString.
  • The method return type is void.
  • The method return type is int.
  • The method has exactly one parameter.
  • The method is marked as static.
• For each node, we systematically verify these facts and record the results as a series of 0s and 1s. Each fact represents one dimension in a multi-dimensional space, and the collection of these values forms a vector. The more facts we consider, the more dimensions our vector space will have.

If we had just the facts we listed above, here’s what this looks like:

• A method named getName() with a return type of String would produce a vector like [1, 0, 0, 0, 0, 0, 0, 0, 0, 0] (indicating it starts with GET but doesn’t match any of the other facts above).
• A method named hashCode() would result in [0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
• The different vectors representing the nodes are grouped into clusters based on shared characteristics. This is achieved using the Birch clustering algorithm, which scales sufficiently well for our use-cases, potentially covering tens of millions of nodes. Each cluster represents a group of nodes that share certain characteristics.
• After forming clusters, the algorithm refines its understanding of the codebase through an iterative process. Clusters are treated as higher-level pseudo-types, which provide a more abstract representation of the nodes they contain. Using these new types, the algorithm generates updated facts about the codebase, recomputes characteristics, and re-clusters nodes. This iterative refinement allows the algorithm to uncover deeper and more nuanced patterns.
• Up to this point, the algorithm has focused on identifying clusters of nodes with shared characteristics, considering the node itself and the subtree it contains. The next step is to examine idiomatic sequences of nodes, which is crucial for identifying recurring patterns in the order of execution or declaration.

For example, we analyze sequences of nodes across the codebase to identify when specific combinations occur significantly more often than expected. If a local variable declaration (e.g., a local variable of type int named i) frequently precedes a while statement in a specific context, we infer a correlation between these constructs. The algorithm identifies such relationships by calculating the frequency of sequences and comparing them to their general distribution.

This step is particularly valuable for uncovering idiomatic sequences of statements, which represent common workflows or patterns in a codebase.

We tested it so far with codebases of hundreds of thousands of lines, and results are produced within minutes. We are confident that it can be scaled to work on codebases that are tens of millions of lines code with reasonable performance. We are working on refining this algorithms, to identify more kinds of facts and better filtering meaningful idioms from the ones constituting noide. While I am sure this will keep us busy for years to come, the results we got make us happy.

How Do We Use Code Idioms to Improve MigrationsIf you want to learn how we design transpilers you may be interested in our article How to Write a Transpiler.

But here it comes a summary.

The Overall Design of Our TranspilersOur transpilers follow a three-step process:

Step 1 – Parsing the Source CodeWe start by parsing the original code to create a detailed Abstract Syntax Tree (AST). This AST captures not just the structure of the code but also all the information it contains, such as types, relationships, and dependencies. For example, in a migration from RPG to Java, we would parse the RPG code to produce an RPG AST.

Step 2 – Transforming the ASTThe core of the migration happens here. We transform the original AST into a target AST that represents the same program but in the target language. For example, we would transform an RPG AST into a Java AST.

Step 3 – Generating Target CodeFinally, we use the target AST to generate code in the target language, such as Java files that can be saved, compiled, and maintained.

Transforming the AST: Construct-to-Construct vs. Idiom-to-IdiomThe transformation step is the heart of our transpilers. It’s where we decide how to map elements from the source AST to the target AST. This transformation can follow two approaches:

• Construct-to-Construct Transformation

This is the more straightforward method. Each construct in the source AST is translated to its equivalent in the target AST. For example, a for statement in RPG might translate to a for statement in Java. These transformations are rule-based and tend to focus on individual constructs in isolation, with some consideration for their immediate context.

• Idiom-to-Idiom Transformation

This is a more sophisticated approach. Instead of translating individual constructs, we recognize idioms in the source AST and map them to corresponding idioms in the target language.

For instance, consider a pattern in RPG that involves iterating over a collection using a complex combination of loops and conditional checks. Instead of translating these elements piece by piece, we recognize the idiom for “iterating over a collection” and translate it directly into a Java for-each loop.

Why Combining the Two Approaches?We combine construct-to-construct and idiom-to-idiom transformations to achieve an optimal balance between coverage and idiomaticity:

• Idioms First: During the AST transformation, we first check if an element is part of a recognized idiom. If it is, we apply an idiom-to-idiom transformation.
• Fallback to Constructs: For elements not covered by idioms, we fall back to construct-to-construct transformations, ensuring that the entire codebase is translated.

This hybrid approach allows us to produce idiomatic target code whenever possible while still maintaining broad coverage for all constructs in the source code.

How does it work in practice?During the transformation process, we examine each element in the source AST and ask:

1. Is This Part of a Recognized Idiom?

If so, the entire idiom is treated as a single input. A transformation is applied to produce one or more nodes in the target AST, representing the idiom’s intent in the target language.

For instance, consider an RPG idiom for reading a file line by line:

READ file;DOUNTIL EOF; processRecord();ENDDO; Translating this construct directly might result in verbose and unidiomatic Java code. Instead, recognizing the idiom allows us to transform it into a clean, idiomatic Java implementation using a BufferedReader:

try (BufferedReader reader = new BufferedReader(new FileReader("file.txt"))) { String line; while ((line = reader.readLine()) != null) { processRecord(line); }} 2. If Not, Apply Construct-Based Transformation

For constructs not part of a recognized idiom, we apply a transformation rule specific to that construct type. These rules can be context-sensitive, considering the construct’s surrounding elements. For example, the transformation of a statement might depend on whether it appears within a method or as part of a field declaration.

One big plus of this approach is that it makes tracking progress much easier—something that’s often tricky in software projects, especially migrations. We start by identifying a set of idioms, which can range from 20 to 100 or more in larger, more complex codebases. For each idiom, we can see whether we’ve defined a transformation, how many instances it covers, and how much of the codebase it represents. The same goes for constructs. This doesn’t mean everything will always move as quickly as we’d like—migrations can be challenging—but at least we always know where we stand. With objective measures, we can make better predictions about when we’ll hit certain milestones or complete the migration, taking most of the guesswork out of the process.

Summary

Code migrations are tough—there’s no way around it. Moving code from one language to another isn’t just a trivial mechanical process; it’s about interpreting the intent behind the original solution. Every codebase tells a story, and idioms are what make that story readable and expressive. Without them, the migrated code feels unnatural, like something that doesn’t quite belong.

At Strumenta, we’ve found that focusing on idioms makes all the difference. By combining idiom-to-idiom transformations (for elegance) with construct-based rules (for comprehensive coverage), we strike a practical balance that gets results. Our approach has worked well for us, and we think it’s worth exploring if you’re tackling a migration yourself.

And by the way, if this sounds like the kind of approach you’d rather not reinvent on your own, we’re always happy to help through our migration services. Whether you want advice or a complete solution, we’re here to make sure your migration is done right.

The post Using Code Idioms to Define Idiomatic Migrations appeared first on Strumenta.

View Details

IntroductionKolasu is an open-source Kotlin library that simplifies the development of language engineering tools, we will talk about contributing semantic highlighting capabilities to Kolasu Language Servers. It has a companion repository that adds support for creating language servers following the Language Server Protocol.

One of the benefits of using the Language Server Protocol is the ability to semantically highlight code, an advanced form of highlighting that allows coloring tokens with contextual information. However, this feature is currently missing in Kolasu. Not for long, as in this tutorial, we will contribute semantic highlighting capabilities to Kolasu:

Furthermore, we will explain the steps to contribute to open-source projects following the GitHub flow workflow. If you want to skip to the end, all the code is available on a GitHub Pull Request. Otherwise, here is the agenda for today:

1. Define the feature we want to contribute precisely.
2. Fork the repository and implement it in a branch.
3. Test the implemented changes locally.
4. Contribute the changes back by opening a clean pull request.

Define the featureThe language server protocol specifies the messages related to semantic highlighting here. In particular, there are predefined categories and modifiers for tokens:

Then, each token has a type and possibly multiple modifiers. For example, a constant variable could have the category variable and the modifier list of [static, readonly].

To highlight a file, we need to list the category and modifiers of all the tokens in it. This list can become very long, and sending this large message can affect the language server’s performance. For that reason, the protocol specifies two optimizations:

1. Token encoding: The semantic token information is represented as relative integers instead of strings to save on message length.

2. Incremental highlighting: Only recalculate semantic tokens for the portion of the code affected by the latest edit. Saving computations significantly in editor environments.

In this tutorial, we will add facilities to Kolasu to:

• Create semantic tokens with categories and modifiers
• Encode semantic token lists to integers following the LSP standard
• Configure language servers to respond to semantic highlighting requests.

We may revisit the incremental highlighting capabilities in a future article.

Implement the featureForkingOnly the repository contributors can push changes to the original repository. However, we can create a fork to create our copy. Go to the repository and click on fork:

You will get a copy of the repository with full permissions like this one. You can play and break everything here; it won’t affect the original repository:

The goal is to implement the changes in a feature branch and open a pull request to merge the changes in the original repository. Let’s clone the repository and create a feature branch:

We are now ready to start writing code.

Data structuresFirst, we will formalize the data structures for semantic tokens. Given the repository is written in Kotlin, we will use enumerated classes to represent all the predefined token types and modifiers:

A semantic token is a portion of source code with a category and possibly multiple modifiers:

Let’s commit our changes and move to the next step.

EncodingWe will also contribute the algorithm to convert a list of semantic tokens into the integer encoding the language server protocol expects. Every semantic token is represented as five integers following the specification:

Calculating the difference between the end and start characters is not enough for the current token’s length since it could span multiple lines. Thankfully, Kolasu provides a helper method for this, given the original source code text.

The list of token modifiers is represented as a single bitmask integer. Each modifier is initialized with a bit value, so we can represent the modifier list by adding up all the individual bit values. Let’s commit the changes and move on.

ConfigurationFinally, we would like to simplify the language server configuration for semantic highlighting. Here, it is essential to follow the project’s existing philosophy. The language server plugin follows the “configurable magic” philosophy: Everything is initialized with a default, sensible configuration, but it is always possible to customize it by overriding the default behavior.

By default, we will initialize the server with the capability for full semantic highlighting using all the predefined types and modifiers as the legend:

We can also contribute the skeleton for responding to the semanticTokens/full request that computes and returns the list of encoded semantic tokens:

By providing this infrastructure, the language engineer needs only to implement the abstract semanticTokens function, which, given an AST, produces the list of all the semantic tokens in it. The library takes care of protocol specifics, but if necessary, the language engineer can customize all the defaults by overriding the initialize and semanticTokensFull methods.

Test the featurePublish to Maven LocalBefore submitting the code for review, it is a good idea to check that it works as expected. To this end, we will publish a new version in our local Maven repository and use it in a language server project.

Running the publishToMavenLocal gradle task builds and deploys the project to the local file system. The deployed artifacts are under the ~/.m2/repository/com/strumenta/kolasu folder for UNIX systems:

Verify that the publishToMavenLocal task outputs these four artifacts. You can find the published version by opening the folders. At the time of writing, the version is 1.0.6-SNAPSHOT.

Use the published pluginWe will try the new version by adding semantic highlighting to the Kuki Language Server, a toy Kolasu Language Server for cooking recipes. This is the project that initially inspired the addition of semantic highlighting to Kolasu. For more context, read the syntactic vs semantic highlighting article. Since we won’t contribute the changes, we can directly clone the Kuki repository and work on our local copy:

First, we add mavenLocal as a plugin source in settings.gradle.kts:

We will also need it as a library repository in build.gradle.kts:

Now, set the plugin to version 1.0.6-SNAPSHOT and refresh Gradle. If you use IntelliJ and have had previous versions installed, you may need to repair the IDE to refresh the dependencies cleanly.

With the new version ready, adding semantic highlighting reduces to implementing the semanticTokens method in the KukiLanguageServer class. This is of course language specific. Here is a portion of how it looks for Kuki:

We navigate the Abstract Syntax Tree and add the Semantic Tokens in order with the corresponding types and modifiers. For example, every ingredient listed at the top of the recipe is of type variable with modifier list [declaration].

To see it in action, we can run the createVscodeExtension and launchVscodeEditor tasks. This will open an instance of VSCode with the Kuki example files ready:

Whenever we edit a .kuki file, the editor sends a semanticTokens/full* request to the language server. The server responds with the encoded semantic tokens list, and the editor colors the tokens correspondingly.

Thanks to semantic highlighting, we can distinguish ingredients from utensils and declarations from references, even though syntactically, they are all identifiers.

Contributing the featureNow that we feel confident about our contribution, let’s publish it for review. Go to the original Github repository and open a draft Pull Request from our branch to the main branch. Here is an example:

I like reviewing the changed files to double-check that we didn’t miss anything or add anything unnecessary. In this case, the pull request is nice and small, simplifying the work for the reviewers.

This repository does not have any Continuous Integration checks at the moment, but if there are, we should wait and check that all the checks keep passing.

When everything is green, we can mark the Pull Request as Ready for Review and feel proud of our contribution to the open-source community!

SummaryWe have learned how to implement semantic highlighting in language servers and contribute the feature to an open-source library. When this gets merged, Kolasu and its language server plugin will be closer to their goal of providing an open-source tool for simplifying language engineering projects.

How about contributing features you find missing? Here are some examples related to the topic:

• Incremental semantic highlighting
• Support for custom semantic categories and modifiers
• An automated way to assign categories from ANTLR tokens

You may also have other features in mind. With open source, you can never go wrong; everything is welcome. Happy coding!

The post Contributing Semantic Highlighting to Kolasu Language Servers appeared first on Strumenta.

View Details

Parsing is typically where we begin and invest much of our enthusiasm. However, completing the parser is just the beginning. Sadly, I must inform you that additional steps are necessary. One step that enhances the value of the Abstract Syntax Trees (ASTs) obtained from parsing is semantic enrichment. In this article, we’ll explore what semantic enrichment is, what it enables us to do, and we are going to share some code.

As always, the code for this article is available on GitHub. You can find it at https://github.com/Strumenta/an-entity-language-sharplasu. In this article we just talk about symbol resolution, so if you want details about the parsing and creation of the AST you will have to look in the repository.

It All Starts With Parsing, but the Assembly Line ContinueWhen we parse, we recognize the structure of the code, by analyzing its syntax.

But what does it mean to parse? It means to:

1. To check the code is syntactically correct
2. To build the Abstract Syntax Tree (AST), we make nodes representing the structures we recognize. For example, we could process the code dog.fly() and return a node representing a method call.

All is good, except for the fact that dogs cannot fly.

The code is syntactically correct and semantically incorrect.

When we process code we start by verifying that it is syntactically correct.

If it is, we move to the semantic analysis. When the code is semantically correct, we can enrich the AST with semantic information.

In essence, we are interested in two things:

1. Connecting references to the corresponding declarations. We call this symbol resolution
2. Calculating the types of elements that can be typed. We call this type calculation

The Link Between Symbol Resolution and Type CalculationI am an engineer by trade, and I really like the step-by-step approach to problem-solving: you solve one part of the problem and only then move to the next one. So you may wonder why I am conflating two apparently different problems, like symbol resolution and type calculation. As for most things I do, it is because there is no better alternative.

The two mechanisms are interconnected: one depends on the other. For example, let’s say that I have this code:

class TimeMachine { Time move(TimeIncrement) Point move(Direction)}class Point { Point add(Direction)}class Time { Time add(TimeIncrement)}myTimeMachine.move(foo).add(bar) Let’s say my goal is to figure out the type of the overall expression myTimeMachine.move(foo).add(bar).

To answer this question I need to figure out which add method we are referring to. If it is the one declared in Point, then the overall result will be of type Point. If instead we are referring to the add method declared in Time, then the overall result will be of type Time.

Ok, but how do I know which add method I am calling? Well, it depends on the type of myTimeMachine.move(foo):

• If that expression has type Point and bar has type Direction, then we are calling the add method in Point.
• If instead that expression has type Time, and the expression bar has type TimeIncrement, then we are calling the add method in Time.

This means, I need to figure out the type of myTimeMachine.move(foo). To do so, I need first to figure out if I am calling the first or the second overloaded variant of TimeMachine.move. And that depends on the type of foo.

So, you see, I cannot extricate the two problems: they affect each other results, and therefore, in principle, we treat them together. In practice, for very simple languages we can get away by treating them separately. Typically, you need to treat the problems in a combined way if there are composite types or cascading functions/method calls.

If you want to read about symbol resolution for a language like Java, you can look at How to Build a Symbol Solver for Java, in Clojure or Resolve method calls in Java code using the JavaSymbolSolver.

A Prerequisite for Any Non-Trivial OperationSemantic Enrichment is a prerequisite for most programmatic handling of code.

Perhaps you may implement a linter or a code formatter without semantic enrichment, but for the most typical language engineering applications you need semantic enrichment:

• Interpretation: To execute code, we need to connect function invocation to their definition
• Migrations: To migrate code in any nontrivial way, we want to take into account the type of the different elements. For example, if we were translating the operation a + b, depending on the target language and the type of the operands, we may want to translate it as a + b or perhaps a.concat(b) or even a + b.toDouble().
• Static analysis and refactoring: Automated code modifications, such as renaming variables or moving functions, depend on knowing which references are linked to which declarations.
• Editors: Autocompletion or error-checking depends on semantic enrichment. But also go-to-definition or find-usages. In essence the difference between a basic editor and an advanced one is in their support for semantic enrichment for the language of interest.

The StarLasu Approach and Semantic EnrichmentWhen it comes to parsing, at Strumenta we apply the principles of the Chisel Method. They are quite established at this point, after years of refining. For Semantic Enrichment, things are not as crystallized as we evolve the approach at each new project, finding new solutions to new challenges. That said, we are finding patterns that work and incorporating them in our core libraries and in Sharplasu in particular.

At this stage, Sharplasu has a module called SymbolResolution which has a reasonably good approach to symbol resolution. Type calculation is instead still implemented ad-hoc for each project, at this time. So we call the type calculation logic from symbol resolution (and viceversa). It is just that we have standard APIs for symbol resolution and not for type calculation.

Let’s See an Example of Semantic EnrichmentIn our example we will work with a simple language that permits us to define entities. These entities have fields, called features with types. They can be initialized with expressions.

This is an example:

module exampleimport standardtype addressclass base {name stringdescription string}class aged : base {age integer}class person : aged {speed integer = 2}class athlete : person {speed integer = person.speed * 2}class car : aged {kilometers integer = age * 10000} This language, while simple, contains some of the elements that we can find in most languages:

• We can import modules
• We can define new types
• We have built-in types such as string and integer
• We can reference features without specifying the context (therefore using the current object as context) or specifying it

Symbol ResolutionLet’s take a look at a portion of our SymbolResolver.

Scope ModuleLevelTypes(Node ctx){ var scope = new Scope(); var module = ctx.FindAncestorOfType<Module>(); if (module != null) { // let's define types module.Types.ForEach(type => scope.Define(type)); foreach (var import in module.Imports) { SymbolResolver.ResolveSymbols(import); if (import.Module.Referred?.Entities != null) { foreach (var entity in import.Module.Referred.Entities) { scope.Define(entity); } } if (import.Module.Referred?.Types != null) { foreach (var type in import.Module.Referred.Types) { scope.Define(type); } } } } return scope;}[..]public ExampleSemantics(IModuleFinder moduleFinder){ ModuleFinder = moduleFinder; SymbolResolver = new DeclarativeLocalSymbolResolver(Issues); [..] SymbolResolver.ScopeFor(typeof(FeatureDecl).GetProperty("Type"), (FeatureDecl feature) => { var scope = ModuleLevelTypes(feature); return scope; }); [..] SymbolResolver.ScopeFor(typeof(Import).GetProperty("Module"), (Import import) => { var scope = new Scope(); if(moduleFinder.FindModule(import.Module.Name) != null) { scope.Define(moduleFinder.FindModule(import.Module.Name)); } return scope; }); TypeCalculator = new EntityTypeCalculator(SymbolResolver); } Here we want to see the basics of symbol resolution and how importing symbols from other elements works. We first see that symbol resolution depends on moduleFinder. The moduleFinder is the thing containing the list of available code for a project. This means files of the project and any library available for the project. You can see on the repository that, for this project, is just a Dictionary that keeps tracks of the name and the corresponding Module object. A Module object is the root of the AST. Given the previous example file, there will a Module with name "example" representing that. The important part is that is the object that tells you if and where modules outside the current one are located.

You can see that to solve imports, as in:

import standard To solve a symbol you need a scope. You can think of scope as a container of available definitions. In Sharplasu, a Scope can have a parent Scope, so you can properly nest them. For example, there could be global scope to solve imports and a class scope to solve features.

So, we create a scope, and then we ask the moduleFinder if there is a module with that name, and, if so, we define that module. Defining a symbol means telling our SymbolResolver object that there is a definition of the argument in the current scope. The SymbolResolver object is based on a Sharplasu class, so you can use that class for all your projects. Later, when we will ask our symbol resolver to resolve the symbols, it will look in the scope of that reference and check if there is valid definition for a reference with that name.

How Importing Modules Affects TypesYou will notice that defining module is necessary to solve the type of features.

class base {name stringdescription string} So, string after name and description are types and name string is a definition of a feature.

To solve the references to the types of features you call the ModuleLevelTypes method. This method will:

• look for definition of types in the current module
• loop through imported module, make sure that all references in imported modules are solved
• then it will define the types in each imported module

Solving imports is therefore crucial to solve types. Particularly as in this example, as in many real languages, types are often the ones from the standard library.

class athlete : person {speed integer = person.speed * 2}class car : aged {kilometers integer = age * 10000} Solving ReferenceExpressionIn our language a ReferenceExpression, like person.speed or age can only have:

• an optional parent/context element that is a class (like person)
• a target element that references a feature (like speed or age)

No nesting or multiple levels are allowed.

So, solving either a reference to the context or target element is similar.

SymbolResolver.ScopeFor(typeof(ReferenceExpression).GetProperty("Context"), (ReferenceExpression reference) =>{ var scope = new Scope(); var classParent = reference.FindAncestorOfType<ClassDecl>(); if (classParent != null) scope = ClassHierarchyEntities(reference.FindAncestorOfType<ClassDecl>()); else Issues.Add(Issue.Semantic("The class containing this expression has no superclasses. The Context cannot be solved.", reference.Position)); return scope;});SymbolResolver.ScopeFor(typeof(ReferenceExpression).GetProperty("Target"), (ReferenceExpression reference) =>{ var scope = new Scope(); if (reference.Context == null) { var classParent = reference.FindAncestorOfType<ClassDecl>(); if (classParent != null) scope.Parent = ClassLevelTypes(classParent); } else if (reference.Context.Resolved) { reference.Context.Referred.Features.ForEach(it => scope.Define(it)); } return scope;}); One difference is just that if the current expression contains a Context element, but the class containing the expression has no superclass, we have an issue, because the reference cannot be solved. The other one is that for solving Target, we must first solve Context, to make sure we only consider the features in the Context element.

Otherwise, we look for the proper elements in the parent class. Let’s just look at how to solve the hierarchy of classes.

Scope ClassHierarchyEntities(ClassDecl ctx){ var scope = new Scope(); var superclass = ctx.Superclass; if (superclass != null && superclass.Resolved) { // let's define the superclass scope.Define(superclass.Referred); scope.Parent = ClassHierarchyEntities(superclass.Referred); } return scope;} If the reference to the superclass of the current class has been solved, we define the current superclass. Then we rise up through the hierarchy of classes to define them all.

class base {name stringdescription string}class aged : base {age integer}class person : aged {speed integer = 2}class athlete : person {speed integer = person.speed * 2} Basically, considering the previous example, to solve the reference to person, in person.speed we define person, because is the superclass of athlete, the class containing the expression, then aged and base.

Symbol Resolution PatternsWe can then see a few rules:

• The way in which we resolve imports is by delegating to the moduleFinder object. This is the case because we need to use some logic to find other files and parse them on demand, possibly managing loops (what if a file import itself, directly or indirectly?)
• When looking for a superclass we use a global scope, for all classes in the module
• For solving references to feature we do not specify any element declared at that level. We just look for features declared in the classes containing them. So to do this, we define a parent scope
• The case of ModuleLevelTypes is interesting because there we can see a combination of elements:
  • We get all the types declared in the module
  • We get all the types declared in the imported modules
  • We could also get all the built-in entities, but we choose to force the user to import a standard module to get them instead

These few rules cover many of the most common patterns we see in languages we work with, either Domain Specific Languages we design or legacy languages for which we build parsers.

Type CalculationLet’s take a look at our simple Type Calculator. Notice that we managed to separate our type calculation from symbol resolution. Our type calculation needs symbol resolution, but not vice versa. We are going to see this simpler case first, then what happens in the standard case.

public override IType CalculateType(Node node){ switch (node) { case OperatorExpression opExpr: var leftType = GetType(opExpr.Left); var rightType = GetType(opExpr.Right); if (leftType == null || rightType == null) return null; switch (opExpr.Operator) { case Operator.Addition: if (leftType == EntityStandardLibrary.StringType && rightType == EntityStandardLibrary.StringType) return EntityStandardLibrary.StringType; else if (leftType == EntityStandardLibrary.StringType && rightType == EntityStandardLibrary.IntegerType) return EntityStandardLibrary.StringType; else if (leftType == EntityStandardLibrary.IntegerType && rightType == EntityStandardLibrary.IntegerType) return EntityStandardLibrary.IntegerType; else throw new NotImplementedException($"Unsupported operand types for addition: {leftType}, {rightType}"); [..] } case ReferenceExpression refExpr: if(refExpr.Context == null) return GetTypeOfReference<ReferenceExpression, FeatureDecl>(refExpr, typeof(ReferenceExpression).GetProperty("Target")); else { SymbolResolver.ResolveNode(refExpr); return GetTypeOfReference<ReferenceExpression, FeatureDecl>(refExpr, typeof(ReferenceExpression).GetProperty("Target")); } [..] case StringLiteralExpression \_: return EntityStandardLibrary.StringType; case BooleanLiteralExpression \_: return EntityStandardLibrary.BooleanType; case IntegerLiteralExpression \_: return EntityStandardLibrary.IntegerType; default: throw new NotImplementedException($"Type calculation not implemented for node type {node.GetType()}"); }} It shows common patterns for type calculation:

• On the bottom you can see that we solve types for literals: we assign a standard type for each literal
• We solve types for binary operations by finding types for the two individual elements (left and right) of the expression and then defining rules for the combination. For each an addition of a string and an integer is considered a concatenation, so the resulting type is an integer. This will vary very much by language and how you choose to handle type conversion between compatible types
• To solve the type of reference expressions, we need to solve the reference and then get its type

Calculating the Type of ReferencesTo solve the type of references we get a look at GetTypeOfReference method. This method has type argument the class that can hold the reference and the information about the property that will hold the reference in the first argument. Notice that in this case we could avoid using the first type argument, since we ReferenceExpression is the only kind of expression containing a reference. However, this code shows that it is easy to generalize this method.

private IType GetTypeOfReference<T, S>(T refHolder, PropertyInfo refAccessor) where T : Node where S : Node, Named { ReferenceByName<S> refValue = refAccessor.GetValue(refHolder) as ReferenceByName<S>; if (refValue != null && !refValue.Resolved) { SymbolResolver?.ResolveProperty(refAccessor, refHolder); } else if (refValue != null && refValue.Resolved != false) return GetType(refValue.Referred as Node); return null;} The method uses a bit of reflection. In essence, it checks if the providing property corresponds to a reference that has been solved. If it is not, then it triggers the symbol resolution, supposing we have a SymbolResolver. Then we get the type referred to. GetType is simply a way to access a Dictionary matching nodes with types.

These are the common patterns to calculate a type. One that we are missing is having a special type for void or unit, that represents no type.

Again, it may sound quite boring, but these are the kind of patterns we routinely see. Of course, things can get more exciting if we throw generics and type inference in the mix, but for this time, let’s keep things simple.

Our types are all classes that inherits from an interface IType. This is not an interface part of Sharplasu, we created it for this example, but it is so simple that you can look it up on your own in the repository.

When Type Calculation and Symbol Resolution IntertwineWe avoided making type calculation and symbol resolution be dependent on each other for a few reasons. Our references had only two levels and we knew that the first one was always a superclass of the current one. Imagine we change that.

class address {note basecity stringstreet stringnumber integer}class person : aged {location addressspeed integer = 2}class athlete : person {deliveryNote string = location.note.descriptionluckyNumber integer = location.number + 3} Now, the features can have as a type a class, other than scalar types. Our references now have a Context property, which is an Expression. So, the node ReferenceExpression for location.note.description will have this structure. At the first level we have a ReferenceExpression with Target description and Context another ReferenceExpression with Target note and so on.

This means that now we cannot determine statically what will be the actual type of a Context object: it could be a scalar type or a class. So, to solve a reference in Target we need to dynamically define the type of Context, which will depend on what the reference in Context resolves to. So, how do we accomplish this? For starters, we need to make a small change in the ModuleLevelTypes method and make sure that we define all entities at the module level.

module.Entities.ForEach(type => scope.Define(type)); Apart from that all we need to change is how we solve symbols for the Target property.

SymbolResolver.ScopeFor(typeof(ReferenceExpression).GetProperty("Target"), (ReferenceExpression reference) =>{ var scope = new Scope(); var classParent = reference.FindAncestorOfType<ClassDecl>(); if (classParent != null) scope.Parent = ClassLevelTypes(classParent); if (reference.Context != null) { SymbolResolver.ResolveNode(reference.Context); var type = TypeCalculator.GetType(reference.Context) as ClassDecl; if (type != null) { type.Features.ForEach(it => scope.Define(it)); } } return scope;}); The pattern is simple:

• We ensure we have resolved the Context node, so we know which feature Context resolves to
• This allows us to get the type, i.e. the class the features has
• We can now define the features of the class

And voilà, we can now solve the current reference.

Using Semantic EnrichmentIt is easy to glue together symbol resolution and type calculation.

public List<Issue> SemanticEnrichment(Node node){ SymbolResolver.ResolveSymbols(node); node.WalkDescendants<Expression>().ToList().ForEach(expression => { TypeCalculator.SetTypeIfNeeded(expression); }); return Issues;} So, we can take any module and trigger symbol resolution. We will then get an AST containing references that have been resolved. Also, the types will be stored in a cache in TypeCalculator.

public abstract class TypeCalculator{ public virtual IType GetType(Node node) { return SetTypeIfNeeded(node); } public IType StrictlyGetType(Node node) { var type = SetTypeIfNeeded(node); if (type == null) throw new InvalidOperationException($"Cannot get type for node {node}"); return type; } public abstract IType CalculateType(Node node); public virtual IType SetTypeIfNeeded(Node node) { if (node.GetTypeSemantics() == null) { var calculatedType = CalculateType(node); node.SetTypeSemantics(calculatedType); } return node.GetTypeSemantics(); }} This means that after invoking the semantic enrichment, we can look each of our nodes of class Expression and get their type using mynode.GetTypeSemantics(). Easy, right?

A Simple TestWhat is life without tests? Here at Strumenta we do not want to imagine such a sorry existence, so our repository has a few tests. Let’s see a simple one:

[TestMethod] public void TestTypeCalculation() { EntitySharplasuParser parser = new EntitySharplasuParser(); string code = @"module exampleimport standardclass base {name stringdescription string}class aged : base {age integer}class person : aged {location addressspeed integer = 2}class address {note basecity stringstreet stringnumber integer}class athlete : person {deliveryNote string = location.note.descriptionluckyNumber integer = location.number + 3}class car : aged {kilometers integer = age * 10000}"; var result = parser.Parse(code); SimpleModuleFinder moduleFinder = new SimpleModuleFinder(); ExampleSemantics semantics = new ExampleSemantics(moduleFinder); List<Issue> issues = semantics.SemanticEnrichment(result.Root); Assert.AreEqual(0, issues.Count); result.Root.AssertAllExpressionsHaveTypes(); Assert.AreEqual("string", result.Root.Entities[4].Features[0].Value.GetTypeSemantics().Name ); } We test that all expressions have a type, then we check one specific expression. The highlighted expression should have type string, since location is of class address which has the note feature of class base, which in turn has a feature description of type string.

ConclusionsWhile parsing organizes code into a syntactic structure, Semantic Enrichment uncovers its meaning by resolving symbols and determining types. Without this critical step, advanced operations such as code generation, interpretation, and refactoring would be impossible. Semantic Enrichment is not trivial to implement.

With this article, we wanted to share some of the principles behind it. And through the support built-in in Sharplasu, we want to provide a way to simplify the implementation of advanced Language Engineering solutions. For us, it has been working pretty well, and we hope it will work similarly well for you.

Have fun with your Language Engineering project!

The post Resolve Symbols and Calculate Types with Sharplasu appeared first on Strumenta.

View Details

In this article, we will present our COBOL parser. It is based on our Chisel methodology and our open-source library, Kolasu. The parser is commercially licensed, but we hope this article will be useful for anybody wanting to learn what a parser is for and how to use one.

If you want to know what a parser is, we will explain it.

We are going to show you an easy to understand example of the things you can build with one: a custom software to understand which external programs your code calls.

If you are looking to design your own parser, we will show you why you might want to follow our Chisel method to build one.

What is a Parser (And What is Not)We work with parsers daily, so it is very clear to us what a parser is and what it can do. However, this might not always be the case for users. So, let’s spend a few words on this point.

Generally, a parser is software that can understand the syntax but not the semantics of some code.

Fundamentally, a parser is software that creates a model of the input source code so that data can be extracted from it.

This model should be clear, useful, and easy to work with. The model that matches all these criteria is the Abstract Syntax Tree (AST). The practice of using an AST to represent code is an industry standard. Of course, how to design the AST for a particular language is open to debate.

A parser can read the code, but it cannot execute it. For example, a parser can recognize both a variable declaration and an expression.

What it cannot do is link the two and understand where a variable used in an expression was declared. This feature is called symbol resolution and it is a functionality built on top of the parser.

So, our COBOL parser does not implement symbol resolution functionality. That means, for example, that if you want to rename some variable you will have to build this functionality on top of the parser yourself. Or, you may hire us to implement it for you.

Play Around on the PlaygroundWe have set up a platform to allow testing the COBOL parser online, without the need to install anything. You can see what an AST looks like. You can test our COBOL parser, or any of our other parsers for that matter, with some example files on our playground.

It is a great way to get a feel of how it works and whether it supports the features you care about. You can try it here: https://playground.strumenta.com. We are also available to test the parser on some examples you provide.

You can also read the documentation on the COBOL parser online.

Chisel MethodologyThe COBOL parser is based on the Chisel methodology. It is the missing link between source code and a convenient structure for its interpretation and manipulation: an AST. When building an interpreter, transpiler, compiler, editor, static analysis tool, etc., at Strumenta we always implement the software using a pipeline. A set of reusable components that can be shared for different projects.

For example, this is a pipeline for an RPG-to-Java transpiler.

The COBOL parser and the Semantic Enricher components can be re-used for, let’s say, building an interpreter. This approach increases productivity and improves the quality of the software. For instance, any improvement to a core component for one project gets automatically shared with others.

StarLasu is the collection of runtime libraries that implement this methodology to support it in Java, Kotlin, Python, Javascript, Typescript, and C#.

At its core, StarLasu permits the definition of ASTs, on which all other functionalities are built. You can navigate and transform ASTs to do everything from reading the original values to simplifying your code. With the features provided by the library, you can do anything from analyzing a codebase to building a transpiler.

Some core features shared by our StarLasu libraries are:

• Navigation: utility methods to traverse, search, and modify the AST
• Serialization: export and print the AST as XML, as JSON, as a parse tree
• EMF and Lionweb interoperability: ASTs and their metamodel can be exported to the EMF or Lionweb formats

Interoperability with EMF and Lionweb is important because these are standard formats used in language engineering so you can mix and match different software, even from different providers.

You can read more about our methodology in a dedicated article.

A Short History of COBOLCompared to a historical rival, like RPG, COBOL has been used by larger companies. This has little to do with the technical quality of the two languages or even their age. They were born in the same year, 1959, and while COBOL was historically considered more versatile than RPG, the difference was not that significant.

COBOL saw more use than RPG in larger companies and governments because it cost more to use, that’s it. This had to do with the business choices of the companies implementing compilers and language tools of the two languages.

COBOL was not developed by one company but by a consortium of companies.

You may ask why you should care about this history. It has one important consequence: there are many variants of COBOL; Wikipedia lists 34 of them. The IBM COBOL variant was initially the predominant one, but that changed over the years. Your codebase could now even contain multiple COBOL variants from the same vendor. The language is 65 years old, so it has seen a lot of changes. So, you need more than a COBOL parser, you need a parser for the specific variant of COBOL you are using.

Why Use a Ready-to-go Parser?COBOL is used for large, important, complicated software and there are many versions of COBOL. So, you need a parser that has been thoroughly tested, which is documented, and gives you someone to call in case you encounter any problems.

We are experts and we have built tons of parsers for our clients. This means that we completely understand the importance of this component and we have a solid methodology.

And we build parsers designed for what our users need. For example, our SAS parser is geared to support data lineage, because that is what the typical user needs.

Our COBOL Parser has also been built for the needs of our clients, so it is battle-tested by our clients and us. For instance, it supports COPY statements, including the content of a COPY statement inside a file. It also handles for you some quirks of this old language. A COBOL program can alter the syntax of the language itself, with the instruction DECIMAL-POINT IS COMMA.

ENVIRONMENT DIVISION.CONFIGURATION SECTION.SPECIAL-NAMES. DECIMAL-POINT IS COMMA. This changes the way the COBOL compiler understands decimal numbers. With this instruction, the decimal separator becomes the comma rather than the dot. Our parser can alter its behavior and automatically parse decimal numbers correctly.

Our parser was designed to support COBOL85. It now also partially supports Micro Focus Visual COBOL and Micro Focus ACUCOBOL-GT.

What Do You Need a COBOL Parser For?Historically COBOL has been used on mainframe computers to build business applications. COBOL is rarely used to write new applications nowadays, but it still powers applications in crucial sectors like finance and government.

COBOL statements have a prose syntax, preferring words rather than symbols. This makes the language readable and self-documenting, but verbose.

Users have entire software stacks built with COBOL, to implement custom ERP software, software to manage financial transactions, etc. Users have clusters of internal applications built over the decades to handle all their needs.

The software is custom, so it is tailored to the needs of the companies. However, it has grown organically over the years, so the code is often poorly organized. It is not rare to have thousands of files without knowing which ones constitute a separate program. So one typical use case is to use a COBOL parser to perform analysis and understand which files constitute each program and how everything works together.

The language uses obsolete designs and practices. For example, programs were meant to be integrated into the specific system they were run on. Each program requires specific sections configuring the environment the program will run, data files used by the program, internal records used by the program, and so on…

For instance, the data definition and the database are effectively part of the program. This is a consequence of the system it was used in.

There Are Many Layers of Backward CompatibilityHaving a long history implies that it is important to preserve backward compatibility. So, while COBOL itself is just a language, programs are often integrated with the systems they were designed for. This is simply how programs were developed back then. Given that there are many variants, even from the same vendor you might end up with a codebase mixing COBOL 85, Micro Focus Visual COBOL, and ACUCOBOL-GT, because of partial transitions done during the years. In this environment even just fixing a bug or improving a program becomes a risky endeavor, that could create a cascade of problems. Even just from a practical point of view: unless a developer has been with the company from the beginning they might be familiar with the latest variant of COBOL adopted. So, they have trouble when trying to work on older programs.

One way to handle that is to build software that automatically translates one variant of COBOL to another. This is technically a transpiler, although a relatively simple one that just needs to transform some statements into others.

When the challenges outweigh the benefits of COBOL, you want to build a full transpiler, to translate COBOL, to, for example, Java. Rewriting all the code from scratch is risky and costly. Throwing all the custom code away results in less productivity, given that the software was designed for the company. So you want to migrate the code from COBOL and do it automatically. The way to do that is with a transpiler. And for building that you first need a parser to understand the code.

How to Setup the ParserThe only required library you need to use the parser is the cobol-parser package. For example, for Java Maven, you would write something like this.

<dependencies> <dependency> <groupId>com.strumenta.langmodules.cobol-parser</groupId> <artifactId>ast</artifactId> <version>1.1.6</version> </dependency></dependencies> This would use the cobol-parser and automatically add its dependencies, like kolasu.

You can easily adapt this for another build system like Gradle.

dependencies {implementation "com.strumenta.langmodules.cobol-parser:ast:1.1.6"} That is all you need to be able to use the parser in your code just as easily as any other library. We also offer the possibility of using a fat jar with all the dependencies included.

How to Use the ParserUsing the COBOL parser requires you to indicate the file you need to parse and the location of the files that can be the target of a COPY statement.

For instance, imagine that a COBOL file contains the following section of code.

WORKING-STORAGE SECTION. * 77 WS-NAME-ERROR-FLAG PIC 9(01) COMP VALUE 0. 77 WS-NAME PIC X(30) VALUE SPACES. * COPY NAMEW. We will need to find and include the text of NAMEW into the original code, in order to be able to parse correctly the original file. The file NAMEW starts this way, so it is not a standalone COBOL file and cannot be parsed on its own.

01 NAME-WORK-AREA. 03 NAW-NAME-WORK PIC X(34). [..] Assuming all the files are in the directory examples1 we can do that very easily.

val parsingResult = CobolKolasuParser( copyResolverUsingDirectory(File("examples1"), "COB")).parse(File("examples1/NAME.cob")) We initialize the CobolKolasuParser class passing as the first argument the copyResolverUsingDirectory method that, unsurprisingly, will resolve all COPY statements by looking for files with the specified extension in the specified directory. Then we pass the location of the main file as an argument to the parse method.

When can then check whether the parsing was successful, by looking at the property correct of parsingResult.

if(parsingResult.correct) println("Success") If we explore the root property of parsingResult, we can see the right RecordDescription in the AST: NAME-WORK-AREA appears as the last variable in workingStorageSection.

How to Collect All Files Needed by the CodeCOBOL was designed before the practice of a neat separation between code and data. So you get things like the INPUT-OUTPUT section and FILE-CONTROL. The FILE-CONTROL paragraph names each file that will be worked on by the COBOL code. Some languages require to define variables, in COBOL you also need to declare files. One positive consequence is that you can have a neat list of the files you need to work with.

Imagine you want to move on from COBOL to a more productive language like Kotlin. You probably want also to update how you use these files.

The next step would be to transform the action of outputting data to a file to save data in the database. You can also format the files, so they can be used to present the information. So, maybe you want to transform some of these files into an HTML page that can be seen with a browser.

This is an example of how moving from an old language to a newer one often requires you more than just translating the code, you need to adapt the things surrounding the code, like data structures, to update them to the newer practices.

Whatever you plan to do with the files, you need to collect them first. All you have to do is to parse the file.

INPUT-OUTPUT SECTION.FILE-CONTROL. SELECT Student-Payment-File ASSIGN TO "STUDPAY.DAT" ORGANIZATION IS LINE SEQUENTIAL. SELECT Print-File ASSIGN TO "FEES.RPT". SELECT Student-Master-File ASSIGN TO "STUDMAST.DAT" ORGANIZATION IS INDEXED ACCESS MODE IS DYNAMIC RECORD KEY IS SM-Student-Number ALTERNATE RECORD KEY IS SM-Student-Name WITH DUPLICATES FILE STATUS IS SM-File-Status. This will transform a section of code like the previous one into an easily accessible AST.

(parsingResult.root!!.mainProgram.environmentDivision?.sections ?.first { it is InputOutputSection } as InputOutputSection).fileControl?.paragraphs?.forEach { println(it.fileName)} The previous code will output this.

Student-Payment-FilePrint-FileStudent-Master-File SummaryWe hope you have seen how easy it is to use and be productive with our COBOL parser. It has been battle-tested and is used in production by various companies.

We have distilled our knowledge about parsing into this software, to make the most productive COBOL parser out there. Built using a solid methodology, that is open to all and can be applied to build your software.

We have seen how to use the parser and to identify the external components of a program, like the files referenced in the code.

You can start playing with the parser on the Strumenta Playground.

The post How to Use the COBOL Parser appeared first on Strumenta.

View Details

We interviewed Samuel Iseli, the Head of Research and Architecture at Vertec Group, a Swiss company. To expand its business to international markets, Vertec wants to migrate the business objects model and the whole database to English.

Federico Tomassetti: Hello. Thank you for joining us. Can you start introducing yourself and tell us a little bit about Vertec?

Samuel Iseli: Yes, sure. My name is Samuel Iseli. I’m a co-founder of Vertec, and I was a long time responsible for the development. My current position is the Chief Architect and Head of Research. Yes, perhaps a little bit about Vertec. We are a product company. We produce an ERP system for the services business. That means for companies that sell services, and it comprises CRM, invoicing, everything, business intelligence. And it’s quite a capable system that’s highly configurable. We have a few thousand customers in Switzerland and Germany mainly. And yes, that’s about Vertec, the company. Vertec is almost 30 years in the market already. Yes, that’s about our company. What did we build together with Strumenta? We built a SQL transpiler, and what we want to do with it? We are in the process of migrating our business objects model and also the whole database to English because we are a swiss german company. So when we started with everything with database schema, the model and so there are quite a lot of German terms in the objects, in the classes, in the tables, columns and so. And as we are targeting now more international markets like not only Germany and Switzerland, we are in the process of transferring our whole data model to English, to good English terms. And what’s always the case with our product, it’s fully backward compatible. So all our customers should be able to just work with the new version without doing changes and assistant.

There are a lot of configurations on client side, so we need to ensure that all the custom SQL snippets there are and so at the customer configurations just work in the German version still. So our way is to just plug hook into the SQL execution and there translate all these SQL fragments from the German based or German english mixed statements into the new one. For this we need to parse the SQL because only then can we identify the tables and column names and translate them based on our data model. That’s the central part of the applications.

Federico Tomassetti: Makes sense. I wanted to say congratulations for these results. I think for myself as a Founder, not as successful as yourself, it’s very inspiring to talk with someone that I can take as an example. Can I ask you, why did you decide to work with Strumenta?

Samuel Iseli: Yeah, I mean we just thought building a parser and a transpiler is quite a specialized work we will do once. And so it, it was attractive for us to search for somebody, a contractor that does it for us. And I did some web research and some mailing list, I don’t know where, and I got across your name some years ago already. So I had it in my notes somewhere and I thought, okay, there’s a guy, he wrote a small book about compilers and it’s a link, if you go to enter, there are links that point to Strumenta. And so I just got somehow an overview who deals with building parsers. And so I came across Strumenta and I thought it appeared like a competent partner to me. So we tried this with you together.

Federico Tomassetti: I hope we maintain that impression also after, at this point I will normally ask you why this project is important to you, but I think you have already explained how these enable you to move to a broader market. So it seems quite important.

Samuel Iseli: I mean, it’s a very technical component. Most customers won’t even, or shouldn’t even notice that it’s there.It’s really very hidden in our product,but still important for this transition.

Federico Tomassetti: Yeah, sure. And are you satisfied with the results that you got from this collaboration?

Samuel Iseli: Yes, yes, yes. There were small, few small boxes in the parser, but we noticed that we added quite a lot of real-life SQL statements from our customers or from our implementers at the customers to just test quite a lot of it and some details. But then they were always fixed quite quickly by Strumenta. So the way of working together was okay for us. You delivered what we ordered, and it works. And as I said, yes, the response to additional or bug fixes is really good.

Federico Tomassetti: Do you think, is there anything that we could have done differently to provide a better service?

Samuel Iseli: I think for this kind of work where the specification is really very clear, just implement the parser for this databank database system. So yeah, it is good. I mean we didn’t have much communication. It was a kickoff meeting. Then you developed and then you delivered and we looked at it and it worked. So, not much. I mean, on a different project it probably would be a closer collaboration that wasn’t really needed in this case. Yeah.

Federico Tomassetti: Yeah. Our goal is to be boring and reliable. So since that. Okay, and would you suggest to someone else with a similar problem to work with Strumenta?

Samuel Iseli: Yeah, sure. I mean I think it’s, I mean if you, if you search for companies that are that build compiler, you find, you find some, some individuals perhaps, but we, we prefer to work with a company that has more than just one developer. I mean we always worked with Gabriele now, but it’s just a bit more than an independent single person developer. So I think in this, and I would really suggest if somebody needs to build a parser or compiler, then I think Strumenta is really the, the place to go.

Federico Tomassetti: Thank you. Okay, we are approaching the end of the interview. Last couple of questions. One is if you can share where someone could find more about Vertec. If you have any pointers or any suggestion, if someone is interested in learning more.

Samuel Iseli: Yeah, sure. Our website, www.vertec.com and there you find everything and we can, we are always happy to consult potential customers. You see also the reference companies, quite some well known names in the lawyers or engineering business that. Yeah.

Federico Tomassetti: And final question, is there anything that I didn’t ask and you want to add?

Samuel Iseli: No, I think that’s okay.

Federico Tomassetti: Yeah. Good then. Thank you very much.It’s been a pleasure

Samuel Iseli: … to talk. Yeah, I know.

The post Interview with Samuel Iseli appeared first on Strumenta.

View Details

IntroductionIn this article, we explain the concepts behind syntactic and semantic highlighting. Humans are great at associating colors with concepts. We link red traffic lights to stopping and blue markers on showers to cold water. The information flows quickly and effortlessly. In programming, we leverage this ability by highlighting pieces of code, making the programs easier to understand and debug.

We then apply the theory by implementing highlighting in Kuki, a language server we created in a previous article for describing cooking recipes. The result is going from no color:

To syntactically highlighted code:

And finally semantically enriched highlighting:

The entire codebase is available on Github on the highlighting branch of the Kuki repository. To keep the article lightweight, we will reference files from the repository and focus on explaining the most important points.

In the following sections, we will introduce syntax highlighting and illustrate how this can be implemented in our example. Then, we will extend our discussion and example implementation to semantic highlighting. Finally, we will discuss the differences between the two approaches.

Syntax highlightingThe idea behind syntax highlighting is to divide the source text into tokens and assign a category to each token depending on its textual representation. For example, in the following javascript code:

if (myNumber === 10)return "My number is ten"; The tokens if and return are keywords, myNumber is an identifier, === is an operator, 10 is a number, “My number is ten” is a string, and the parentheses and the semicolon are punctuation marks.

Notice that we can deduce that without context just by looking at the characters of each token. We express these character patterns with regular expressions:

| Pattern | Explanation | | if | return | A sequence of characters exactly matching if or return is a keyword | | “.*?” | Characters surrounded by double quotes are strings | | [a-zA-Z]+ | A sequence of letters is an identifier |

Consider that prioritization of the patterns is essential; otherwise, if will match both the keyword and the identifier rule.

Most modern editors, including Vscode and IntelliJ, use Textmate grammars to define syntax highlighting rules. They comprise a prioritized list of rules that map regular expression patterns to scopes. Scopes are dot-separated names like entity.name.function.kuki.

Themes select scopes using CSS style selectors and style them with colors and font properties. For example, the selector entity.name.function assigns styles to all function names in any language, while entity.name would apply styles to all the names. These styles can be overridden for Kuki function names by using the entity.name.function.kuki selector because it has the highest specificity.

The article’s goal is not to teach how Textmate grammars work, there is already great documentation online. However, let’s go over the thinking process behind defining the syntax rules for our cooking language:

Syntax highlighting in KukiTextmate grammars can be written in XML, JSON or YAML format. In this case, we decided to write the grammar in JSON format. Here is a summary of the rules:

| Rule | Pattern | Scope | | Section titles | INGREDIENTS|UTENSILS|STEPS | entity.name.section.kuki | | Functions | Mix|Slice|Place|Add… | entity.name.function.kuki | | Keywords | the|into|in|to|for|and | keyword.other.kuki | | Numbers | \d+(.\d+)? | constant.numeric.kuki | | Punctuation | :|-|. | punctuation.kuki | | Variables | [a-zA-Z]+ | variable.other.kuki |

Textmate grammars are free form, meaning you can assign any scope name to the tokens. However, there is a standardized set of names that is highly recommended to use. That way, existing themes will be readily compatible with your language.

That is why we aim to pick the closest match even though the standard names do not perfectly fit our esoteric programming language. As language engineers, we focus on assigning the correct scopes to each token; then, theme builders style the tokens with pretty colors. Here is the result with a couple of different built-in Vscode themes:

Looking good, but what if we want to distinguish ingredients from utensils? This is not possible with syntax highlighting because they are both syntactically in the same category.

We need context to determine that almond is actually an ingredient because it has been declared in the ingredients block. The same happens in standard programming languages when we want to differentiate local variables from global variables or parameters. Semantic highlighting fixes this problem.

Semantic highlightingSemantic highlighting uses context-aware language comprehension tools to assign refined categories to tokens.

It is usually found in the realm of language servers. Whenever code changes, the editor requests the server for semantic tokens through the Language Server Protocol (LSP). Thanks to advanced parsers, language servers can understand source files deeply, allowing them to specify more fine-grained semantic categories.

The list of standard semantic categories is fixed. It can be expanded with custom categories, but this should be avoided if possible. In addition to categories, the protocol defines standard semantic modifiers. Tokens can have multiple modifiers at once. For example, a variable may have the read-only and declaration modifiers.

We can inspect the syntactic and semantic scopes assigned to a token by using Vscode’s scope inspector:

Open the command palette (F1 in my configuration) and search for scope inspector. When you place the cursor over a token, a context menu appears with all the scope information. This is very helpful during development for checking and debugging your rules.

Let’s improve the highlighting of Kuki code by adding semantic highlighting capabilities to the language server.

Semantic highlighting in KukiThe goal is to semantically differentiate identifiers into recipe titles, ingredients, and utensils:

The Kolasu language server plugin defines semantic tokens as a range of code with a semantic category and a list of modifiers:

data class SemanticToken( val position: Position, val type: SemanticTokenType, val modifiers: List<SemanticTokenModifier>) The plan is to create a list of semantic tokens, one per identifier, and list them in the order they appear in the source code. The language server plugin will then encode this information into a list of numbers following the encoding rules the LSP uses for performance reasons. At the time of writing, this feature is unavailable in the plugin’s public release; it will be available once this issue is marked as closed.

To implement this, we override the semanticTokensFull request of the KolasuServer class in Main.kt. In particular, we create a semantic token for the recipe title with the category Type:

val titlePosition = recipe.name.positiontokens.add(SemanticToken(titlePosition, SemanticTokenType.TYPE, listOf())) We then add a semantic token with the declaration modifier for each ingredient and utensil declaration:

for (ingredient in recipe.ingredients) { addIngredientAt(ingredient.declaration.position, isDeclaration = true)}for (utensil in recipe.utensils) { addUtensilAt(utensil.declaration.position, isDeclaration = true)} Finally, for each item reference, we check the parent node’s type to determine if it is an ingredient or a utensil reference:

fun addTokenFor(item: ItemReference) { when (item.reference.referred?.parent) { is Ingredient -> addIngredientAt(item.position) is Utensil -> addUtensilAt(item.position) else -> addIngredientAt(item.position) }} We finish the implementation by calling the plugin’s encode function:

return CompletableFuture.completedFuture(encode(tokens)) Syntactic vs semantic highlightingHere is a table that highlights (heh) the differences between syntactic and semantic highlighting:

| Feature | Syntax highlighting | Semantic highlighting | | Performance | Fast | Can be arbitrarily slow | | Runs on | Same thread as renderer (no delay) | In the language server process | | Reusability | Editors that support Textmate grammars | Editors that support the Language Server Protocol | | Limitations | Limited to regular expressions | Unbounded logic complexity |

As a general rule, syntactic highlighting is more performant and user-friendly because it is fast and introduces no delay. However, semantic highlighting enables arbitrarily complex rules for coloring code and is available in all the editors that support the LSP.

The good news is that we don’t need to choose between one or another. A language can be highlighted both syntactically and semantically. Semantic rules take precedence over syntax rules, but if they are not present or supported by the theme, syntactic rules will apply.

SummaryWe have learned how to add syntactic and semantic highlighting to programming languages. For language servers, syntax highlighting happens on the client side by providing Textmate grammars in XML, JSON, or YAML format.

Semantic highlighting works on top of syntactic highlighting by adding contextual information.

The LSP specifies a list of categories and modifiers we can programmatically set to tokens in the code. The themes will use this to pick the correct color.

By leveraging the power of Kolasu and its language server plugin, we can focus on defining the data (the syntax grammar and the list of semantic tokens), and the rest will be taken care of.

Want to put your semantic highlighting tools into practice? How about grabbing the code from Github and assigning different modifiers to solid and liquid ingredients? Or visualizing ingredients high in protein prominently? With semantic highlighting, the logic used to highlight code is limitless.

The post Syntactic vs Semantic Highlighting appeared first on Strumenta.

View Details

In this article, we present our Teradata SQL parser – a commercially licensed parser for Teradata SQL. Reading this article, you will learn why, when, and how you would want to use our ready-to-go parser.

In the following sections, we will first provide a short introduction to Teradata SQL – what it is and why you might want to have a parser for it. Then, we will briefly explain what a parser is and why ready-to-go parsers might represent the right solution for you. Finally, we will illustrate how our Teradata SQL parser can be integrated and used in both JVM and Python projects.

Farewell Teradata SQLTeradata SQL is the Structured Query Language (SQL) dialect used to interact with Teradata databases. Like many other SQL dialects, Teradata SQL extends the ANSI SQL standard with language constructs targeting the needs of its specific use cases – large-scale data warehousing and analytics.

Starting from the early 1980s, Teradata has imposed itself as a prominent provider for large-scale data warehousing and analytics with its robust and highly-parallel Database Management System (DBMS).

However, the demand for moving on-premises data warehouses to the cloud has increased over the recent years and relevant competitors providing cheaper and more agile solutions have emerged, e.g. Snowflake. An increasing number of customers has therefore started considering the migration of their Teradata solutions to more modern technologies.

Manual migrations might not represent a reasonable option for large-scale Teradata solutions. In these cases, (semi-)automated procedures are often preferred – and that is where a parser takes the stage (along with other precious components)!

Fantastic Parsers and Where to Find ThemThe term parser is part of our daily jargon at Strumenta, we know what a parser is, what can be done with it and how to build one to be proud of.

In short, a parser is a piece of software that understands the syntax of code written in some programming language – and can read and extract structured knowledge from it. This information is typically used in wider systems analyzing and manipulating source code, such as refactoring tools.

Want to know more about parsing? Take a look at this article we wrote some time ago.

Choose any language and there is probably going to be an open-source parser available for it, so why bother buying a commercial one? Glad you asked!

Developing high-quality parsers is time-consuming and requires knowledge of the subject. We are specialized in this field and we have built tons of parsers for our customers over the years. In doing so, we have developed a solid methodology to ensure high quality in our products from inception to maintenance. Our ready-to-go parsers are battle-tested and continuously improved over time in terms of language coverage and use cases.

Curious about what methodology we use to build parsers? We like to call it Chisel – check it out!

Introducing our Teradata SQL ParserOur Teradata SQL parser is implemented in Kotlin using ANTLR4 and Kolasu – the Kotlin implementation of our runtime libraries supporting the StarLasu methodology. The parser provides support for parsing Teradata SQL code from both strings and files and is distributed as a Java Archive (JAR) file.

It is possible to interact with the parser from the terminal through a simple Command-Line Interface (CLI). In the following snippets, we illustrate how this can be done using a file or string:

```

Parsing from file...java -jar parse # Parsing from string...java -jar parse "INPUT_STRING"

``` The parser can also be invoked programmatically if included as a dependency in a JVM project. It is worth mentioning that, while we are going to use Kotlin in the remaining sections of this article, our Teradata SQL parser can be used from Java and other JVM languages as well. The following code snippet illustrates a possible Gradle build configuration integrating the parser from its physical JAR file located into a libs folder:

implementation files('libs/kolasu-teradata-sql-parser-<VERSION>.jar') Once we’ve added the parser as a dependency, we can invoke it as follows:

// Declare the input file pathval path: Path = ...;// Create char stream from file pathval file: CharStream = CharStreams.fromPath(path);// Create new Teradata SQL parser instanceval parser: TeradataSqlKolasuParser = TeradataSqlKolasuParser();// Parse the input file using the parserval result: ParsingResult<TeradataSqlCompilationUnit> = parser.parse(file); We also provide a Python implementation of the Teradata SQL parser – which wraps the original JAR file and handles data exchange to integrate it within the Python world. The Python parser is distributed as an ordinary wheel package.

In this case, the distributed archive can be placed into a libs folder and declared as a dependency. For example, in a setup.py file, we could write:

setup( #...other configurations... install\_requires=[ f"teradata-sql-parser @ <archive\_absolute\_path>", #...other dependencies... ]) Once done, it will be possible to invoke the parser as follows:

```

required imports (pylasu)from pylasu.model.traversing import walk# required imports (teradata-sql-parser)from teradata_sql.parser import TeradataSqlParser# Declare the input file pathfilename: str = ...# Create new Teradata SQL parser instanceparser = TeradataSqlParser()# Parse the input file using the parserparsingResult = parser.get_ast(filename)

``` Please note that, in this case, the Kotlin executable is wrapped into a Python project handling data exchange. A valid Java installation is therefore still required (to execute the contained JAR executable file).

Extract Information from Teradata SQL CodeAt this point, you might be wondering what the ParsingResult data structure returned back by the parser contains and what could be done with it.

In addition to the root of parsed AST, the Teradata SQL parser returns back a list containing possible issues encountered during the parsing process. The parsing resultsimply contains these two sources of information.

The first operation that could be performed using the parsing result consists of verifying that no error occurred – i.e. the parsed source code is syntactically correct. This could be done by checking that the list of issues is empty, as illustrated in the following examples using Java:

assertTrue { parsingResult.issues.isEmpty() } and using Python:

assert len(parsing\_result.issues) == 0 Starting from the AST root, it is also possible to perform arbitrary traversals to implement various tasks. Let us consider the following Create Table Statement defining a table People with Id, Name and Age columns:

CREATE TABLE People (Id INT, Name VARCHAR(255), Age INT); Let us imagine now that we are interested in programmatically extracting information about the declared column names of the table and printing these to the standard output stream. The code snippet below illustrates how this could be achieved in Kotlin and Kolasu through the following steps:

1. First, we parse the initial statement and extract an AST using our Teradata SQL parser;
2. The parser will return back a parsing result from which we retrieve the AST root and start traversing its nodes;
3. Among the AST nodes, we focus our search on column definitions – i.e. TeradataSqlSimpleColumnPartition;
4. From each column definition, we extract a string representation of their declared name by processing their TeradataSqlQualifiedIdentifier;
5. Finally, we print out the extracted string representations to the standard output;

// store the input statement into a stringval input = "CREATE TABLE People(Id INT, Name VARCHAR(255), Age INT);"// (1) parse the input string containing the create table statementval parsingResult = KolasuTeradataSqlParser().parse(input); // (2) traverse the obtained AST from its root nodeparsingResult.root.walk() // (3) extract all column definition nodes.filterIsInstance<TeradataSqlSimpleColumnPartition>() // (4) extract name from each column definition.map { col -> col.fName.fIdentifiers.joinToString(".") { it.fValue } }// (5) print out the name for each column definition.forEach { println("Found column with name $it") } The same task can be performed using Pylasu and Python as follows:

```

required imports (pylasu)from pylasu.model.traversing import walk# required imports (teradata-sql-parser)from teradata_sql.ast import SimpleColumnPartitionDefinitionfrom teradata_sql.parser import TeradataSqlParser# store the input statement into a stringinput = "CREATE TABLE People(Id INT, Name VARCHAR(255), Age INT);"# (1) parse the input string containing the create table statementparsing_result = TeradataSqlParser().get_ast_from_string(input)# (2) traverse obtained AST from root node and (3) extract columns definitionscolumn_definitions = [node for node in walk(parsing_result.root) if isinstance(node, SimpleColumnPartitionDefinition)]# (4) extract name from each column definitioncolumn_names = [column_definition.column_name.name for column_definition in column_definitions]# (5) print out the name for each column definitionfor column_name in column_names: print(f"Found column with name {column_name}")

``` Executing the code on the initial statement will print out the following output:

Found column with name IdFound column with name NameFound column with name Age SummaryIn this article, we have seen how easy it is to use and be productive with our Teradata SQL parser. It is not perfect, but various companies battle-tested it, used it in production and will continue to do so.

If you are interested, please find out more about our parsers at Parser ready-to-go and schedule a meeting with us to get to know more.

Read moreYou can discover more about our Teradata SQL parser Ready-to-go here.

The post How to Use the Teradata SQL Parser appeared first on Strumenta.

View Details

In this article, we will present our RPG parser. It is based on our Chisel methodology and our open-source library, Kolasu. The parser is commercially licensed, but we hope this article will be useful for anybody wanting to learn what a parser is for and how to use one.

If you are looking to know what is a parser, we are going to explain it.

If you are looking to know what you can use a parser for, we are going to show you three examples of the things you can build with one:

• a custom software to understand which external programs your code calls
• a tool to detect duplicate code
• documentation in the form of diagrams

If you are looking to design your own parser, we are going to show you why you might want to follow our Chisel method to build one.

What Is a Parser (And What Is Not)We work with parsers every day, so for us it is very clear what a parser is and what it can do. However, this might not always be the case for our clients. So, let’s spend a few words on this point.

In general terms, a parser is a software that can understand the syntax, but not the semantics of some code.

Fundamentally, a parser is software that creates a model of the input, so we can extract data from it.

This model should be clear, useful, and easy to work with. The model that matches all these criteria is the Abstract Syntax Tree (AST). The practice of using an AST to represent code is an industry standard. Of course, how to design the AST for a particular language is open to debate.

A parser can read the code, but it cannot execute it. For example, a parser can both recognize a variable declaration and an expression.

What it cannot do is linking the two and understand where a variable used in an expression was declared. This feature is called symbol resolution and it is a functionality built on top of the parser.

Given the needs of our clients, we implemented some symbol resolution functionality. This is technically outside the scope of a parser, but it was needed, so we added it. For example, our RPG parser can resolve references to DDS definitions. We are going to see some examples of what our specific parser can do, later. However, it is an important point to keep in mind in case you are comparing different parsers.

Play Around on the PlaygroundWe have set up a platform to allow testing the RPG parser online, without the need to install anything. You can see what an AST looks like. You can test our RPG parser, or any of our other parsers for that matter, with some example files on our playground.

It is a great way to get a feel of how it works and whether it supports the features you care about. You can try it here: https://playground.strumenta.com. If you need it, you can ask us to test the parser on some examples you provide.

You can also read the documentation on the RPG parser online.

Chisel MethodologyThe RPG parser is based on the Chisel methodology. It is the missing link between source code and a convenient structure for its interpretation and manipulation: an AST. When building an interpreter, transpiler, compiler, editor, static analysis tool, etc., at Strumenta we always implement the software using a pipeline. A set of reusable components that can be shared for different projects.

For example, this is a pipeline for an RPG-to-Java transpiler.

The RPG parser and the Semantic Enricher components can be re-used for, let’s say, building an interpreter or an editor. This approach increases productivity and improves the quality of the software. For instance, any improvement to a core component for one project gets automatically shared with others.

StarLasu is the collection of runtime libraries that implement this methodology to support it in Java, Kotlin, Python, Javascript, Typescript, and C#.

At its core, StarLasu permits the definition of ASTs, on which all other functionalities are built. You can navigate and transform ASTs to do everything from reading the original values to simplifying your code. With the features provided by the library, you can do anything from analyzing a codebase to building a transpiler.

Some core features shared by our StarLasu libraries are:

• Navigation: utility methods to traverse, search, and modify the AST
• Serialization: export and print the AST as XML, as JSON, as a parse tree
• EMF and Lionweb interoperability: ASTs and their metamodel can be exported to the EMF or Lionweb formats

Interoperability with EMF and Lionweb is important because these are standard formats used in language engineering so you can mix and match different software, even from different providers.

You can read more about our methodology in a dedicated article.

What Do You Need an RPG Parser For?RPG is a language often used to build applications from the ground up. Companies have built custom ERP software, software to manager their warehouses, etc. Users have clusters of internal applications built over the decades to handle all their needs.

The software is custom, so it is tailored to the needs of the companies. However, it has grown organically over the years, so the code is often not well organized. It is not rare to have thousands of files without knowing which ones constitute a separate program. So one typical use case is to use an RPG parser to perform analysis and understand which files create each program and how everything works together. A related application is creating documentation for the code, like representing the code behavior with UML diagrams.

RPG is still used in many companies but it was developed decades ago. It works ok when you are maintaining old and thoroughly tested programs, but it is less productive for creating new programs. It is also not great when you need to integrate new technologies. The reason is partly because of the long history and partly because of the design patterns that were common at that time.

For instance, the data definition and the database are effectively part of the program. This is a consequence of the system it was used in.

RPG Programs Are Integrated With Their PlatformHaving a long history implies that it is important to preserve backward compatibility. IBM leadership was still very much concerned with selling hardware, so the idea was to make the software integrated with the whole platform (i.e., the operating system that was designed for the specific line of hardware). So, while RPG itself is just a language, programs are often integrated with the systems they were designed for. This is simply how programs were developed back then.

These are just a general description of the challenges, we have written an article to discuss the issues with legacy code on a platform using RPG, like the AS/400 or System i.

When such challenges outweigh the benefits of RPG, you want to build a transpiler. Rewriting all the code from scratch is risky and costly. Throwing all the custom code away results in less productivity, given that the software was designed for the company. So you want to migrate the code from RPG and do it automatically. The way to do that is with a transpiler. And for building that you first need a parser to understand the code.

There are also standard uses for a parser that are valid for an RPG parser,, too. For instance you can use s a parser for building an editor: it will help the editor to provide syntax highlighting and autocompletion. You can create a static analysis tool, etc.

Why Use a Ready-to-go Parser?You need a parser that has been thoroughly tested, which is documented, and gives you someone to call in case you encounter any problems.

We are experts and we have built tons of parsers for our clients. This means that we completely understand the importance of this component and we have a solid methodology. And we build parsers designed for what our users need. For example, our SAS parser is geared to support data lineage, because that is what the typical user needs.

Our RPG Parser has also been built for the needs of our clients, so it is battle-tested and used in production. It supports all RPG statements and most Data Description Specifications (DDS).

It can handle both the traditional fixed format, free format, and the two formats mixed together. This is important because the fixed format is quite obnoxious to use, so new developers prefer start using the free format as soon as possible. The result is that actively maintained files soon go from fixed format to mixed format.

How to Setup the ParserThe only two requirements you need to use the parser are rpg-parser and Kolasu packages. For example, for a Java Maven, you would write something like this.

<dependencies> <dependency> <groupId>com.strumenta</groupId> <artifactId>rpg-parser</artifactId> <version>2.1.52</version> </dependency> <dependency> <groupId>com.strumenta.kolasu</groupId> <artifactId>kolasu-javalib</artifactId> <version>1.5.56</version> </dependency> </dependencies> This would use the rpg-parser and kolasu-javalib, the version of Kolasu tailored for use from Java.

You can easily adapt this for another build system like Gradle.

dependencies { implementation "com.strumenta.kolasu:kolasu-core:1.5.56" implementation "com.strumenta.kolasu:kolasu-javalib:1.5.56" implementation "com.strumenta:rpg-parser:2.1.52.} That is all you need to be able to use the parser in your code just as easily as any other library.

Understanding Which Programs Your Code CallsOne feature of RPG is that you can easily call external programs with just one statement. Imagine you want to get an understanding of all the external programs every file calls.

So, you want to collect all statements like the following one.

C CALL 'MAGIC' C PARM NAME C PARM PASS You can do that, all with a few lines of code. We are going to use Kotlin for this example, but of course, you could also do the same with Java.

val compilationUnits = File("examples\_dir/").walk().map { RPGKolasuParser.parserFromExtension(it).parse(FileInputStream(it))}compilationUnits.forEach { assert(it.root != null) assert(it.correct)}var calls = compilationUnits.flatMap { it.root!!.walk().filterIsInstance<CallProgramStatement>()}.toList() In the first statement, we parse all the examples in the creatively called folder examples_dir.

In the second one, we ensure that the parsing succeeded, by checking that root of the AST is a valid node and no issues have been reported.

Then we walk through all the ASTs, one for each file, and we look for nodes of type CallProgramStatement. The walk method comes from Kolasu; it explores the whole tree, visiting each node. Flatmap is a method from the standard Kotlin library that transforms a series of lists into one list.

How to Detect Duplicate CodeRPG was designed before was common to encapsulate code into classes or even functions. You can separate code into procedures, but you can also put all your statements in the global scope. One of the consequences of this pattern is that a lot of code gets reused.

Imagine you have a program to calculate sales taxes in state A and you need to calculate sales taxes in state B. You just copy the code from the first program and start changing some bits here and there.

This is not ideal, because if you ever find a bug, you will probably fix it in one program but not the other one. The problem is how do you find all that similar code and who wants to fix all that code? Well, we cannot help you with finding the will, but the RPG parser can help you in detecting similarities.

Why We Need the RPG LexerWhat you need to use is to use the lexer that comes with the parser. You can think of the lexer as the first step of the whole parser. The lexer identifies simple patterns in the text and transforms them into tokens. For instance, it takes a series of characters enclosed between two quotes becomes a STRING token. The parser will then look at this sequence of tokens for the whole file and use it to create an AST. So, the lexer reads the stream of characters of the input and transforms into a stream of tokens. The parser organize this stream of tokens into an AST.

This image represents the relationship between the two.

If you are interested in more information about the whole parsing process, you can read it in our Guide to Parsing: Algorithms and Terminology.

You need to use the lexer because it is difficult to compare two ASTs, since they might have a different structure. Imagine having two files with the same statements but in one file they are contained in different subroutines, but in the other one are directly in the global scope. It is difficult to find a way to detect similarities between them programmatically. This in addition to the practical difficulties of equality comparison between complex objects.

It is Just a List of TokensIt is easier to do this with the result of lexing, because that is just a list of tokens. So, all you need to do is to compare the elements of two lists.

val lexingResult1 = RPGKolasuLexer().lex(File("file1.rpgle/"))val lexingResult2 = RPGKolasuLexer().lex(File("file2.rpgle/") You can lex a file by using RPGKolasuLexer and the method lex.

Then you can compare the two lists of tokens.

val codes : List<RPGToken> findSimilarPiecesOfCode(lexingResult1, lexingResult2) Now, there are many possible ways of comparing two sequences. Here we choose a trivial one. We look at the first element of the first sequence. We compare this element with any of the second sequence until we find a match or we reach the end of the second sequence.

If we find a match, we advance in parallel, adding the text of the match to our list of matches (we save them in the variable codes).

fun findSimilarPiecesOfCode(first : LexingResult<RPGToken>, second: LexingResult<RPGToken>) : List<String> { val codes = mutableListOf<String>() var indexSecond = 0 var index = 0 var lastMatch = 0 while(index < first.tokens.size) { if(first.tokens[index] == second.tokens[indexSecond]) { var text = "" while(first.tokens[index] == second.tokens[indexSecond] && index < first.tokens.size && indexSecond < second.tokens.size) { text += first.tokens[index].text index++ indexSecond++ } lastMatch = indexSecond codes.add(text) } else { indexSecond++ } if (indexSecond >= second.tokens.size) { indexSecond = lastMatch index++ } } return codes} When we reach the end of the second sequence, we advance the first sequence by one and restart the comparison from the last match with the second sequence.

This is obviously a simple way of doing a comparison, riddled with caveats, but it works in getting a quick list of equal sections of code.

Generate Documentation Diagrams From CodeRPG is not a bad language, but it is hard to follow what the code does at a glance. If nothing else for the impossibility of using formatting and whitespace to group code in a logical way.

So, one thing you might want to do is to generate diagrams to represent the workflow of code. A diagram would also make the code easier to understand for people new to RPG and therefore do not understand its syntax.

You want to get a diagram like this one.

This is a diagram generated by PlantUML, so all we need to is to transform the AST into a textual representation understandable by PlantUML, which will generate the diagram.

The textual representation looks like this.

@startuml'https://plantuml.com/sequence-diagram!pragma teoz truehide footboxskinparam sequence { ArrowColor Black LifeLineBorderColor #000000 LifeLineBackgroundColor #FFFFFF ParticipantBorderColor #000000 ParticipantBackgroundColor #FFFFFF ParticipantFontColor #000000}client -> CUS300.rpgle :CUS300.rpgle -> CUS300.rpgle : inzsrCUS300.rpgle -> clrsum : clrsum[..] So it is basically a list of configuration statements. We need to get from the AST to this.

In Where We Accidentally Build a TranspilerWe already know how to parse the code, so we assume that you have done that and now have an AST that you need to transform into a diagram.

In the following example, an AST is given as the argument model of the method.

/** * Transforms a given RPG AST into a PUML diagram. * * @param model the RPG AST to be transformed. * @return the AST representing the PUML diagram. * @throws Exception if the transformation fails. */@Overridepublic Node transform(Node model) throws Exception { PUMLDiagram target = new PUMLDiagram(); String file = inputFile.getName(); if (model instanceof CompilationUnit) { CompilationUnit cu = (CompilationUnit) model; // RPG code contains an initialization routine it is executed first for (Subroutine s : cu.getSubroutines()) { if (s.isInitializationSubroutine()) { target.add( new PUMInvoke(file, file, "inzsr", List.of() )); } } // Process the main statements for (Statement s : cu.getMainStatements()) { target.add(transformStatement(cu, s)); } return target; } throw new Exception(String.format("Invalid input Model: %s", model.getClass().getName()));} As you can see, it is fairly easy, we first check for any standard initialization routine. This is automatically executed, so we need to add it to a diagram manually. That is because the call is implicit and not in the code itself.

Then we walk through all the global statements, that are conveniently accessible using the getMainStatements method in the AST root (CompilationUnit).

The transformStatement method is the one doing the work of transforming an AST node into an element of a PlantUML diagram. Each statement is then transformed into a string. For instance, an IfSstatement of RPG is transformed into an IF node for PlantUML, PUMLIf. Then the PUMLIf node is translated into a string with a string formatter.

nodePrinters.put(PUMLIf.class, (statement) -> { PUMLIf s = (PUMLIf) statement; ArrayList<String> sequence = new ArrayList<>(); sequence.add(String.format("group IF %s",s.getCondition()) ); for(PUMLStatement ps : s.getBody()) { sequence.add(print(ps)); } sequence.add("end"); return String.join("\n",sequence); }); Then this list of strings is inserted into a template to generate the final textual representation. This way, we also accidentally learned how to create a transpiler. Because that is a transpiler: a software that transforms a code in one language into another one. In this case RPG Code into PlantUML code. Our production transpilers are more sophisticated, but the general principle remains the same: you transform the original AST into your target AST and then transform your target AST into text.

If you are interested in looking deeper into this example, you can read the dedicated article on how to use the RPG parser to generate sequence diagrams.

Using the Parser From the Command LineMost users want to integrate the parser into their software, so they are going to use it as a library. However, you can use the parser JAR just as a command line tool, to create save your AST into a EMF representation, if you wish. It is as easy as this.

java -jar rpgparser-<verion>.jar <input.rpgle> --output ./ This would generate a JSON file for each input file (you can also provide a directory as input). Then you could take advantage of any EMF library to work with the result. This feature is included with every Kolasu-based parser, so you do not have to worry about it. And you can also have it for free if you build your own Kolasu-based parser.

Performing Symbol Resolution on an RPG ProjectA typical situation that we need to deal with is that a company has a bunch of files and a few software. The problem is that they do not know which files implement which software. That might seem surprising if you have never used RPG. If you have, you know why that happens. Given that RPG software is supposed to be integrated with the whole platform, calling an external program is just as easy as calling a function.

The issue is that this confusion makes it harder to migrate the whole software stack. You want to start the migration from the programs that have no dependencies on others and move up from there. This is easier to manage and reduces the risk of getting stuck in the migration. So, your first step is to identify the programs you have.

Given the reality of RPG codebases to do that you need to solve all references in a RPG program. So, all you start with is a bunch of RPG files and a bunch of DDS files and you need to understand which ones go together. To do that you can follow how the Data Descriptions are used in an RPG file and from which DDS comes from.

All you need to do is write three lines of code.

val compilationUnit: CompilationUnit = parseRPGLE(File("src/test/resources/rpgle/CALCFIBF.rpgle"))RPGExternalProcessor.resolve(compilationUnit, ddsFolder = File("src/test/resources/dds"))rpgSemantics().symbolResolver.resolve(compilationUnit) The first line parses the RPG code.

The second one tells our processor where the DDS files are.

The third one resolves the symbols.

Symbol Resolution in ActionThis code takes a file like this.

* Calculates number of Fibonacci in an iterative way D NBR S 8 0 D RESULT S 8 0 INZ(0) D COUNT S 8 0 D A S 8 0 INZ(0) D B S 8 0 INZ(1) D DSP S 50 FLOGFILE Up E Disk FPHYFILE2 Up E Disk *--------------------------------------------------------------* [..] *--------------------------------------------------------------* C *INZSR BEGSR C EVAL DSP='INZSR' C DSPLY DSP C ENDSR C* Entry Point C *LOVAL SETLL LOGFILE C READ LOGFILE C DOW NOT %EOF(LOGFILE) C EVAL NBR = %DEC(SNBR : 8 : 0) And a DDS file, called LOGFILE.dds like this one. A* A********************************************************************** A R RCALCFIB PFILE(PHYFILE1) A K SNBR And a DDS file, called LOGFILE.dds like this one.

A* A********************************************************************** A R RCALCFIB PFILE(PHYFILE1) A K SNBR And resolve all symbols, providing information on symbols like the one used in the SetLowerLimit statement LOGFILE, in the third-to-last line. The resolver first looks up the FileEntry FLOGFILE to identify the file and then it can link to the proper definition in the file from any reference in the source code.

The reference to SNBR in the last line of the code is now linked to the proper Data definition in LOGFILE.dds.

SummaryWe hope you have seen how easy it is to use our RPG parser. It has been battle-tested and is used in production by various companies.

We have distilled our knowledge about parsing into this software, to make the most productive RPG parser out there. Built using a solid methodology, that is open to all and can be applied to build your software. This multiplies our ability to improve the software we build.

We have seen how to use the parser to identify how programs call each other, to find similarities between files, and even to document the code.

You can start playing with the parser on the Strumenta Playground.

ResourcesPlay with parsers on the Strumenta Playground

Discover our parsers Ready-to-go

The post How to Use the RPG Parser appeared first on Strumenta.

View Details

The challenge will be transforming RPG code into Clear Sequence Diagrams. In the dynamic landscape of software development, legacy languages like RPG continue to play a crucial role, particularly in sectors like finance and banking. Navigating and maintaining legacy code, however, presents a unique set of challenges. Addressing this gap, Strumenta has developed an RPG parser that can be used to develop many tools such as transpilers, language servers, or code analysis tools. In this article, we present a simple Java implementation of a transpiler that converts RPG code into a PlantUML sequence diagram.

RPG parserBefore jumping into the implementation let’s take a look at the features of the Strumenta RPG parser. The parser supports fixed and free format RPG code and it can also parse physical and logical file definitions (DDS). A symbol resolution module is available as an extension of the parser. The parser is written in Kotlin and is based on the open source Kolasu AST library. The code in this article uses the Java programming language to illustrate that Kotlin is not a requirement to run the parser. The parser produces as output an Abstract Syntax Tree (AST) that can be traversed and processed programmatically using the Kolasu API.

The Language Engineering PipelineThe Java code presented is a basic implementation of the Language Engineering Pipeline architecture, a structured approach to language processing and translation. This approach is rooted in Model Driven Development, where models (in this case the Abstract Syntax Trees) play a crucial role.

In this architecture, the output of a component is used as input for the next component in the pipeline. The diagram below illustrates the conceptual architecture of the pipeline.

The pipeline has 3 main components:

• SourceToModel: This component focuses on parsing the RPG source code using the RPG parser and generating the model (AST).
• ModelToModel: This component is responsible for the core conversion logic. It walks the RPG AST, analyzing each statement or expression type. Based on the type, it employs the transformation conversion logic to generate the corresponding PlantUML AST node.
• ModelToSource: This component bridges the gap between the internal PlantUML model and the textual PlantUML code.

The Java implementation of the pipeline looks like the following code.

You can find the repository at the following link: RPGtoPUML.java

Pipeline pipeline = new Pipeline( new SourceToModel(inputFile,outputFile), new ModelToModel(inputFile,outputFile), new ModelToSource(inputFile,outputFile) ); Each component of the pipeline is implemented in a specific Java class that performs a specific transformation. For example, the SourceToModelclass makes use of the Strumenta RPG parser to transform the RPGLE source code into the corresponding AST.

You can find the repository at the following link: SourceToModel.java

/** * Performs the transformation of RPG source code into an AST. * Reads the source code from the input file, parses it using * RPGKolasuParser and returns the resulting AST. * * @param model Not used in this implementation as the transformation is * from source code to AST. * @return The root node of the generated AST representing the * parsed RPG code. * @throws Exception if the input file is invalid or the parsing fails. */@Overridepublic Node transform(Node model) throws Exception { if (inputFile.isFile() && inputFile.exists()) { RPGKolasuParser rpgParser = RPGKolasuParser.parserFromExtension(inputFile); ParsingResult result = rpgParser.parse(inputFile); if(result.getCorrect()) { return result.getRoot(); } } throw new Exception(String.format("Invalid file '%s'", inputFile.toPath())); } The code shows how the RPGKolasuParseris instantiated, in this case detecting the extension of the input file, and then the instance is used to parse the file and create the AST. Although the RPG parser is implemented in Kotlin, the integration in Java happens seamlessly.

The second step ModelToModeltransforms an RPG AST into the PUML (PlantUML) AST and provides the implementation for transforming various RPG statements into their corresponding PUML AST node representations. Below is the class diagram of the PlantUML AST.

You can find the repository at the following link: PUMLDiagram.java

The abstract syntax tree implements just a few of the constructs available in PlantUML, but it is enough for this example. The class PUMLNode extends the Node class provided by the Kolasu AST library. Below is the Java code of the ModelToModel transformation implementation.

You can find the repository at the following link: ModelToModel.java

/** * Transforms a given RPG AST into a PUML diagram. * * @param model the RPG AST to be transformed. * @return the AST representing the PUML diagram. * @throws Exception if the transformation fails. */@Overridepublic Node transform(Node model) throws Exception { PUMLDiagram target = new PUMLDiagram(); String file = inputFile.getName(); if (model instanceof CompilationUnit) { CompilationUnit cu = (CompilationUnit) model; // RPG code contains an initialization routine it is executed first for (Subroutine s : cu.getSubroutines()) { if (s.isInitializationSubroutine()) { target.add( new PUMInvoke(file, file, "inzsr", List.of() )); } } // Process the main statements for (Statement s : cu.getMainStatements()) { target.add(transformStatement(cu, s)); } return target; } throw new Exception(String.format("Invalid input Model: %s", model.getClass().getName()));} The code performs a loop on the Subroutinecollection to find if there is an initialization subroutine (INZSR in RPG). If present, it adds it as the first invocation. The transformStatement processes the AST nodes of the main program statements. The implementation in this example is designed to identify some common patterns in RPG, such as loops and file operations.

The last step of the pipeline, ModelToSource, consists of the PlantUML code generation. It utilizes a template-based approach, where placeholders are filled with the generated PlantUML statements based on the processed AST. Additionally, it employs a strategy pattern (using a nodePrinters map) to associate specific PUML node types with their corresponding printing functions, ensuring proper code generation for different elements.

Check out PUMLCodeGenerator.java source code on GitHub.

The table below summarizes some of the various constructs of the RPG code that are transformed into PlantUML code and how they are displayed. CUS300.rpgleis a sample program.

| RPG code | PlantUML code | Diagram | | INZSR | CUS300.rpgle -> CUS300.rpgle : inzsr | | | SETLL LOVAL CUSTOMER | CUS300.rpgle -> CUSTOMER : SETLL LOVAL CUSTOMER | | | READ CUSTOMER | CUS300.rpgle -> CUSTOMER : READ CUSTOMER | | | DOU NOT %EOF(CUSTOMER) | loop UNTIL NOT %EOF(CUSTOMER)end | | | EXSR clrsum | CUS300.rpgle -> clrsum : clrsum | |

In this implementation, in addition to the calls to subroutines, we also wanted to highlight the access to the files so that by examining the sequence diagram it would be possible to identify what files are involved and their mode of operation (Read/Write).

Generating the DiagramLet’s examine the example RPG program used in this article and the relative PlantUML sequence diagram. The CUS300.rpgleprogram uses 3 files: CUSTOMER, ORDERS, and ORDSUM; it defines the INZSR initialization subroutine; and defines 3 subroutines: clrsum,dspcus, and calctotal.

The code is written in free format and contains some classic RPG constructs to read/write data files.

CUS300.rpgle

F**********************************************************************F* *F* PROGRAM ID : CUS300 *F* PROGRAM NAME: SAMPLE PROGRAM *F* *F**********************************************************************D DSP S 50 INZ('CUSTOMER')D TOTAL S 9P 2 INZ(0)D NUM S 9P 2 INZ(1)D CNT S 9P 2 INZ(0)FCUSTOMER UF E K DiskFORDERS IF E K DiskFORDSUM IF E K DiskC *INZSR BEGSRC EVAL DSP='CUSTOMER REPORT'C DSPLY DSPC ENDSR/free CNT = 0; TOTAL = 0; EXSR clrsum; DSPLY '------ Forward ------'; Setll *Loval CUSTOMER; Dou NOT %EOF(CUSTOMER); Read CUSTOMER; If NOT %EOF(CUSTOMER); EXSR calctotal; EXSR dspcus; If TOTAL > 0; OSCUID = CUID; TOTAL *= (TOTAL / CNT +1) * 0.1; OSTOT = TOTAL; OSCUNM = CUSTNM; Write ORDSUM; EndIf; EndIf; EndDO; Begsr calctotal; CNT = 0; TOTAL = 0; Setll *Loval ORDERS; Dou NOT %EOF(ORDERS); Read ORDERS; If NOT %EOF(ORDERS); If CUID = ORCUID; TOTAL += ORTOT; CNT += 1; Update ORDERS; EndIf; EndIf; EndDo; EndSr; Begsr dspcus; If TOTAL > 0; eval DSP='CUSTOMER: ' + CUSTNM + ' $' + TOTAL; DSPLY DSP; EndIf; EndSr; Begsr clrsum; CNT = 0; DSPLY '------ Delete ------'; Setll *Loval ORDSUM; Dou NOT %EOF(ORDSUM); Read ORDSUM; If NOT %EOF(ORDSUM); delete ORDSUM; CNT+=1; EndIf; EndDO; DSPLY 'DELETED: ' + CNT + ' RECORDS'; EndSr; Running the transpiler on the CUS300.rpgle file will generate the PlantUML diagram file named CUS300.rpgle.puml.

@startuml'https://plantuml.com/sequence-diagram!pragma teoz truehide footboxskinparam sequence { ArrowColor Black LifeLineBorderColor #000000 LifeLineBackgroundColor #FFFFFF ParticipantBorderColor #000000 ParticipantBackgroundColor #FFFFFF ParticipantFontColor #000000}client -> CUS300.rpgle :CUS300.rpgle -> CUS300.rpgle : inzsrCUS300.rpgle -> clrsum : clrsumclrsum -> ORDSUM : SETLL *LOVAL ORDSUMloop UNTIL NOT %EOF(ORDSUM)clrsum -> ORDSUM : READ ORDSUM group IF NOT %EOF(ORDSUM)clrsum -> ORDSUM : DELETE ORDSUMendendCUS300.rpgle -> CUSTOMER : SETLL *LOVAL CUSTOMERloop UNTIL NOT %EOF(CUSTOMER)CUS300.rpgle -> CUSTOMER : READ CUSTOMERgroup IF NOT %EOF(CUSTOMER)CUS300.rpgle -> calctotal : calctotalcalctotal -> ORDERS : SETLL *LOVAL ORDERSloop UNTIL NOT %EOF(ORDERS)calctotal -> ORDERS : READ ORDERSgroup IF NOT %EOF(ORDERS)group IF CUID = ORCUIDcalctotal -> ORDERS : UPDATE ORDERSendendendCUS300.rpgle -> dspcus : dspcusgroup IF TOTAL > 0endgroup IF TOTAL > 0CUS300.rpgle -> ORDSUM : WRITE ORDSUMendendend@enduml Below is the diagram corresponding to the PlantUML code.

The diagram presents the sequence of operations performed by the RPG program. It starts with the execution of INZSR then executes the crlsum subroutine which executes a loop on the record of the ORDSUMfile and performs a delete operation. Then the program continues processing the record of the CUSTOMER table and so forth. Transforming RPG code into PlantUML sequence diagrams offers several advantages:

• Improved Code Comprehension: Visualizing the program flow through a PlantUML diagram makes it easier to understand the relationships between different parts of the code, especially for complex logic or interactions between subroutines.
• Enhanced Collaboration: PlantUML diagrams provide a universal language for developers to discuss and document the program’s functionality. This can improve communication and collaboration within a team.
• Efficient Debugging: By visualizing the execution flow, pinpointing errors or unexpected behavior in the code becomes more straightforward. Developers can identify issues by tracing the diagram and analyzing the interactions between elements.

The Java code presented in this example can be easily customized by implementing a specific logic for selected RPG statements. The same pipeline could be used to generate code for other text-to-diagram tools such as d2lang.

Once your diagrams are in text you can transform your text-based diagrams into a powerhouse of insights with your preferred Large Language Model (LLM), effortlessly generating detailed documentation, obtaining tailored best practices, and unlocking optimization strategies that align perfectly with your architecture. But that’s not all – take it a step further by vectorizing these insights for seamless storage in a vector database to elevate your project’s potential and streamline your workflow in ways you never thought possible.

Source code available on GitHub: https://github.com/Strumenta/rpg-puml-sequence

SummaryIn this article, we have explored the features of the Strumenta RPG parser to create a configurable pipeline approach to the development of language processing tools, allowing developers to create a transpiler from RPG to PlantUML easily.

With its support for ASTs, traversing, transformations, and cross-referencing, the RPG parser

offers a comprehensive set of tools for working with source code.

By leveraging these features, developers can build powerful language processing tools to explore things that simply could not be seen before.

The post Transforming RPG Code into Clear Sequence Diagrams appeared first on Strumenta.

View Details

Sneak peekFood. For many, one of the great pleasures of life. Some would argue that programming is also up there. Why not mix both? Have your cake and code it, too. We will build an editor with Kolasu and Language Servers.

In this article, we will design a domain-specific language for cooking recipes and build a code editor for it:

We will explore how to create an editor with syntax highlighting, error diagnostics, semantic validation, symbol navigation, code generation, and more.

All under 1000 lines of code by leveraging the power of open source tools: Kolasu, VSCode, and a newborn Strumenta plugin for creating language servers.

The entire codebase of the project is on GitHub. To keep the article lightweight, we will reference files from the repository and focus on explaining their most important points.

Let’s do this, byte by byte.

A language for cooking recipesLet me introduce you to Kuki, a language for cooking recipes:

We have recipes for almond cookies, Spanish omelets, and even spaghetti carbonara, that will likely trigger some of my Italian colleagues. What can I say, it is spaghetti code after all.

We can model the domain with an AST. Recipes have a list of ingredients, utensils, and steps to prepare the dish. We have polymorphic steps to represent cooking actions like Mix or Bake.

Notice that we have dedicated nodes for item declarations and references:

data class ItemDeclaration(override val name: String): Node(), Nameddata class ItemReference(val reference: ReferenceByName<ItemDeclaration>): Node() This is useful for storing fine-grained position information and implementing symbol navigation later.

Items can be declared in the ingredients and utensils blocks, but also inside a step. For example, the following Mix step declares a dough item that is referenced in the next step:

1. Mix the almond, sugar and egg white into dough.2. Slice the dough into circles With the mental model of the code ready, we can follow the Chisel method to build the parser.

We create the ast Gradle submodule, define an ANTLR lexer and parser, and map the ANTLR parse tree to the AST model using a Kolasu transformer.

Nothing exceptional here, we add optional punctuation and English words to try to make the language more friendly and allow identifiers with spaces for ingredients like egg white or black pepper.

itemDeclaration: ID+;itemReference: ID+;itemList: THE? items+=itemReference ((COMMA THE? items+=itemReference)* AND THE? items+=itemReference)?; Lastly, we create a Kolasu parser that given a string of code will parse it with ANTLR and return to us the transformed AST. We can check this with tests:

fun testAlmondCookies() { val code = File("../examples/Almond cookies.kuki").readText() val parser = KukiKolasuParser() val parsingResult = parser.parse(code) assertEquals(0, parsingResult.issues.size)} Tests are great for avoiding regressions, but having an editor lets us play with the code and get the issues reported in real-time.

Testing the language on an editorWith a Kolasu parser in hand, we can create a language server and a VSCode extension with the kolasu-language-server Gradle plugin—a new open-source initiative at Strumenta for facilitating the creation of language servers.

For the time being, the plugin only supports Java 18+. We are actively developing the plugin and supporting libraries. This article will focus on its first public release, version “1.0.0”.

First, we initialize the language-server Gradle submodule with the plugin. The plugin follows Gradle’s convention over configuration philosophy and expects a standard project configuration by default. However, most of the process can be configured by adding a languageServer configuration block in the Gradle build script.

In this tutorial, we will follow the expected standard layout and naming conventions. In particular, it is important to set the name of the project in the root Gradle project’s settings:

rootProject.name = "kuki" The language server plugin adds two tasks to the project under the language server group:

Running the createVscodeExtension generates an entry point file, we will customize it later.

It also creates a valid VSCode extension under build/vscode. We can test the extension by running the launchVscodeEditor task.

The task launches a code (VSCode) process that opens the examples folder of the project.

Now, we can edit the recipes in real-time and observe the parsing errors as diagnostics positioned over the code.

We can also navigate the code more easily by opening the outline view. Every node with a name has an entry in the outline hierarchy by default.

That’s a good start, but language servers can do much more.

Add some colorThe more colors in a dish, the tastier it seems. Adding colors to an editor just makes the programmers more productive.

We can add syntax highlighting by specifying a Textmate grammar. If we place it at scr/main/resources/grammar.tmLanguage the plugin will pick it up automatically. For this language, we simply associate regular expressions with token categories:

<dict> <key>name</key> <string>variable.other.kuki</string> <key>match</key> <string>\b([a-zA-Z]+)\b</string></dict> Then, the editor themes will map each token category to a color in their palette. Try changing themes quickly for some psychedelic action.

Similarly, we can specify a custom file icon for the recognized files by placing an image in src/main/resources/fileIcon.png. How about a chef hat?

We can reuse it as the logo of the extension by placing a copy in src/main/resources/logo.png.

Finally, let’s explore how to customize the outline generated by the language server. All the functionalities of the language server can be overridden by subclassing it. In this case, we want to override the symbol kind associated with each named node:

override fun symbolKindOf(node: Node): SymbolKind { return if (node is Step) { SymbolKind.Function } else if (node.parent is Utensil) { SymbolKind.Struct } else { SymbolKind.Variable }} Now, we can distinguish between ingredients, utensils, and steps in the outline:

Symbol resolution and symbol navigationMany programming languages have variables that can be declared and referenced afterward. Our language Kuki is no exception: We declare ingredients and utensils and reference them in the preparation steps.

Kuki follows common semantic rules for variables:

• Each item can be declared only once
• Every item needs to be referenced in the preparation steps
• Items need to be declared before being referenced

Considering all variables are global in the file, we have implemented an ad hoc symbol resolver. It walks the tree depth-first keeping track of the declared symbols and creates an issue when the validation rules fail:

fun define (name: String, item: ItemDeclaration) { if (declarations[name] == null) {= declarations[name] = item } else { issues.add(Issue.semantic("$name already declared", position = item.position)) }}fun checkReferences (items: List<ItemReference>) { for (item in items) { val declaration = declarations[item.reference.name] if (declaration == null) { issues.add(Issue.semantic(item.reference.name + " is undeclared", position = item.position)) } else { item.reference.referred = declaration referencedDeclarations.add(item.reference.name) } }} When symbols are correctly referenced, we resolve the symbol by setting their referred attribute. For more advanced languages, Kolasu has a declarative API for implementing symbol resolution and semantic validation.

With these improvements, the editor will show error diagnostics when semantic validation fails:

In this example, we get two issues:

• The symbol Egg white is never used
• The symbol water has not been declared before referencing it

However, we can enrich the editor further by enabling symbol navigation. The language server specifies its capabilities during initialization. Let’s override the initialize method to enable the Go to Definition and References features:

capabilities.setDefinitionProvider(true)capabilities.setReferencesProvider(true) Now, we can navigate by right-clicking a symbol and selecting Go to Definition or Find References. Alternatively, we can use the VSCode keybindings (F12 and Shift + F12 respectively). This feature becomes more and more useful the bigger the project gets.

Code generationFinally, we will add a code generation module to support a full compiler pipeline: From parsing code to validating it and generating some output.

For our example, we will generate three versions of the recipe, adapting the ingredients for 1, 2, and 4 people. We will standardize the formatting and syntax of the generated recipes.

The code generator itself walks the tree and serializes it back to code using string templating. It repeats the process adapting the ingredients for 1, 2, and 4 people, and writes the output code to a file under the Recipe book folder:

for (amount in listOf(1, 2, 4)) { val file = File(URI(uri)).parent + File.separator + "Recipe book" + File.separator + tree.name + File.separator + tree.name + " for $amount.txt" val code = generateRecipeFor(amount, tree) File(file).parentFile.mkdirs() File(file).writeText(code)} For more complex scenarios, I suggest taking a look at Kolasu’s declarative API for code generation.

To integrate the code generator with the language server, we need to modify the KukiServer again. We can pass the code generator instance in the constructor:

class KukiServer : KolasuServer<Recipe>(KukiKolasuParser(), "kuki", listOf("kuki"), generator = KukiCodeGenerator()) { and enable the save notifications in the server initialization:

capabilities.setTextDocumentSync(TextDocumentSyncOptions().apply { openClose = true change = TextDocumentSyncKind.Full save = Either.forLeft(true)}) Now, whenever the user saves a Kuki file, the code generator will kick in and generate the recipe versions under the Recipe book:

Here is what they look like:

SummaryWe have learned how to create an editor for editing cooking recipes with symbol navigation, code generation, syntax highlighting, code outlines, and error diagnostics. Of course, the same procedure applies to different languages and domains.

But now that we have built a tool to digitize and edit recipes comfortably, how about adding more domain-specific features like:

• Finding recipes that use an ingredient
• Classifying vegetarian and celiac-friendly recipes
• Estimating the total time for preparation
• Computing the nutritional value starting from the ingredients

Then, we could fetch vegan recipes with aubergine that are low in fats and can be prepared in under 10 minutes. In the meantime, I will stick with my spaghetti carbonara.

The post Build an Editor With Kolasu and Language Servers appeared first on Strumenta.

View Details

Your company depends on codebases older than the new intern hired last week. So what? Should legacy modernization be part of your strategy?

In this article, we discuss the implications of having a legacy codebase. Let me spoil the conclusions: in some cases it is just fine, while in other cases there are risks connected to this situation. We will discuss how to distinguish these situations and what to do about them.

Can you tell me if this is going to be fine?When you ask the doctor if you are fine,they usually respond with a series of questions, starting from ‘Where does it hurt?’. Now, I am not that kind of doctor, but I will ask a few questions of that sort anyway. Based on your answers we can understand if you are fine. And there is no need to lie, as I promise that you will not get a shot in any case.

We can look at this from two points of view: the business perspective and the technical perspective.

The business perspectiveFrom the business perspective we should consider:

• Are you spending a ton of money on hardware?

This could be the case if your code can only run on certain proprietary hardware that certain vendors rent to you for a substantial amount of money (did someone say IBM?). The question you should consider is: irrespectively for the absolute amount, is that cost making a difference for you? Is paying this price making your business less viable?

• Are you spending a ton of money on software licenses?

This could be the case if you run software depending on some software platform that is licensed to you for a non trivial amount of money (did someone say SAS?)

• Are you worried about the future of the platform you are using?

Are you depending on hardware or software that you think could not be supported a few years from now? This is a tricky one, as it requires you to be able to predict the future, to some extent. Because you cannot just wait for your vendor to go out of business or stop supporting a product, but you need to anticipate it. The reason is that, if you need to migrate, you will need some time and you do not want to do that under a crazy deadline.

• Do you have issues finding the right developers?

If you cannot find at all developers who can maintain your system, that seems a pretty clear sign that you have a problem. But, depending on your situation, you could have a problem also if you cannot find skilled developers and you need to train them yourself. In this scenario you may end up spending a lot of time and money to train your developers, just to have them leave for greener pastures as they get a bit of experience under their belt. This may be a problem, because it takes time to familiarize with your venerable code base.

The technical perspectiveFrom the technical perspective we should consider:

• Do you have issues scaling the system to support more users or process more data?

Is your system having issues coping with a growing user base? Are batch tasks that used to run in one hour now taking the whole night to be completed? Is your system designated in a way that you cannot just throw an extra server and be done with it? If this is the case, it seems like you have a bottleneck preventing you to get more business.

• Do you have issues integrating your system with other systems?

Does your system need to integrate with web applications, with external APIs, or with some other solution and is this just a pain causing strategic integrations to be canceled or delayed?

• Does the speed at which development goes affect your business?

Does it take you quarters to develop features that your competitors can release in weeks or months? Is this putting you at a significant disadvantage?

• Does the cost of development affect your business?

Perhaps you can develop at the same pace of your competitors (or at a comparable pace), but you can do that by throwing way more money at the problem that they need to, and this is causing your costs to be way higher than theirs or just not leaving the margins you want.

• Do you have trouble keeping up with changes due to regulations?

Are you working in a field where changes in regulations have to be reflected in your software in a reasonable timeframe? This is the case for example for software for tax calculations, or software that needs to follow healthcare or financial regulations. If this is your situation, are you regularly facing challenges implementing the changes at the speed imposed by the regulators?

What is the response?I hope the previous points helped you to look at the situation from different angles. If after this check up you feel that it does not hurt in any specific place, then it is all good. I think you can go back to look into other problems because your legacy code base is not the problem you need to fix right away.

However, if you encounter some issues after all, now we need to figure out what you should do about it. We will see that in the next paragraph.

Ok, so what can I do about this?Well, I am an engineer, I can only answer “it depends”. The long answer is contained in a course on Legacy Modernization we created that is a few hours long, but here we want to look at this from a 10,000-feet-view.

1. You need to understand if you have a technical problem or an organizational problem

As much as we would like to think that we can solve all issues “simply” by acting on the technological level, the harsh truth is that this is not the case. Some of the problems we identified in the section above could be either due to organizational challenges or to technical challenges.

For example, development could be way too slow not because of the technologies you adopted but because you cannot converge on the right requirements in a reasonable time.

This is something to consider, because as much as we would be happy to help you with legacy modernization, if your issues are organizational, it would not solve your problems and you would not be a happy client.

1. Then you need to look at the alternatives

Assuming your problem is technological or it is also a technological problem, we can look into the specific problem to check which solution you can apply. In our studies we identified seven different techniques we can apply to solve problems with legacy code. Some of them are solutions to very particular situations and are rarely applied. In this article we want to keep things as simple as possible, so we will focus on the three that are most commonly used, as they cover well over 90% of the situations. In the section below we will look at them one by one.

We are ignoring one option that is often marketed as a “Legacy Modernization technique”. I am referring to the option that consists in keeping the code largely as it is, but spending some effort in documenting the old code base, refactoring it a little bit, maybe adopting a new IDE or buying some tool to help navigate the bowl of spaghetti code you have and be done with it. This technique is the “lipstick on a pig” technique.

The “lipstick on a big technique” has its role, and if your pain is not too big then it may work. However this is a rather obvious solution, and this is probably something you have already considered. So we only focus on the techniques that cut a little deeper, and take out the problem at its roots.

1. You need to make a decision

Here is when the ball is on your side and you need to pick an option. We will share some suggestions below.

The three most common approaches to Legacy ModernizationBelow we list the three ways in which you can go from a legacy codebase to one that you feel confident maintaining and evolving. They are going for a manual rewrite, or perform transpilations, either once and for all, or continuously. Let’s see what this means and what are the consequences.

Manual rewriteThis technique is rather obvious: you just pick your existing application and you rewrite it, from scratch. It is very interesting from a sociological perspective, how this is a very frequently chosen solution, while being the most costly and the most risky. But I guess we have an intrepid spirit, or we would have chosen a different career. What typically happens is that we look at a system that has been around for a long while and we:

i) vastly underestimate the effort that was put in creating the system and evolve it

ii) we think that we can do so incredibly better than the people who preceded us.

The result is that we try to recreate in 18 months what took 18 years to create. Often we just run out of money and time. I would suggest adopting this solution only if either the code base is relatively small (under 200,000 lines of code) or the requirements changed so dramatically that there is little value in the business logic trapped in the existing system.

One-time transpilation Here the idea is that you take a module at the time of your application, you run a transpiler on it and you translate the business logic to a different programming language, for example you translate RPG into Java. You could also apply the same approach to move to a different framework, for example from Struts to Spring.

After the transpilation you throw away the original code and start maintaining the newly generated code. This means that you need to re-train your development team, or hire new developers to maintain the portions of your systems that have already been moved to the new technology.

This is typically something attempted after a manual rewrite has previously failed. The challenge here is to produce code that is idiomatic and maintainable, and not all solutions are able to do that. This made some people skeptical about this approach.

Ongoing transpilationWhen going this route you keep developing using the old programming language, but you just insert a transpiler in your build pipeline, so that every time your developers write, let’s say, RPG, the code is translated behind the scenes in, let’s say, Java. Your developers only look at RPG code but you can actually deploy a Java application.

This solution is not that frequent, but it is useful when you are reasonably satisfied with the productivity of your team, you have many developers that are familiar with the old language, and your main problems are the deployment platform or the integration with other technologies.

Time to make a decision: what should we consider?Now that you have looked at the alternatives (at least the main ones), you have a big decision to make.

The typical decision is “let’s wait a little bit longer”. You have many things to worry about, and perhaps if you do not think about your problems with legacy code they will go away. Or perhaps next year there will be more time and resources to deal with it. We are only human, after all, and this is how we typically react.

One part of the problem is that there is just too much uncertainty when looking at such a big decision, and either one makes an enormous leap of faith and just picks one route, or they stand where they are. I think we should instead make exploratory studies to reduce that uncertainty, which means, in our analogy, reducing the gap you need to jump over.

So I would start first of all studying your codebase: examining how it works, examining how much dead code there is, studying how interrelated components are, identifying recurring patterns, looking at how they could be migrated. This is work that can be done for the low tens of thousands of dollars or euros and can give the clarity to make an informed decision. The way we tackle this is through what we call Audit and Analysis. You may perhaps take a look to see what kind of information can be obtained on your codebase and how they can help assessing the situation, make a decision, and then a plan.

SummaryAs someone who loves software development I resisted for a simple truth: Corporations are not successful because they have good software in good condition. These are successful despite having bad software in a bad state (thanks to Miroslav Diviš for expressing this so eloquently!).

However sometimes the bad software becomes such an hindrance that is canceling out the advantages the company has, and putting it at a disadvantage. If and when that time comes, it makes sense to examine the situation, understand where the problem lies (is it really a technological problem?), and what we can do about that. Then it is just a matter of looking at the options, assessing your situation and moving forward. This is something that can be done, and there are professionals able to help, so there is no need to ignore the problem and hope for the best.

The post Legacy Modernization: should it be part of your strategy? appeared first on Strumenta.

View Details

IntroductionThis is the first of two articles in which we’ll go through the design and implementation of a domain-specific language (DSL) for Minecraft mods. We’ll use the most advanced techniques that we have learned at Strumenta, and we’ll only use open-source tools and libraries.

In this first part, we’ll give an introduction to DSL design and will start planning for an implementation, but we won’t write any code yet. In the second part, we’ll produce a working prototype that we’ll use to build a simple Minecraft mod.

Why a DSL for Modding Minecraft?Our motivation for writing this article is simple: we want to advocate for DSLs and specifically for our approach to writing DSLs.

However, it’s hard to find good examples of domain-specific languages. Most real-world DSLs, with larger user bases, are kept as closely guarded industrial secrets. On the other hand, languages invented for articles or technology showcases do not make good examples. Let’s see why.

The Secret IngredientCompanies often keep their DSLs as industrial secrets because:

• those DSLs represent a competitive advantage;
• they are the result of significant investments resulting in years of research, development, and maintenance;
• perhaps most importantly, they’ve used their DSLs to express their business processes, or the business rules in their products and solutions; so, their code, and to some extent the languages themselves, reflect their organizational practices, their expertise, and their know-how.

So, it’s comprehensible that they don’t want to show any of that to the outside world. It would be like if they shared internal, classified documents as examples of good document writing.

Making Up ExamplesMade-up languages and examples, instead, have several issues, including:

• they are too simplistic; they work great as toy examples, but don’t capture the complexity of real-world problems;
• the “D” part of the DSL is lacking. They don’t make a good case for any relevant domain.

Usually, companies like Strumenta who develop DSLs (or software in general) have limited or no knowledge of the application domain, and collect the information they need from subject-matter experts (SMEs). When writing a DSL for an article or presentation, authors have no option but to skip the part where, in a real project, they would interview SMEs, learn, think, discuss ideas and prototypes, and iterate. As a result, their examples of DSLs:

• are too generic, thus not domain-specific at all (e.g. a language for doing math), or
• target a domain that’s software-related (e.g. software build scripts), because who wrote the article is a software SME, i.e. a developer or closely related role.

While there’s nothing wrong with languages that simplify software development and engineering, they’re not good examples of what we usually develop. Most users of our DSLs are not software developers; they are subject-matter experts. We don’t want to further spread the idea of DSLs as advanced tools for elite developers requiring esoteric knowledge – because they’re not.

MinecraftSo, by choosing Minecraft mods, we’ve identified a domain that:

• is only partially related to software;
• is known to many people, either directly or through their children or younger relatives.

Yes, we’re talking about modding a video game, i.e. modifying a piece of software, which sounds a lot like development. However, concepts in the language will relate to the fictional in-game world, and the details of how they are to be represented in software and installed in the game are not visible to the users of the DSL.

Indeed, one of the most appealing features of DSLs is that they can be read and written by non-developers, but they still can be interpreted and eventually executed by a computer. In practice, DSLs enable users to write executable logic in a form that doesn’t necessarily look like code, and may borrow existing notations from their area of expertise.

Plus, by choosing to target Minecraft, we can show pictures of the results of executing DSL code. Often, that’s not possible, because there isn’t any graphical output; or it’s not practical, because the results of the execution are themselves domain-specific and only meaningful for subject-matter experts.

Language DesignLanguage development doesn’t have to follow a rigid “waterfall” model (design everything first, then proceed to implement, test, and ship). Still, we typically start by assessing the requirements and by designing an initial sketch of the language together with a small group of SMEs and interested parties. That way, we can produce an early prototype that will then be the basis for further iterations.

Then, we’ll go back and forth between design and implementation, learning something new about the domain and how our abstractions actually work when applied to problems in the domain.

That said, for Minecraft in particular, we’ll have to play two roles at once: language architect and SME. So, let’s proceed with interviewing ourselves.

Who are the intended users of the language? We aim at kids and adults with potentially little or no programming experience. So, similarity with existing programming languages or notations (e.g. JSON) is not our goal.

What is Minecraft in a nutshell? It’s a “sandbox” type game where the procedurally-generated 3D world is entirely made of blocks. A block is a cube made of some material; the properties of the block depend on the material. The player explores the world in first person and can destroy most of the blocks to obtain items in their inventory. Some represent blocks that the player can place in the game world, while others represent tools that the player can use to interact with the world and, for example, speed up the breakup of blocks (e.g. with a pickaxe). By combining some items in a certain pattern, the user can craft other items.

What should the language enable users to do? A set of common modding tasks (such as adding new block types) that would otherwise require writing a Java library. Testing is out of scope; users will test their mods by running them in the game.

What are the main concepts in the language? I.e. what are “programs” or recipes made of, for the most part? For example, in a OO programming language like Java, concepts would include classes, methods, variables, …

In our case, the Minecraft game world is made of a number of entities that are natural candidates for concepts in the language: blocks, items, tools, mobs (friendly creatures and monsters), etc. We’ll see later that the modding SDK comes pre-filled with a list of those entities that we may add or tweak. However, given the limited scope of this tutorial, we’ll restrict the bounds of our domain and only deal with blocks.

What’s the execution model? That is, how do we turn recipes in the language into behavior, i.e. running code? Common options are code generation targeting an existing platform, and writing an interpreter. In our case, the answer is simplified because we’re targeting Minecraft, so the game will be our execution engine – we don’t even have to develop and maintain it ourselves! To be able to load code into Minecraft, we’ll have to generate Java code and compile it into a mod. Since our users are not developers, we’ll hide this complexity from them; that is, we won’t require that users build the extension by themselves, or even see the extension project at all.

What additional tool(s) are needed? Since this is a tutorial article, we’ll only sketch the language (parser and code generator). We won’t develop any additional tools like an editor, IDE plugins, linter, etc. In real-world scenarios, we usually provide at least a limited editor with syntax highlighting and error reporting. Indeed, we want to ensure that users actually adopt the language and that their productivity in the long run doesn’t suffer – in fact, it should rather improve.

Starting from the Bottom: the RuntimeAs often happens, we can improve our design if we include concerns and insights “from the bottom up”; in other words, learning from what we’ll eventually generate. Since we have the possibility to look at the desired result of compiling and running a recipe – that is, a working Minecraft mod, as a software project, and as behavior in a running instance of the game – we can use that information to learn more about the application domain, and potentially to inform our language design.

That’s also what happens when the DSL is a new “skin” or layer over an existing API, as a means to simplify the use of a certain library or tool, or to make it available to non-developers. Sometimes, confusingly, if these existing APIs are designed in a so-called “fluent” style, they get called an “internal DSL”. We prefer to avoid this confusion and only use the term DSL for an actual language with a parser. Still, this confusion is a symptom of APIs and DSLs sharing some design concerns.

In these scenarios, studying the existing runtime can help us design a better DSL – but we also have to be careful not to draw existing cruft into our design. “Cruft” includes poor abstractions introduced because of a lack of knowledge about the domain, or due to a hastily produced implementation, or limitations of the programming language or framework used to implement the existing APIs.

Anyway, it’s not always the case that we target an existing library or framework; often, no runtime exists, and we have to design one alongside the language if we want to make it executable. In any case, expertise and experience, as well as a thorough design process, are always necessary to design a good DSL, not only when no existing reference implementation or runtime exists.

Note that here we’re taking it for granted that we’ll be generating code from recipes in our language. Again, that derives from the existing tooling to author Minecraft mods, as we’ll see in the next section. However, in other cases, it could make more sense to develop an interpreter instead of, or in addition to, a code generator. An interpreter makes it possible (or easier and quicker) to run code on the fly, thus it’s a great choice for simulators, test harnesses, and all situations when we need to rapidly and repeatedly iterate between authoring and validation of behavior.

In our case, an interpreter cannot exist outside of the game engine, as we need the game to run our mods. Integrating an interpreter into the game is a possibility in theory, but it probably wouldn’t make much sense – we’d still have to reboot the game to update our mod, given how the game works, and that would eliminate the development speed advantage that an interpreter would provide, while retaining all the complexity and resource utilization of an interpreter. Anyway, we’ve ruled it out because of the time and space constraints of this tutorial.

Modding BasicsWith that in mind, let’s look at how one writes Minecraft mods today. We’ll follow the Getting Started with Forge tutorial. We need a Java installation for the following steps to work, but if we have Minecraft Java Edition installed, we’re already set up.

The first step is to install Minecraft Forge so that it patches the base game with mod-loading powers. To do that, we’ll download the “Installer” from https://files.minecraftforge.net/net/minecraftforge/forge/ and run it. We’ll use this to test the mod in a running instance of the game. The users of our mod will have to do the same to play with it. If the installation is successful, the next time we log into the game, we ought to see something like the following:

Note the “forge” entry in the dropdown menu.

Then, we’ll download and unzip the Forge Mod Developer Kit (MDK). That is the smaller “MDK” link on the page, not the “Installer” link we followed previously. Once we’ve downloaded the MDK, we can expand it to a directory of our choice. The “mod developer kit” is just a Java project built with Gradle; modders are supposed to modify it to introduce the features they want. As we’ll see, the project comes with sample code to add a new type of block to the game – this is what we’ll bring into our DSL.

Building Our ModHowever, before looking at how our new mod is structured, let’s first familiarize ourselves with building it. Building is the process through which we go from human-readable source code to machine-readable object code, packaged in a way that the target platform can load. In our case, Minecraft is a Java application, so building our module will result in a JAR (Java ARchive) containing byte-compiled Java classes and other resource files.

We can build the project with ./gradlew build on OSX and Linux, or just gradlew build on Windows. gradlew is a shell script that will download the appropriate version of Gradle if it’s not available, and use it to build the project according to the contents of the build.gradle file. This file comes with the MDK and isn’t meant to be edited by the mod developer.

After the build completes successfully, we’ll find a jar file (Java Archive) in the build/libs directory, such as examplemod-1.0.0.jar. This is the file that we’ll have to install into the Minecraft mod directory to make it available in the game. For example, on my OSX machine, Minecraft loads mods from ~/Library/Application Support/minecraft/mods. If you don’t know where this directory is located on your computer, read on.

If we’ve built and installed our mod correctly, we’ll see it in the game when we play the “Forge” version and click on the “mods” button:

Incidentally, here we can also click on “Open mods folder” to reveal where mods are stored on our machine – so if you don’t know where to copy your jar, you can boot the game and ask it that. Note, however, that we have to quit and restart the game every time we want to install a new version of our mod – Minecraft won’t pick it up while it’s running.

Now, let’s concentrate on the initial steps we have to take to configure and build our mod. These will inform the design of our first language construct.

Changing Basic Mod PropertiesOne of the first things we’ll want to do is change the mod ID and version. These are the coordinates that identify our mod. They are defined in the gradle.properties file under the root of the project.

This file contains key-value pairs that the Gradle build tool may interpolate into its build configuration. In particular, we’re interested in the mod_id and mod_version properties. Luckily this file is heavily commented so we can learn about all the configurable properties without leaving our development environment.

So, let’s change the mod_id property from “examplemod” to, say, “my_wonderful_mod”:

## Mod Properties

# The unique mod identifier for the mod. Must be lowercase in English locale. Must fit the regex [a-z][a-z0-9_]{1,63}

# Must match the String constant located in the main mod class annotated with @Mod.

mod_id=my_wonderful_mod

# The human-readable display name for the mod.

mod_name=My Wonderful Mod

# The license of the mod. Review your options at https://choosealicense.com/. All Rights Reserved is the default.

mod_license=All Rights Reserved

# The mod version. See https://semver.org/

mod_version=1.0.0

Here, we’ve also changed the “mod_name” that is shown as a nice description in the “mods” screen in the game.

Let’s also keep in mind that the mod ID is also referenced in the mod’s Java code. Let’s open the file src/main/java/com/example/examplemod/ExampleMod.java; we can see the mod id at line 36:

public static final String MODID = “examplemod”;

Let’s change it to match the value we’ve set in the gradle.properties file. If we’ve done everything correctly, after we’ve run “gradlew build”, and copied the jar file into Minecraft Forge’s mods directory, we’ll see the updated mod in our copy of the game:

The first feature of our language will be to realize the same outcome we’ve produced here. That is, generating an “empty” mod with a given name, version, and description, that we’ll be able to install into the game by dropping it into the mods directory.

That’s All for NowWith that we conclude the first part of our two-article tutorial. We haven’t written any code yet; instead, we’ve explored our reasons for designing our Minecraft mod language, and we’ve started looking at the concepts that will make up our language, as well as the desired output of our code generator.

Follow us for Part 2 of this tutorial!

The post DSL for Modding Minecraft appeared first on Strumenta.

View Details

This is a comprehensive guide to Software Language Engineering. Software Language Engineering is the discipline focused on the science, strategies, patterns, and tools behind the creation and processing of languages. It can teach you how to create better languages and developer tools. It uses science but also insights about people coding, all to create better tools for better work.

If you want to learn more about it, you are in the right place: this article will not teach you everything, but it can be the beginning of an interesting journey.

This is going to be an introductory article on the subject. We are going to explain what language engineering is, present the main topics, and list a few representative articles on common issues and discussions related to the topic.

How to Begin Working on Language EngineeringLanguage Engineering is a fascinating field because every programmer is affected by the language they use. It is natural to get curious about the tools you use every day and see if you can improve something. In fact, this is how many of us started working on such issues. For example, you might start developing an extension for an editor or adding configuration to support syntax highlighting, and then you move on from there.

It is easy to start working in the field, you do not need to begin working on a thesis on Language Engineering for your Ph.D, like Polyglot Software Development and the benefits and drawbacks of using multiple languages for a specific project. I mean, you can, but it is not required.

My point is that the field is vast and you can find something interesting to learn at any level of expertise or formal training. This is a field in which academic experts do create actual products used for work, like ANTLR, which was initially created by a computer science professor. Your formal training can be beneficial if you have it; however, the lack of it does not pose a barrier.

Examples of Language Engineering IssuesThe field itself spans from very technical topics to strategic issues.

An example of technical arguments could be developing patterns for transpiling goto in languages without such a feature. You might say that sounds complicated: I do not even know what a transpiler is! Well, it is not that complicated, once you understand the basic ideas.

Just like an interpreter or a compiler, a transpiler transforms the code to make it executable. The difference is that it does not transform code into a machine-readable format. It transforms the code into another language. So, a transpiler might transform Typescript in JavaScript. That is why it is also known as a source-to-source compiler.. That is why it is also known as a source-to-source compiler.

Now you can see that developing patterns for transpiling goto in languages without such a feature, means understanding how you can transform some piece of code that uses goto statements into one without it. Or more plainly, how to get rid of gotos. We have just started and you have already learnt something!

An example of a more high-level issue is understanding why the developers of C#, and the LINQ technology, decided to use a syntax that starts with the source instead of the selected elements.

LINQ, or Language Integrated Query, is a .NET feature developed to provide query capabilities to languages such as C#. You can use it to get data from various sources like databases or lists. It is inspired by SQL, but if you have ever used it, you will have noticed that the LINQ queries start with the source (i.e., FROM) rather than the elements selected (i.e., SELECT). This is the opposite of what happens in SQL and feels somewhat unnatural.

The reason they picked this syntax is because this way autocomplete can be supported. In fact, if you start with the source, you can then offer suggestions when selecting elements (WHERE clause). So, they made this choice because IDEs are now a crucial tool for developers and a language can benefit when it is designed with tools in mind.

TopicsAs with any discipline as large as Language Engineering, there are many ways to organize it and topics to include. Here we provide our own view, but we are very welcoming of suggestions and improvements: feel free to write to us about any that you might have.

We are going to present the main topics and list a few representative articles on common issues and discussions related to the topic. The list of articles will be by no means exhaustive, just a place to start your exploration of the common ideas you will encounter.

• Language design. How to design languages to be effective. This topic includes everything from strategies for creating languages that are easy to learn to discussions about tools used in designing languages. A few notable articles:
  • How would I go about creating a programming language?, a handy overview to guide you in creating a programming language
  • 68 Resources To Help You To Create Programming Languages, resources presenting everything from designing to actually building your language
  • Teaching Programming With Hedy, a presentation of Hedy, a new gradual language and approach to make programming languages easier to learn
  • Racket a Language for Creating New Languages, a presentation of Racket, a language and system to create new languages
  • Building a language: tool support, a video about the issue of tool support for languages
• Model driven development. Creating software working on domain models
  • Interview with Matteo Mortari on process automation, interview with a Software Engineer at Red Hat, working on Drools, the rule engine, and DMN, Decision Model Notation
  • Telosys: a Code Generation Tool by Laurent Guerin, interview with the author of Telosys, a code generation tool
• Domain specific languages. Domain Specific Languages (DSL) are programming languages tailored for a specific purpose and audience. They require thoughtful design and specific tools to be used at their best
  • The complete guide to (external) Domain Specific Languages, a great overview of DSLs, what they are, and what they are good for
  • Designing a DSL for accounting: use a DSL to describe taxes, pension contributions, and general financial calculations, a tutorial on designing a DSL, in this example one for accounting
  • Are You Abusing Excel? You Need Something Different, entire companies rely on Excel, a fantastic software when it is used correctly, but that sometimes should be replaced with a DSL
  • When you need low-code or no-code and when you need DSLs, a pragmatic look at how to choose between low-code and DSLs
  • Experiences of Practical DSLs usages: a talk with Glen Braun, an interview about a real experience in adopting a DSL
• Application modernization. Even the best-designed software gets old and becomes a liability. You need to understand how to modernize the code, keep the value the old software provides, and get rid of the old approach.
  • Interview with Graham Cunningham on legacy modernization, an interview with an expert on legacy modernization, is a great way to start understanding the topic
  • Why you should not use (f)lex, yacc and bison, a discussion about why you should not use famous but outdated parsing tools; complete with their history
  • Comparing the cost of migrating to rewriting, an article about some common approaches to legacy modernization
• Parsing. Parsing is about extracting information from some text written in a meaningful format.
  • The ANTLR Mega Tutorial, a comprehensive tutorial about ANTLR, the most used parsing generator
  • EBNF: How to Describe the Grammar of a Language, parsers are defined using a grammar, this article explains EBNF, the most used format to describe a grammar
  • A Guide to Parsing: Algorithms and Terminology, a comprehensive guide about the basic theory of parsing, from the terminology to an overview of the common algorithms
  • Parsing HTML: A Guide to Select the Right Library, an overview of the most commonly used ways to parse HTML
  • Parsing SQL, a list of the libraries, tools, and approaches to parsing SQL
  • Building advanced parsers using Kolasu, our approach to building professional parsers
  • Challenges in Parsing Legacy Languages: The Case of SAS Macros, a discussion of a real-life issue in parsing legacy languages and how to deal with such problems
• Compilers, Interpreters and Transpilers. Once you have parsed some code, you have essentially a series of instructions. You then need to execute them in some way, this is when you need to build a compiler, interpreter, or transpiler
  • A tutorial on how to write a compiler using LLVM, LLVM is a technology that greatly simplifies creating professional compilers, this is a tutorial to get you started on using it
  • Language2Language Transformers: machine learning to build transpilers, a novel way to build a transpiler using machine learning and lot of examples
  • How to write a transpiler, an introduction to transpiler and a tutorial on how to build one
• Code processing. Extracting data from code, analyzing it and programmatically transforming it. In other words, we talk about static analysis, automated refactoring, and code generation.
  • How and Why to Analyze, Generate and Transform Java Code Using Spoon, a tutorial and overview of Spool, a tool to analyze, generate, and transform Java code.
  • Convert PL/SQL code to Java, some code processing is so much requested that there ready-to-use tool for the job. This is a discussion of two tools to convert PL/SQL code to Java
  • Generate diagrams from C# source code using Roslyn, if you can understand code you can transform it into a lot of things, in this article we look at the example of generating diagrams from C# code
  • Getting started with JavaParser: analyzing Java Code programmatically, a tutorial on JavaParser, a parser for Java, and how to perform automatic operations on the code
• Editors. Programming languages are professional tools that require specific tools to be most productive. There are many aspects to understanding how to develop an editor, from design to pragmatic considerations.
  • Code Completion with ANTLR4-c3, code completion, or autocomplete is a great productivity boost for developers. In this article, we see how to implement it based on an ANTLR parser
  • Go To Definition in the Language Server Protocol, the Language Server Protocol is a protocol that revolutionized the way editors work, by providing a standard way to make language tools communicate with editors. In this article, we discuss how to implement the Go To definition for a language
  • Writing a browser based editor using Monaco and ANTLR, Monaco is the web editor component of Visual Studio Code, the most widely used code editor. In this article, we discuss how to use it to create your own custom web editor
• Language Workbenches. Language workbenches are IDEs designed specifically to build languages. They support you in all steps, from language definition to creating supporting tools like compilers and editors.
  • A tutorial on Spoofax, a Language Workbench, a tutorial on a platform environment that permits quickly creating DSLs
  • MPSServer: enabling integration with MPS, MPSServer is a tool to make MPS accessible remotely. It can be used to build web editors but also create diagrams, or integrate a build mechanism
  • Saving JetBrains MPS models in a database using Modelix, JetBrains MPS is an integrated environment, which makes it hard to integrate with external software. In this article, we discuss how to deal with the specific issue of saving MPS models in a database, which would allow sharing data between users
• Community. As with any other professional field, participating in the community is the best way to keep you updated and learn more.
  • Strumenta Community. The best and probably only community about language engineering. So, you really have no choice but to join us. We welcome everybody from novices to experts.
  • LangDev. An informal and annual meeting of language engineering enthusiasts from both industry and academia. We come together to discuss the state-of-the-art and state-of-the-practice of language engineering.
  • SplashCon. The ACM SIGPLAN conference on Systems, Programming, Languages, and Applications: Software for Humanity embraces all aspects of software construction and delivery, to make it the premier conference on the applications of programming languages – at the intersection of programming languages and software engineering.
  • SIGPLAN, The ACM Special Interest Group on Programming Languages, organizes a lot of conferences in addition to SplashCon.
  • Interviews. We have a nice list of interviews with people working in the field that you might be interested in. They have both videos and transcripts, so you can watch or read them.

ToolsA brief list of the main tools we use in Language Engineering. A little note beforehand: we found that people who are learning a new subject want a clear answer to the question: what to use? They lack the knowledge to discriminate between many options. Instead, people who are trying to deepen their knowledge want to know all that is available.

Since this is a general article, we believe both kinds of people will read the article. So, we are providing a first option that is good enough for everybody and then alternatives that are best for specific cases or personal tastes.

ANTLRANTLR is a parser generator and the main tool we use in our everyday job. A parser generator increases our productivity and allows us and our clients to build and maintain a parser. ANTLR supports many different languages, from Java to JavaScript and it is therefore our primary choice.

ANTLR is widely used, so there are many libraries and tools built for and upon it. Our favorite tool is the VS Code extension for ANTLR4 grammars. VS Code is also widely popular and supports many languages, so this leads to a reliable and productive setup. We also like the official ANTLR plugin for IntelliJ, but of course, this works only for Java and Kotlin projects given the focus of the IntelliJ IDE. You can find a list of plugins for several editors on the ANTLR website.

We are so confident about using ANTLR that we also built a Kotlin target for ANTLR.

We created and keep supporting a set of Starlasu libraries compatible with ANTLR. They are designed to create AST and are ideal for transforming an ANTLR parse tree in AST tailored for your application.

Alternative Parsing ToolsThere are many parsing tools available out there, from parsing generators, like ANTLR, to parsing libraries, like Chevrotain. We know it well because we research extensively on the subject, gathering lists of Parsing tools and libraries for:

• Java
• Python
• JavaScript
• C#

There is really an embarrassment of riches of parsing support available in every language.

So, if you are interested mainly in one language or one type of project you might want something else that best fits your needs. We mainly work on ANTLR because it has great flexibility and productivity.

JetBrains MPSIf you need to create a DSL that will be used on a desktop PC, JetBrains MPS is the first tool you should look at. It is a Language Workbench, a tool designed to create languages. JetBrains MPS is the most popular language workbench available, for a few good reasons:

• It is supported by JetBrains, the well-respected developers of programming tools
• It permits to prototype a DSL very quickly
• It supports multiple notations: textual, tabular, graphical, and more.
• It has all the necessary features to build advanced languages and editors: type system, constraints, etc., in this way, we can build editors that support users effectively
• It is a rich environment that permits to build great integrated tooling: interpreters, simulators, debuggers, documentation generators, etc. In our experience, the tools built around DSLs make a difference to increase productivity
• It allows you to evolve languages without breaking existing code. This is very important because DSLs are always designed and evolved in an incremental way

The main drawback of JetBrains MPS is that it is a desktop application designed for developers. It has a UI that looks complex and daunting to non-developers and it is not easily integrated into command-line workflows or the web. We are actually working with the JetBrains MPS community to support a web use of MPS, but the work is still in progress.

The other significant drawback is that it is a standalone system. Your users will need to run JetBrains MPS in some way or another to use any language you are going to define. It is opensource, but is quite complex and designed to work as-is. It is not easy to integrate with other code.

Alternative Language WorkBenchesAn alternative Language Workbench is Spoofax, you can read it more in our Spoofax tutorial. It is widely used and adopted in industrial applications. Its main drawback might be that it is based upon Eclipse, which is a platform with a lot of admirers, but it is less polished than an IDE from JetBrains or Visual Studio.

Xtext is also a popular Language Workbench based on Eclipse and EMF, the worst way to describe it is a set of plugins for Eclipse that transform it into an IDE to create languages. The best way is from their website:

Xtext is a framework for development of programming languages and domain-specific languages. With Xtext you define your language using a powerful grammar language. As a result, you get a full infrastructure, including parser, linker, typechecker, compiler as well as editing support for Eclipse, any editor that supports the Language Server Protocol and your favorite web browser

Essentially Xtext is a complete tool for creating programming languages, based on open-source software. The neat advantage of Xtext is that it is an open system, you can create a language with Xtext and then integrate it with the rest of your codebase, as you wish.

A tool similar in spirit to Xtext is Langium. They are both built upon open-source libraries and tools: Xtext is built upon Eclipse and ANTLR, while Langium is built upon Visual Code and Chevrotain. The difference is that Langium aids you in creating quicker individual, simple languages and that is all in TypeScript to be easily integrated with Visual Code. This makes it a great choice for web or cross-platform projects.

A different option is MetaEdit+, it is a commercial language workbench, so it is designed for companies. The interesting thing is that it was created to provide the tooling to customers who already know the domain to formalize it into a language. So, it is a tool to make it easier for domain experts to perform the work of a language engineer. In particular, it caters to companies working on well-defined products.

It is a nice example of what language engineering can do for people that are not language engineers. We have interviewed Juha-Pekka Tolvanen, one of the people behind MetaEdit+, so you may watch that interview to learn more about MetaEdit+.

If you are interested in starting slow with language workbenches, you might want to start with textX. It is essentially a suite of Python modules that come together to create a simple but functional language workbench to create languages from Python. It can only be used from Python, but it is easy to start. You can read our tutorial: Quick Domain-Specific Languages in Python with textX.

SummaryWe have seen how to get into the fascinating field of Language Engineering: how to start, a map of the main topics, and the tools you can use. We hoped to have succeeded in transforming an abstract topic into a series of understandable topics that can help you navigate the field.

The post A Comprehensive Guide to Software Language Engineering appeared first on Strumenta.

View Details

IntroductionIn this article, we will learn how to implement parsers in Python using Pylasu and ANTLR. In order, we will:

• Create an ANTLR grammar for a simple ‘kinda-functional’ programming language called Slang and generate a parser from it;
• Define an Abstract Syntax Tree (AST) using Pylasu and learn how to build these from ANTLR parse trees;
• Integrate our parser with a Command-Line Interface (CLI) application, allowing users to parse Slang code from both strings and files and visualize a JSON representation of the corresponding AST;

In order to keep the article from being too verbose, some code details might be omitted. Check out the complete project source code on GitHub. Feel free to fork the project, play with it and share your improvements, comments or ideas with us!

Parsers and Abstract Syntax TreesMaybe as part of our latest attempt to create a shiny-looking innovative programming language or data format, or maybe to dust off an existing one – teaching computers to understand languages is a fun drive!

In this, the absolute first step consists in transforming text into something that can be easily manipulated by computers. At first glance, venturing into the dark world of regular expressions might seem like the right path. After all, we are talking about text, aren’t we?

Unfortunately, regular expressions might not provide enough expressive power to process the syntax of language constructs. Most of the existing ones, indeed, require Context-Free Grammars (CFGs) – thanks Prof. Chomsky. Also, manually implemented regular expressions might easily become cryptic and hard to maintain in this context.

In practice, CFGs are used to describe the syntax of most computer languages in terms of rules and dedicated software components are implemented to recognize syntactic constructs in conformance with such grammars. Put simply, we use Parsers.

The typical output of a parser consists in a tree-like data structure called parse tree. In this, the root and the intermediate nodes represent non-terminal rule matches, while leave nodes correspond to terminals. For example, here is a possible parse tree representation (right-hand side) for a simple series of Javascript statements (left-hand side).

Numerous parsing algorithms exist in the literature, providing different trade-offs between expressive power and computational complexity.

Parsing algorithms can be implemented manually, but a definitely more convenient approach consists in relying on parser generators. These take a grammar specification and produce a conforming parser implementation.

Among parser generators, we love ANTLR at Strumenta – we use it in all sorts of parsing projects on a daily basis and it always worked well for us!

The tree-like nature of parse trees entails ease of navigation and manipulation of the input. However, at some point, the purely syntactic nodes included in it might become irrelevant. For example, parentheses can be useful to understand the order of the operations in text and determine how the parse tree is organized. Once the parse tree is created these do not provide useful information anymore and can be removed. This is different from what happens with identifiers or literals, which remain valuable as they provide information through their content.

For this reason, additional transformation steps are implemented into parsers to extract what’s called Abstract Syntax Tree (AST). Going back to our Javascript example, this is a possible abstract syntax tree representation where purely syntactic keywords and punctuation has been removed.

The Slang LanguageSlang is a toy programming language showcasing some typical language constructs found in both imperative and functional languages. As such, feel free to interpret each construct with your preferred semantics, the following discussion will rather only focus on the syntactic aspects of the language.

Example – Fibonacci using SlangThe function computing the Nth number of the Fibonacci series seems to be a quite popular example to give a taste of the syntax of a programming language. We like the Fibonacci series, so here is a possible outstanding implementation using Slang

function fibonacci(n) { if (n <= 1) return n; else return fibonacci(n - 1) + fibonacci(n - 2);} In the following paragraphs, we introduce the various language constructs for Slang. For each construct, we provide the corresponding ANTLR parser grammar rule. The complete ANTLR parser and lexer grammar specifications can be found in SlangParser.g4and SlangLexer.g4files on GitHub, respectively.

WorkspacesEach file contains a single workspace definition in Slang. This acts as a container construct for zero or more function definitions and standalone statements.

workspace: functions+=function* statements+=statement*; FunctionsFunction definitions consist of a name, zero or more parameters and zero or more statements. Parameters are identified by their name and do not include type constraints or specifications.

function: FUNCTION name=NAME LPAREN (parameters+=NAME (parameters+=NAME)*)? RPAREN LBRACE statements+=statement* RBRACE; StatementsSlang provides support for the following kinds of statement:

• Return Statement – can be used to return back some value specified through an expression from a function;
• Print Statement – can be used to display the representation of a value somewhere, e.g. standard output;
• Conditional (aka If-Then-Else) Statement – can be used to execute different logic depending on some condition;
• Binding Statement – can be used to associate names to values;
• Expression Statement – can be used to wrap expressions as statements;

Each statement, except for conditionals having multiple statements in their body, ends with a semicolon. Curly brackets are used to wrap multiple statements in conditionals and can be omitted otherwise.

statement : RETURN value=expression COLON #returnStatement | PRINT argument=expression COLON #printStatement | IF LPAREN condition=expression RPAREN (LBRACE positive+=statement* RBRACE |positive+=statement) (ELSE (LBRACE negative\_branch+=statement* RBRACE |negative\_branch+=statement))? #conditionalStatement | name=NAME BND value=expression COLON#bindingStatement | expression COLON#expressionStatement ; ExpressionsSlang provides support for the following kinds of expressions:

• Grouping Expression – can be used to override the ordinary operator evaluation order using parentheses;
• Unary Operation Expression – can be used to apply both logical and arithmetic negation, as well as arithmetic identity;
• Binary Operation Expression – can be used to perform arithmetic calculations and comparisons;
• Invocation Expression – can be used to invoke functions with expression arguments;
• Reference Expression – can be used to reference the value of parameters and variables;
• Literal Expression – can be used to define constant integer values;

expression : LPAREN expression RPAREN #groupingExpression | operator=(NOT|ADD|SUB) operand=expression #unaryOperationExpression | left=expression operator=(MUL|DIV) right=expression #binaryOperationExpression | left=expression operator=(ADD|SUB) right=expression #binaryOperationExpression | left=expression operator=(LT|GT|LTQ|GTQ) right=expression #binaryOperationExpression | left=expression operator=(EQ|NQ) right=expression #binaryOperationExpression | target=NAME LPAREN (arguments+=expression (COMMA arguments+=expression)*)? RPAREN#invocationExpression | target=NAME #referenceExpression | value=NUMBER #literalExpression ; Generating Parsers with ANTLRIn order to generate a parser from a grammar specification using ANTLR, there are first a couple of tools that need to be installed. Generally, ANTLR can be used by downloading and running its JAR – hence requiring Java to be installed on our machine. However, there is a simpler solution for us – antlr4-tools.

Using this, the only requirement is reduced to Python, which typically comes installed on all operating systems. The tool will create two executables to generate and invoke parsers – respectively antlr4 and antlr4-parse. The first time any of these commands will be invoked, the installation of both Java and ANTLR will be silently handled.

Checkout the antlr4-tools GitHub repository for detailed installation instructions.

Let’s get back to our example! We are using the PDM package and dependency manager, hence the following command is executed to install antlr4-tools:

pdm add -dG antlr antlr4-tools Given our SlangParser.g4 and SlangLexer.g4 grammar files, we can then generate an ANTLRv4.9.3 parser targeting Python3 with the following command:

antlr4 -v 4.9.3 -Dlanguage=Python3 -no-visitor -no-listener SlangLexer.g4 SlangParser.g4 The process will generate our parser in SlangParser.py, while the SlangLexer.py file will contain the lexer. The -no-visitor and -no-listener options specify that we are not interested in generating ANTLR4 base visitors and listeners in this case.

Check out our ANTLR Mega Tutorial for further details and start your path towards becoming an ANTLR ninja!

In order to rapidly test out the generated parser, we can use the antlr4-parse command as follows:

```

parse and print out the tokensantlr4-parse SlangParser.g4 SlangLexer.g4 workspace -tokens# parse and print out the parse treeantlr4-parse SlangParser.g4 SlangLexer.g4 workspace -tree

``` Otherwise, we could also install the ANTLR Python runtime and programmatically invoke the parser as follows:

```

Install antlr4-python-runtime with: pdm add antlr4-python-runtimefrom antlr4 import CommonTokenStream, InputStreamfrom slang.parser.antlr.SlangLexer import SlangLexerfrom slang.parser.antlr.SlangParser import SlangLexer# the code that we want to parsecode = "...slang code..."# tokenize the input using the lexerlexer = SlangLexer(InputStream(code)) # or FileStream(absolute_path) for files# parse the token stream using the parserparser = SlangParser(CommonTokenStream(lexer))# retrieve the parse tree rootparse_tree = parser.workspace()# print out a formatted representation of the parse treeprint (parse_tree.toStringTree())

``` Printing out the parse tree for our fibonacci function will produce the following output – don’t worry, we will make it more readable soon:

(workspace:1 (function:1 function fibonacci ( n ) { (statement:3 if ( (expression:5 (expression:8 n) <= (expression:9 1)) ) (statement:1 return (expression:8 n) ;) else (statement:1 return (expression:4 (expression:7 fibonacci ( (expression:4 (expression:8 n) - (expression:9 1)) )) + (expression:7 fibonacci ( (expression:4 (expression:8 n) - (expression:9 2)) ))) ;)) })) Defining ASTs with PylasuUntil now, we defined an ANTLR grammar specification for our language, generated a parser from it and saw how to print out the parse tree for a given file or string. Let us now introduce an abstract syntax tree using Pylasu.

Pylasu is an open-source library supporting the StarLasu methodology in Python. Other equivalent libraries exist to enable support in Kotlin (and the JVM), Typescript and C#.

Abstract syntax tree nodes can be defined through classes extending the Node base class in Pylasu. Nodes come with various built-in features supporting the implementation of recurring tasks when implementing language processing applications:

• Position – each node keeps track of its corresponding position in the source text in terms of line and column number;
• Traversal API – various functions can be used to navigate nodes, their children and properties;
• Transformation API – dedicated classes and functions can be used to transform nodes into other nodes or text;
• Origin and Destination – when involved into a series of transformations, each node keeps track of its origin and destination, that is the nodes where it originates from and those produced from it;

In addition to these core features, StarLasu libraries provide support for advanced concepts, such as symbol resolution, type computation, serialization, testing. Check out the documentation for further details!

Going back to our language, the following snippet shows how the node representing function definitions can be defined using a Python dataclass:

from dataclasses import dataclass, fieldfrom typing import List@dataclassclass Function(Node): name: str = field(default\_factory=str) parameters: List[str] = field(default\_factory=list) statements: List['Statement'] = field(default\_factory=list) That’s it, really! Extending the Nodebase class will integrate our node specification into the Pylasu infrastructure. Therefore, we will be able to navigate through its children, access its corresponding position in the source code (if coming from a textual source) and so on.

Node classes can also contain no properties or extend each other. For example, we decided to define a dedicated AST node for each unary operation. In the parse tree, instead, these are all represented using a single UnaryOperationExpression node. The following snippet illustrates how unary operation expression nodes can be modeled for our purpose:

from dataclasses import dataclass, fieldfrom pylasu.model import Nodefrom typing import Optional@dataclassclass Expression(Node): '''AST node representing an expression''' pass@dataclassclass UnaryOperation(Expression): '''AST node representing a unary operation expression''' operand: Optional[Expression] = field(default=None)@dataclass class Not(UnaryOperation): '''AST node representing logical negation operations''' pass@dataclassclass Minus(UnaryOperation): '''AST node representing arithmetic negation operations''' pass@dataclassclass Plus(UnaryOperation): '''AST node representing arithmetic identity operation''' pass The rest of the AST node definitions is omitted for brevity, you can find it inside the nodes.py module.

From ANTLR Parse Trees to Pylasu ASTsWe are now ready to transform our Parse Tree into an AST.

In Pylasu, transformations can be implemented by creating and configuring ASTTransformer instances. In the specific case of a transformation mapping ANTLR Parse Trees to ASTs, the dedicated ParseTreeToASTTransformerclass can be used. A transformer instance can be created as follows:

from pylasu.mapping.parse\_tree\_to\_ast\_transformer import ( ParseTreeToASTTransformer)transformer=ParseTreeToASTTransformer(allow\_generic\_node=False, issues=[]) The idea behind transformers consists in supporting a mostly declarative specification of transformation rules using NodeFactories, while taking care of all details concerning the position of nodes – that is, the line and column number at which these appear in text.

There are various ways a NodeFactory can be defined, the simplest one consists in specifying the source and target node types, along with their children – used to guide the transformation. For example, mapping a WorkspaceContext parse tree node into our Workspace AST node, the following factory can be registered:

from pylasu.transformation import PropertyReffrom slang.ast.nodes import Workspacefrom slang.parser.antlr.SlangParser import SlangParser as \_# workspace (constructor-node-factory-with-child)( transformer.register\_node\_factory(\_.WorkspaceContext, Workspace) .with\_child(PropertyRef('functions'), PropertyRef('functions')) .with\_child(PropertyRef('statements'), PropertyRef('statements'))) In short, we are instructing the transformer to create a Workspace instance whenever a WorkspaceContext is encountered in the parse tree, and populate the functions and statements properties by mapping each element with the appropriate factory – the rules mapping FunctionContext with our Function node, for example. PropertyRefis a Pylasu class used to represent references to node properties.

Sometimes, however, we might need a bit more freedom. Passing a reference to the class constructor – e.g. Workspace – requires this being a parameter-less one. Other classes could have mandatory constructor parameters, therefore, we also provide support for more complex kinds of factories – any Callable producing the right AST node instance.

Let’s use lambdas, for example, to define our node factory mapping BindingContext parse tree nodes into Binding AST nodes, as follows:

from pylasu.transformation import PropertyReffrom slang.ast.nodes import Bindingfrom slang.parser.antlr.SlangParser import SlangParser as \_# statement - bindingtransformer.register\_node\_factory( \_.BindingStatementContext, lambda source: Binding(name=source.name.text) # handle 'name' property).with\_child(PropertyRef('value'), PropertyRef('value')) As you can see, we still use children references to map the value property, but the name property is manually handled. In particular, we are setting it using the text property of ANTLR terminal nodes – the NAME token in this case.

Defining lambdas for complex logic is cumbersome, often not possible and generally discouraged in Python. In these cases, we can use user-defined function references – defs.

As mentioned in the previous section, while unary operation expressions are represented using a single kind of parse tree node, we decided to define dedicated nodes in our AST. Therefore, we need some slightly more sophisticated logic to map UnaryOperationExpressionContext parse tree nodes into UnaryOperation AST node subclasses, as follows:

from pylasu.parsing.parse\_tree import to\_positionfrom pylasu.validation import Issue, IssueSeverity, IssueTypefrom pylasu.transformation import PropertyReffrom slang.ast.nodes import Minus, Not, Plusfrom slang.parser.antlr.SlangParser import SlangParser as \_# node factory from user-defined functiondef unary\_operation\_node\_factory(source: \_.UnaryOperationExpressionContext): # common builder function for unary operations def build\_unary\_operation(unary\_operation\_constructor: ...): unary\_operation = unary\_operation\_constructor() # manually invoke the transformer to transform the 'expression' property unary\_operation.expression = transformer.transform(source.operand) return unary\_operation # branch depending on the operator if source.NOT(): # create a 'Not' AST node instance return build\_unary\_operation(Not) elif source.ADD(): # create an 'Add' AST node instance return build\_unary\_operation(Plus) elif source.MINUS(): # create a 'Minus' AST node instance return build\_unary\_operation(Minus) else: # oops... issues.append( Issue( IssueType.SYNTACTIC, f"Unsupported unary operation: {source.operator.text}", IssueSeverity.ERROR, to\_position(source) ) ) return None# register the node factorytransformer.register\_node\_factory( \_.UnaryOperationExpressionContext, unary\_operation\_node\_factory) See the oops there?This example also illustrates how issues can be added during the transformation. In this case, we are creating a syntactic error issue in case an unexpected unary operator is found. This should not happen given that the ANTLR parser would not recognize other unary operators…but better be safe than surprised.

All the other mappings fall into one of these categories and are omitted in this article for brevity. Please check the complete source code in the transformations.pymodule on GitHub.

‘Packaging’ our ParserAs mentioned at the beginning of this article, our objective consists in implementing a parser and integrating this into a CLI allowing users to parse strings and files.

Before doing so, it is convenient to wrap our parser capabilities into a well-defined interface. More specifically, we expose two functions from a main.py module, namely parse_string and parse_file:

from antlr4 import CommonTokenStream, FileStream, InputStreamfrom pylasu.validation import Resultdef parse\_string(code: str) -> Result: return parse\_input\_stream(InputStream(code))def parse\_file(filename: str) -> Result: return parse\_input\_stream(FileStream(filename)) Both functions wrap their input into an ANTLR input stream and delegate the actual work to another function named parse_input_stream. The result consists in a Resultinstance, which is a utility class provided by Pylasu to represent parsing results. More specifically, it keeps a list of possible issues encountered during the parsing process and a reference to the root of the AST.

The issues list can be populated with errors thrown by the internal ANTLR parser or user-defined ones emerged during subsequent phases, e.g. as seen with our unary operation node factory in the previous section.

What is happening inside the parse_input_stream function? Well, we put all the previous steps together and build a Result instance out of it, as follows:

from antlr4 import InputStreamfrom pylasu.validation import Result, Issuedef parse\_input\_stream(input\_stream: InputStream) -> Result: # keep track of all issues issues: List[Issue] = [] # 1. build the ANTLR Parse Tree parse\_tree = \_build\_slang\_parse\_tree(input\_stream, issues) # 2. transform the ANTLR Parse Tree into the Pylasu AST abstract\_syntax\_tree = \_build\_slang\_abstract\_syntax\_tree(parse\_tree, issues) # 3. build a Result instance and return it back return \_build\_pylasu\_result(abstract\_syntax\_tree, issues) The complete source code is omitted for brevity, please checkout the main.py module on GitHub for further details.

A Pretty-Cool Command-Line InterfaceWe are finally ready to implement our PC-CLI!

For this, we are going to cheat a bit and delegate the ‘make-it-cool’ task to Typer, a Python package supporting the creation of command-line interfaces with built-in auto-completion, argument validation and so on.

Sort of a version of Click on steroids – psss…that is actually what they are using inside! Check out their documentation for a complete overview of the supported features and feel free to make our CLI even cooler.

We can install the package as follows (if you are using our project setup):

pdm add "typer[all]" Once completed the installation, setting up a series of commands is pretty straightforward. In our case, we want to structure our interface so that users can run the following commands:

slang parse string "<slang\_code>" # parse slang code from a stringslang parse file <file\_path> # parse slang code from a file In order to do so, we define the following decorated functions:

from pathlib import Pathfrom typer import Argument, Typerfrom typing\_extensions import Annotatedfrom slang.parser import parse\_file# initialize a Typer applicationapp = Typer()# add the command group 'parse'parser\_app = Typer()app.add\_typer(parser\_app, "parse", "Parse slang code from strings or files")# add command 'string' to the 'parse' group@parser\_app.command("string", "Parse slang code from a string")@track\_progress("Parsing")def from\_string(code: Annotated[str, Argument(help="...")]): return build\_report(parse\_string(code))# add command 'file' to the 'parse' group@parser\_app.command("file", "Parse slang code from a file.")@track\_progress("Parsing")def from\_file(path: Annotated[Path, Argument(help="...")]): return build\_report(parse\_file(str(path.absolute()))) Thetrack_progressdecorator function keeps the user entertained while the parsing process is running, i.e. it shows a progress bar. Instead, the build_report function just serializes our AST into JSON. Their complete specification is omitted for brevity, please check out their source code on GitHub.

There are various ways to expose scripts from a Python package, you can find the complete configuration inside our pyproject.tomlfile. In our case, we just added a script in the tool.pdm.scripts configuration to be able to execute commands like pdm run slang, as follows:

[tool.pdm.scripts]slang = { call = "slang.cli:app" } In order install the package and be able to run it using python -m slang, we can just add a __main__.py file into our package, create an instance of our command-line interface and change its default name, as follows:

from .cli import appapp(prog\_name='slang') One last thing, update the pyproject.toml file with a project.script configuration as follows to be able to install the package and use it as an ordinary prompt command:

[project.scripts]slang = "slang.cli:app" Now, we are finally able to play with our CLI. Here are some execution examples, feel free to run these by yourself and have fun!

Asking for help Help…again Parsing strings The full output is omitted here, run it to see the entire AST.

…and parsing files In this case, we parsed our fibonacci function and successfully got back the corresponding AST with no issues, yay!

ConclusionsIn this article, we implemented a parser for the Slang toy programming language in Python using Pylasu and ANTLR.

First, we defined the ANTLR grammars and saw how antlr-tools can be used to trigger the generation process. Then, we defined our AST using Pylasu and implemented a model-to-model transformation to create instances of it from ANTLR parse trees. Finally, we integrated our parser into a CLI providing users the possibility to parse Slang code from both strings and files.

Pss…check out the project repository to see how the whole ANTLR generation process can be integrated into the build workflow using Tox. This might be a topic for a future article, let us know what do you think about it.

You did it, you reached the end of this article and we hope this was a fun and interesting drive! Get in touch with us on social media, through the project repository – or carrier pigeons if you like – for comments, feedback, everything really…and have a nice day!

The post Implement Parsers with Pylasu appeared first on Strumenta.

View Details

Sometimes, we need to parse some code for all sorts of purposes: building an interpreter, generating something out of it, or perhaps building an editor for that language. When this is the case, we may just start considering our options: should we build a parser ourselves, license a commercial parser or just use an open-source one? In this article, we explore these options and share some guidelines to support you in your decision.

What kind of applications can benefit from a parser?A parser can be used for many different goals.

Here we list the most common examples:

• You may want to transpile your code to some other language to reuse some tools supporting that target language. For example, you may want to transpile code in a certain language to Java or C to reuse existing Java or C compilers. To do that, a parser is needed
• You may want to generate something out of some code. For example, you may want to generate sequence diagrams from some code, or documentation. In this case you will need to parse the code to recognize the structure and extract the relevant information (e.g., the comments)
• You may want to build an interpreter or a compiler for an existing language because there is not one for your platform of interest. For example, someone at some point decided that running Javascript outside the browser and on their desktop computer was a good idea and they had to build a parser for Javascript
• You may want to design a new language, and therefore you need to provide a set of tools for your language such as an interpreter or a compiler, an editor and maybe some less widely used tools like a simulator. All of those tools would require a parser

What role does a parser play in these applications In all of the scenarios that we have seen a parser is a first component in a larger application. All of the applications we have listed are Language Engineering applications and we define their architecture using the concept of Language Engineering Pipelines.

In these Language Engineering Pipelines we combine different components together, where each one (but the first one) is consuming the output of the previous one and each one (but the last one) is producing something useful to the next stage. The first component does not take its input from the previous component but from the external world, and the last component does not provide its output to the next component but to the external world.

In all of these Language Engineering Pipelines the first component is always a parser that takes some code and produces an Abstract Syntax Tree (AST). The AST is a data-structure which contains all the information extracted from the code, in a form that simplifies implementing the rest of the Language Engineering Application.

This means that, while the parser per-se is not an application, it is a very useful component that enables us to build all sorts of interesting applications for a given language.

Important characteristics in a parserIf you are considering adopting a parser (possibly after having built it) we think you should consider these points:

• Implementation language: this is the language in which the parser itself is built, not the language recognized by the parser. You could for example have a parser written in Java (implementation language) that can recognize RPG code (parsed language). Why is this important? This is important because a parser is more often than not used as a library that you use inside your larger application. For example, if you want to generate syntax diagrams for RPG code, you can use a parser that can recognize RPG and then use its output as an input for your logic that actually prints the diagram. If the implementation language of the parser is Java, you will be able to write such logic in Java (or in a JVM compatible language). So you may want to be sure to choose an implementation language that your team is comfortable with.
• AST APIs: the parser produces an AST, which is then consumed by the other components of your application. Now, the better the API exposed by the AST, the easier it is to write the following components. Some ASTs provide advanced API to find the elements you are interested in, filtering them, transforming them or generating other ASTs from them. Others instead offer a more basic API relying on a visitor or a listener. If you are not familiar with Language Engineering applications this is a point you may tend to underestimate, but it can make a difference and significantly impact the complexity of writing and maintaining the other components in your Language Engineering application.
• Completeness/Correctness: can your parser parse all the valid files you need to process? And can it do that correctly, e.g., without producing errors that are not there or recognizing constructs incorrectly? While having a complete and correct parser is always desirable, this may be more or less important depending on the type of application you are building. A compiler that you intend to use to process tens of thousands of files may need to be more complete and correct than a parser you want to use to generate a few syntax diagrams.
• Maintainability: in this case we mean, in how much time an identified problem can be solved? Is this something that could take weeks and a prayer, or can you expect problems to be solved in a matter of hours or days? Depending on the situation this may be very relevant or not at all. If you use the parser inside a compiler that is vital for you, you may need to be sure to get the problem fixed quickly, while if you are doing a migration planned to take a very long time you may be more relaxed
• Time to adopt: how urgent is it to be able to start using the parser? Can you afford waiting a few months or do you need the parser for something you would like to put in production in a couple of months?
• License: does the license under which the parser is obtained work for your goals?

Building your own parser: what does it mean?We have seen what a parser can be used for, and we have seen which characteristics are important in a parser, so let’s see what it means to build your own parser with respect to those characteristics.

Note that most of the following considerations apply also if you have someone else build a parser specifically for you (so that you end up owning the resulting codebase).

• Implementation language: if you build your own parser you can implement it in the language you prefer, provided there are parser generators for that language. While it is true that you could also build your own parser without relying on a parser generator, that requires way more effort and we would not advise doing that. We suggest using a parser generator and ANTLR in particular. ANTLR is a tool that, given a grammar, generates a parser written in any of these implementation languages: C++, C#, Dart, Java, JavaScript, PHP, Python3, Swift, TypeScript.
• AST APIs: if you build your parser using ANTLR exclusively you will get a parser providing barebone APIs, however you can add a level on top of it to get more powerful APIs. To do that you can use one of our open-source libraries, which are collectively called StarLasu. At this time they are:
• Kolasu, written in Kotlin. You can find a tutorial for it here: Building advanced parsers using Kolasu
• Tylasu, written in Typescript
• Pylasu, written in Python
• Sharplasu, written in C#
• Completeness/Correctness: here it is up to you to develop the parser up to the required standards for your use case. Besides the effort you can spend on it, you can be limited by the availability of examples, specifications, or the experience in testing parsers.
• Maintainability: also in this case you are a master of your own destiny. Which may or may not be a good thing. It may be a good thing if your team has experience in building parsers, because in that case, if you can reserve capacity for supporting the parser, every problem should be fixed in a timely manner. If you do not have the skills or you cannot protect some time for maintaining the parser, then issues can take an unpredictable time to be solved. And you may meet problems you just do not know how to solve.
• Time to adopt: building a parser from scratch can take a developer 4 to 8 months, for most typical languages. Of course the complexity of the language, the completeness of the parser, and the quality of the APIs exposed are all important factors, but this is a ballpark figure for your reflections. These values are for developers who already know how to build a parser, so you may need to add a few months for your team to get up to speed with the parsing technologies needed, if they have no previous experience.
• License: no problem at all here. If the code is yours, you can do whatever you want with it.

Using an open-source parserLet’s see what happens when you adopt an open-source parser.

• Implementation language: here you need a bit of luck. The fact is that you are looking for a parser able to recognize a certain language and implemented in a language that your team is familiar with. For example, you may need to process Java code inside a Javascript application. In your quest for a parser you may find JavaParser, a parser that recognizes Java, and which is written in Java. Bad luck: you cannot use it in your parser. Now, if you can work with a parser with barebone APIs, you can then just pick an ANTLR grammar and from that generate the parser in one of the ten implementation languages supported by ANTLR, so that could help increase your chances.
• AST APIs: most open-source parsers we encountered provide basic APIs, as they are mostly based on ANTLR and do not have a proper AST on top. There are notable exceptions, like JavaParser, which offers advanced APIs. There are also other advanced parsers for very widespread languages, like XML or JSON. Here you have way better chances if your language is very, very popular.
• Completeness/Correctness: your mileage can vary, as there are parsers which have been around for many years, and have been used and refined a lot. If you see that the project has at least 1,000 commits, chances are that some time and care have been invested in the parser. Consider that for example JavaParser at this time has more than 9,000 commits (1,663 from myself :D). If your parser has less than 100 commits, you can consider it as a stub that may be a good starting point to build your own parser, but do not expect it to be something you can use “as-is”
• Maintainability: here our advice is to look at the average age of issues. If there are hundreds of issues that have been around for years, you may have two factors causing it: 1) the team of contributors may be just not big enough to handle all the requests or 2) the parser is actually hard to maintain. In the first case, you can have your own team contribute to the parser. You may need to invest some time in familiarizing with the codebase but that is an option. In the second case you have instead a problem that is harder to crack.
• Time to adopt: good news, you can start using the parser right away.
• License: you need to check if the parser is released under a license that works for you. Chances are that MIT, BSD, or Apache License v2 could work, while GPL would probably not work if you plan to use the parser in a commercial project.

Licensing a parser, how does that work?The third option is to license a parser that someone else has built. Let’s see what it means.

• Implementation language: like when adopting an open-source parser, you need to find a parser that processes that language you are interested in and also is implemented in a language you are comfortable with. Finding the right parser that satisfies these two requirements can require some luck, as there are not so many commercial parsers available out there.
• AST APIs: a proper commercial parser should have advanced APIs. It should have been built by professionals that do this all day long. Also typically these vendors also provide consulting services around their parsers, and therefore have the interest of having parsers that can be used successfully in Language Engineering projects. A key factor for that is having advanced AST APIs.
• Completeness/Correctness: if the parser has been around for a while it should be reasonably complete and correct. And you should ask the vendor to demonstrate that to you.
• Maintainability: if there is a vendor behind the parser, it is reasonable to expect that some form of support is provided.
• Time to adopt: as in the case of an open-source parser, it is just a matter of starting using the parser. If your purchasing department does not get in the way, obviously.
• License: here you may want to read the fine prints. Typically commercial parsers are provided to companies that want to use them for commercial purposes, so licenses should permit this kind of usage, but here your lawyers can earn their salary and double check that.

What is the best option in my situation?Deciding which ways to go depends on your situation, but a few points you could reflect on is:

• Have your team time available to build or maintain the parser? If not, a licensed parser, with an adequate support contract can be the way to go
• How soon do you need to get the parser? If you have time pressure for getting the project out of the gate, then building your own parser is something you cannot afford
• Is your project going to be open-source? If so, using a licensed parser may not be feasible
• Is there an existing parser, open-source or licensed, that can parse the language you are interested in and it is implemented in your language of choice? If not, then you can only build this parser yourself

SummaryDeciding how to source the parser for your Language Engineering project is a complex decision, especially if it is the first time that you look into such problems. With this article we listed a few aspects we suggest to consider, based on our experience. While some are more obvious (like the license), others are sometimes overlooked (like the AST APIs or the implementation language). Hopefully this guide can help you in making up your mind. And if you are not sure, we can provide consulting options to help make a decision.

Also, if you decide to build your own parser, we suggest taking a look at our video course on building parsers with ANTLR. You could also apply the principle of the Chisel method for building parsers, a method we developed at Strumenta. It would help in ensuring maintainability and correctness for your parser.

All the best for your decision and your project!

The post Getting a parser: build it, use an open-source one, or a commercial one? appeared first on Strumenta.

View Details

We interviewed Rutger Van Beusekom the CTO at Verum, a company based in The Netherlands. We talk about a domain specific language that they have been developed at Verum for supporting formal verification of code, in particular for embedded systems.

Federico Tomassetti

So, hello Rutger, it’s very nice to have you here today. How are you today?

Rutger Van Beusekom

I’m doing well, suffering from serious cold, having traveled to San Francisco. 9 hours time difference and sleeping not too well, but beyond that, I’m doing great. Nice to meet you. And thank you for giving us the opportunity to talk about our passions.

Federico Tomassetti

Okay, so we can appreciate a lot that you took the effort after just coming back from San Francisco to take this interview. And yeah, we’ll just say that you are the CTO at Verum, a company based in The Netherlands. Today, we are here to discuss a domain specific language that you have been developed at Verum for supporting formal verification of code, in particular for embedded systems. So I hope I get the overview, right. But of course, we will dive into more details. As we proceed in the interview. I think it makes sense to start with telling us a little bit about your company Verum.

Rutger Van Beusekom

They’re actually founded in 2004. So we’ve been around for a while, and the original founders have left in the meantime; and I’ve taken over in the role of CTO. Officially since 2019, but effectively already since 2014. Yeah we’ve been serving with an original product, which was called ASD at the beginning and later on the rebooted the company in creative Dezyne. We’ve been serving companies like ASML, Phillips and Thermo Fisher, which all exist in the Eindhoven region in the Netherlands. So or focus has been quite close to home.And now we’re starting to expand our horizon.

Federico Tomassetti

Well, quite a list of impressive clients. So it’s not easy to find clients at that level. Okay, and today, in particular, we want to discuss about Dezyne, your domain specific language. So why did you create it in the first place?

Rutger Van Beusekom

I grew up actually, within Philips, that’s where I started, and Phillips, was doing, was adopting object orientation at that time. And I spent less time doing object orientation, which is inherently imperative. Doing that in a concurrent environment, is basically a recipe for disaster. It is just too complicated to get right. Especially on embedded systems, where you interact with the physical world, and having some form of correctness is paramount. And I was introduced to the concept of model checking at my time at Phillips by the original founder of Verum. And having seen that was a eureka moment for me. I have to do something with this. So that’s the mission. Bringing formal methods, model checking, to the masses, basically.

Federico Tomassetti

Yes. And can you tell us about the ideal user that you have in mind for design?

Rutger Van Beusekom

Preferably, in the future, and that’s not where we are today. Anyone using a language like C or C++, or C#, Java, Python, doing a serious amount of development, where either asynchronicity or concurrency plays a role, whether it’s in a cyber physical system or not, but with a high level of complexity, they should benefit from fully automated verification. So that’s the group we are aiming for. As far as I’m concerned, basically every programmer. But we’re trying to lift to the level of abstraction, in order to capture requirements at the formal level, such that you can prove that throughout the entire state space of your software system, and even the system of systems. This sounds very ambitious, we’re not quite there yet. That’s the end goal.

Federico Tomassetti

Good. So. Okay, so it’s clear that you’re targeting developers. And so is the main advantage for them, basically, they get formal verification for free by using the language or is something else?

Rutger Van Beusekom

I want to say yes, but verification is not a free lunch, you still have to express what you want to prove, formally. So if you’re not willing to do that work, like doing test driven development, you’ll have to write the tests, and you have to do that intelligently. Otherwise, they will not help you. The same holds for describing the essential properties that you need to prove automatically. But that’s, that’s the unique selling point, yes. The scope of what we’re trying to do is larger, effectively, we’re trying to augment our limited human mental capacity, using the power of the computer. There are, of course, automation is one big thing, so we’re doing code generation, we’re doing simulation, we’re doing verification. That automates a lot of the work that we, as humans can do with our brain, but we’re not just not that good at it, to do it systematically and completely and correctly. It’s better to have the computer do that for us. So effectively, we’re trying to figure out a new way of software engineering, which connects to the existing ways of software engineering, but raising the bar.

Federico Tomassetti

So, you right, correct me when I say that formal verification comes for free, because it doesn’t come for free, or if I understand correctly, require an extra effort. And so to your users, or the people that you discuss with value, formal verification enough to accept this extra effort. So do, do they see the value this?

Rutger Van Beusekom

We have quite, we’ve been able to quite naturally embed certain implicit requirements into our language.I will briefly explain what the nature of the language is. So I talked about object orientation in concurrent environment. Our language supports component based development, which I look at as a subset of object orientation, a stricter subject. In object orientation, nothing limits you, you can make it as complicated as you want, in our approach components, are very strict. A component is completely encapsulated behavior, being unaware of what the state or the implementation details behind its formal interfaces are. So design requires you to model any interaction with a formal interface. And an interface is just like an API, it lists the functions you could call out or receive, with data or not, with return values or not. But we also describe the protocol of that API explicitly. And then, in our model checking, we prove that you’ve got an effective protocol contract that implementations on either side of the interface maintain that contract. And that’s a very natural way for most software engineers with at least some object oriented background to describe their software. On top of that, we have user definable properties that can be expressed as boolean predicates across many interfaces, or all across the local components state variables. Language is inherently imperative. But also declarative, somehow, I won’t go into detail. If you want to know more, have a look dezyne.org, we can find all the information you want.

Federico Tomassetti

Can what were the main challenges in designing such a language? I suppose it’s the first language you design, or not?

Rutger Van Beusekom

When I joined Verum, there was a prototype of the concept of the original CTO, which is called ASD, or Analytical Software Design, I think it was called. And effectively, it was inspired by the clean room method by Harlan Mills, I believe, which was a tabular notation effectively of a state machine. And it already had the concept of something like components and interfaces. But because it was a table, it was not actually a language, as such. So in Dezyne, we added the textual notation. And there has been many, many challenges, from a business point of view. From a management point of view, or funding point of view, technology, and people point of view. I don’t know. I guess your listeners would be mostly interested in the technical side of things.

Federico Tomassetti

All the aspects are interesting, because, you know, if you’ve write a great language from the technical point of view, but then you don’t get support from people, then there is no point. So

Rutger Van Beusekom

Exactly. That’s still our biggest challenge, stepping out into the world and, well finding recognition. That’s why we’re still actively developing the language, we don’t believe ourselves that it’s quite finished yet. Cooperating with our current customers is a great help there. Yeah, if you’ve got more specific questions, for challenges, I had more than happy to answer those. Otherwise, I’ll just continue ranting

Federico Tomassetti

then I would be interested in editor support because one aspect that sometimes people building language underestimate is the difficulty of building an editor. So I wanted to ask, what efforts have you done in this area or

Rutger Van Beusekom

We’ve had many experiments, and enough failures, which is a successful experiment, of course, finding out that something doesn’t work. And on the editor support we definitely had a few of them. So our original product was a completely purpose built Excel like editor, which I personally found horrible, but some people really liked. But that was the old product, when we started designing a textual language, it was the intent to be able to use any editor to write the language but of course, modern programmers expect significant editor support. And initially, we selected Eclipse as the launching editor to be in, and we effectively developed using Xtend, I think it’s called, the language support. But in the end, it resulted in having, I believe, four or five separate parsers for the same language. So I really wanted to unify all of those parts into a single implementation they could serve all the different uses of parsing. So, finally, we settled on. We use scheme to do most of our developments. Dezyne is actually developed, written in scheme. In scheme, we use Guile as our scheme implementation. It has a PEG, a Parser Expression grammar library, which we extended, which is now being used in our LSP, Langiage Server Protocol plugin, we use it in our compiler for all the different use cases that that our language offers, so we’re able to make a single parser to rule them all. But that has been quite a challenge.

Federico Tomassetti

And it’s it’s an experience that I can relate to because I also started my adventure in this area by using Xtext that I think at the time was great than then probably Eclipse didn’t evolve, as well as we were hoping, maybe. But about evolutions, I think it will be interesting to hear about your plans for language, if you plans more of more work on the technical side or on the promotion side or any other direction, which you may want to work.

Rutger Van Beusekom

So as CTO, my two primary goals or assignments are to, to push for, for more recognition in the outside world, build a community. But in doing that, of course, address or close the gap to what people actually need and expect from such a language. And for the foreseeable future, we identified that we need to make data a first class citizen in Dezyne. So we do support data, but if you take a single 32 bit integer, the state space of that integer is already vast. From my head, it’s more than 4 billion numbers. And in combining a few entities would blow up the state space during verification. So doing this effectively, is what we need to achieve. Because that’s one of the extensions we’re thinking of. And a long standing feature that we want is system wide functional verification. So I talked about those user defined properties. We’ve come up with the language extension to express those properties, just boolean predicates. But we need to lift that to the composition of our components to be able to do that performantly, at the system level. So we believe that that is what the community is lacking for large scale pickup of Dezyne.

Federico Tomassetti

And maybe this is connected to the promotion side. But I also saw that you had a couple of publications on your work. So I wanted to ask you about your speed and see if about your experience in general with scientific publications.

Rutger Van Beusekom

I’d like to spend more time publishing about Dezyne, because I think that’s an essential platform to get the message across. And it’s also a way to have at least academic community validates what we’re doing, or invalidates such that we can actually address those things. Yeah, I think there’s three papers out there now on Dezyne. Actually hoped to start promotion work at some point and use that as a driving force to produce more of those papers. I think that the last paper was written by Michele, I forget his last name, which I contributed to. So yeah, it would be great if the scientific community wants to research on the basis of what we’re doing? Because we are very much an R&D organization that we can use more help. Publishing is is one of the ways to do that.

Federico Tomassetti

And that you’re lucky to be based in the Netherlands because I think there are several research group doing a lot of work at Delft or Amsterdam in this area. So nice place for being based. And did you find any resistance from developers in adopting domain specific language, because sometimes developers, you know, consider themselves a C++ developer, and you know, it is really part of their identity. So everything else is not their job.

Rutger Van Beusekom

Yeah, not acceptable. That’s an inherent fact, we haven’t figured out a true solution for that. Our approach to this is to have. Dezyne is actually intended as a coordinative language. So about 90% of the software has more or less been written, and we focus on the 10%, which is missing as a society. So we have to be able to deal with the 90%, which is already written. So people should not have to be forced to rewrite that all of that code into a new language. So we’re trying to make Dezyne more like a coordinative language, we’ve, we’ve added model based testing for existing implementations in the hope to bridge the gap for those, for instance, C++ lovers not to have to say goodbye to the code they’re still attached to. And of course, from a monetary point of view, leveraging existing code, which works is essential. But fundamentally, if someone is really in love with a specific language, and they don’t see the benefit of what Dezyne offers, I can only say more power to you. And I expect at some point, you’ll find that it’s it’s a lot of work in conventional languages to be truly productive, and to achieve a level of correctness, which I think society will push to raise the bar on, in the foreseeable future.

Federico Tomassetti

Not sure if I understand exactly to mine, but do you mean that I can have components written in other languages? And right, maybe the combination of these components in Dezyne, or did I misunderstood

Rutger Van Beusekom

Yeah Dezyne is intended to be a hybrid meta language effectively. It principle, you can develop a new code in Dezyne because we generate C++ or C or Java or Python, at request. So if you’re doing greenfield development, all you need is Dezyne. But no one is doing greenfield development, it’s usually brownfield development. So in order to add automated verification to a brownfield, that requires adding specification for what is already there, and proving that, that specification is consistent with those implementations. And we figured out a way to do that using model based testing. And when we have those specifications, we can build the new code using Dezyne on top of that, and know that it’s consistent and correct using model checking. That’s the hybrid approach. So the targets will always, the target code will always be C++ or C, or whatever a user chooses. But they will do their, their orchestration and their verification at the model level an with a push of a button generate the missing bits.

Federico Tomassetti

Makes sense. And so I think, in this interview, we explain the value of Dezyne and so I hope that at this point, people will be interested in learning more about the language so can you give us any pointer for learning more?

Rutger Van Beusekom

Yeah, we have two websites. Free Software, free as in free speech of freedom, not as gratis is dezyne.org, this is where we publish open-source or open-core for Dezyne, were are actually a NON-GNU project and hope to be a full blown GNU project in the future. And we’ve got the commercial websites verum.com where you can get your commercial support. On the only dezyne.org website, we publish the language reference manual, we’ve got tutorials, you can find the source code. And on the commercial site, you can find a binary version and get in contact with our sales.

Federico Tomassetti

So we will make sure to add all the links in the transcript. And so my final question is, what do you plan to work on next?

Rutger Van Beusekom

I think priority one is community building, and secondary, extending the expressiveness of the language and the corresponding tooling. I think we’ve got about five feature branches still up in the air that needs to become part of our release train. So I think in the last 12 months, we have fundamentally extended the language we we’ve added the concept of collateral blocking we’ve added a defer keyword we’ve added the concept of constraining interfaces and shared interface state. So these are all fundamental changes to the language making it more expressive. And we’ll continue to do that in the future. I talked about the system wide functional verification, I think that has the main attention on either side which we will integrate making data as a first class citizen into. Yeah I think that is it for now!

Federico Tomassetti

Enought to keep you busy for a while. Okay, so I think it’s time for me to thank you again for giving this interview and for creating these domain specific language. I think it can be an inspiration for authors to learn from your effort on the technical level and community building level on your experience building also editor support. So I think this has been very valuable and thank you

Rutger Van Beusekom

Thank you very much. I hope it will inspire some someone out there.

Federico Tomassetti

right, so Good

Rutger Van Beusekom

Bye bye

The post Interview with Rutger Van Beusekom appeared first on Strumenta.

View Details

IntroductionIn this article, we will explore how to build a rich JSON validator website with Tylasu. As developers, most of us are familiar with JSON. It has become the go-to format to represent data. Chances are that, at some point, you have wanted to validate JSON data or ensure it contains all the information you expect.

Some existing websites like jsonlint or jsonformatter offer you that. But have you ever wondered how you could build such a website? Of course, there are a million ways to do it. But choosing the right tools can make the task easy, even fun.

Here at Strumenta, we build all kinds of tools related to languages, and data validators are no exception. In this tutorial, we will show you how to leverage the power of Tylasu, ANTLR, and Monaco, all open-source tools, to build a friendly JSON validator website. Here is what it looks like:

You can play with it online at https://strumenta.github.io/tylasu-tutorial

It is a website that looks and acts like an editor. In fact, it’s based on the Monaco editor by Microsoft. Whenever you type on it, it parses the code expecting some JSON. It displays error markers over incorrect code, and colors each member depending on the type of its value.

It even has a secret feature: if an "applyMonacoConfiguration": true member is present, the code will be interpreted as Monaco configuration and applied to the editor on the spot. Try changing the font size or the editor’s theme like this:

The complete codebase is on Github, with a dedicated git tag for each of the seven tutorial steps. We can use them to check our progress or skip over steps completely. Everything ready? Let’s start!

1. Creating the editorWe are huge believers in having something visual and interactive to play with as soon as possible.
   So, let’s start with the editor.
   It’s a regular HTML, CSS and JS static website:

<head> <link rel="stylesheet" href="index.css"> <link rel="stylesheet" href="bundle.css"> <script src="bundle.js" defer></script></head><body> <section id="editor"></section></body>

body { margin: 0;}#editor { height: 100vh;}

import * as monaco from "monaco-editor";monaco.editor.create(document.getElementById("editor")!, { theme: "vs-dark", fontSize: 22 }); The page has a single element, editor, that is styled to take up the whole viewport.
The code creates a Monaco editor inside this element with a dark theme and a bigger font size. We will give the user the ability to customize the editor later in the tutorial.

Of course, we need to bundle the TypeScript code with its dependencies into the bundle.js and bundle.css that the HTML file is expecting.

First, let’s install the monaco-editor node package as dependency:

$ npm install monaco-editor Then, we’ll create the bundle executing the esbuild node package with npx:

$ npx esbuild index.ts --bundle --outfile=bundle.js --loader:.ttf=empty And the website is ready!
We can serve it locally using the serve package:

$ npx serve . -p 3000 Now we can open a browser at http://localhost:3000 and enjoy typing some code.
Developers who are familiar with VS Code can use the same keyboard shortcuts here. In fact, Microsoft created Monaco to serve as the editor in VS Code.

1. Structure of the projectNow that we have tasted the cake, let’s take a moment to organize our files.
   We’ll follow a standard project structure, that happens to be the same that Maven projects use in Java.
   Let’s move the HTML and CSS to src/main/resources, let’s put index.ts under src/main/typescript and leave the package.json in the root folder with some useful modifications:

{ "dependencies": { "monaco-editor": "^0.44.0" }, "devDependencies": { "esbuild": "^0.19.8", "serve": "^14.2.1" }, "scripts": { "all": "npm install && npm run createDistribution && npm run bundle && npm run serve", "createDistribution": "mkdir -p dist && cp src/main/resources/* dist", "bundle": "esbuild src/main/typescript/index.ts --bundle --outfile=dist/bundle.js --loader:.ttf=empty", "serve": "serve dist -p 3000" }} We don’t need access to esbuild and serve at runtime, so we can mark them as devDependencies.
Defining scripts inside package.json allows us to run tasks using npm run <task-name>.
In particular, the npm run all task will build the entire project from scratch by:

• Installing all the dependencies
• Creating a dist folder and moving the HTML and CSS to it
• Bundling the code and its dependencies into dist/bundle.js and dist/bundle.css
• Serving the dist folder at http://localhost:3000

Now, every time we make a change, we can run

$ npm run all to serve the website with the latest changes applied.

1. Parsing the Code in the EditorLet’s provide some feedback depending on the code the user types in the editor.

We will use ANTLR, an industry-standard tool for language recognition. ANTLR is very powerful and can understand everything from small DSL programs to full C++ projects. To keep the example familiar to most developers, we will use it to parse JSON, but feel free to experiment with other languages.

ANTLR takes formal language grammars as input and produces the source code of a parser as output. Thankfully, the official ANTLR repository includes many grammars of popular languages ready to be used, including one for JSON. Let’s say thanks and copy the json grammar into src/main/antlr/JSON.g4.

For anyone curious about ANTLR parsers, we have many resources available.

ANTLR for TypeScript comes in two packages:

• the antlr4ts package that we will use as a runtime dependency
• the antlr4ts-cli devDependency that generates the TypeScript files that parse the given grammar

Unfortunately, antlr4ts is not ready to be used on the browser out of the box. It expects to find node modules that do not exist in the browser like assert and process. We can walk around this issue by installing the assertnpm package and replacing the code process.env.NODE_DEBUG with false with esbuild:

{ "dependencies": { "monaco-editor": "0.44.0", "antlr4ts": "0.5.0-alpha.4", "assert": "2.1.0" }, "devDependencies": { "esbuild": "0.19.8", "serve": "14.2.1", "antlr4ts-cli": "0.5.0-alpha.4" }, "scripts": { "all": "npm install && npm run createDistribution && npm run bundle && npm run serve", "generateParser": "antlr4ts -Dlanguage=TypeScript src/main/antlr/JSON.g4 -no-listener", "createDistribution": "mkdir -p dist && cp src/main/resources/* dist", "bundle": "esbuild src/main/typescript/index.ts --bundle --outfile=dist/bundle.js --loader:.ttf=empty --define:process.env.NODE\_DEBUG=false", "serve": "serve dist -p 3000" }} A promising new node package called antlr4ng has been recently published with the intention to supersede antlr4ts. We will support it in the future, but for now, let’s stay on known territory.

Now npm run all executes the generateParser task that will create the JSONLexer.ts and JSONParser.ts files under src/main/antlr. We can use these to parse the code present in the editor:

editor.onDidChangeModelContent(() => { const code = editor.getValue(); const characters = CharStreams.fromString(code); const lexer = new JSONLexer(characters); const tokens = new CommonTokenStream(lexer); const parser = new JSONParser(tokens); const tree = parser.json(); console.log(tree);}); This is the standard pipeline for a parser. It reads the code from a character stream, the lexer transforms it into a stream of tokens and finally the parser transforms the tokens into a tree of structured content.

We should now see errors and parse trees printed on the browser’s developer console. We can confirm that when the code is valid JSON we get no errors:

1. Showing issues over the codeIf you have played with the ANTLR parser, you may have noticed that neither the errors nor the resulting tree are very friendly. The trees are hard to mutate, and the nodes’ information is tied to the grammar rather than to what they represent. This is where our last piece comes in: Tylasu.

Tylasu is an open-source package to create and process beautiful abstract syntax trees (ASTs for short). It provides utilities to model nodes, their relationships, and transformations between them. It simplifies the creation of parsers to enable the development of sophisticated language engineering tools on top, like compilers or editors.

Let’s see how it can help us show friendly errors in the editor. First, we install the @strumenta/tylasu package and modify the bundle script one last time to disable loading the fsmodule on the browser:

{ "dependencies": { "monaco-editor": "0.44.0", "antlr4ts": "0.5.0-alpha.4", "assert": "2.1.0", "@strumenta/tylasu": "1.4.5" }, "devDependencies": { "esbuild": "0.19.8", "serve": "14.2.1", "antlr4ts-cli": "0.5.0-alpha.4" }, "scripts": { "all": "npm install && npm run createDistribution && npm run bundle && npm run serve", "generateParser": "antlr4ts -Dlanguage=TypeScript src/main/antlr/JSON.g4 -no-listener", "createDistribution": "mkdir -p dist && cp src/main/resources/* dist", "bundle": "esbuild src/main/typescript/index.ts --bundle --outfile=dist/bundle.js --loader:.ttf=empty --define:process.env.NODE\_DEBUG=false --external:fs && sed -i 13d dist/bundle.js", "serve": "serve dist -p 3000" }} Now, we can create a JSONTylasu parser at src/main/typescript/ast/parser.ts

export class JSONTylasuParser extends TylasuParser<Node, JSONParser, JsonContext, TylasuANTLRToken>{ constructor() { super(new ANTLRTokenFactory()); } protected createANTLRLexer(inputStream: CharStream): Lexer | undefined { let lexer = new JSONLexer(inputStream); lexer.removeErrorListeners(); return lexer; } protected createANTLRParser(tokenStream: TokenStream): JSONParser { let parser = new JSONParser(tokenStream); parser.removeErrorListeners(); return parser; } protected parseTreeToAst(parseTreeRoot: JsonContext, considerPosition: boolean, issues: Issue[], source?: Source | undefined): Node | undefined { return undefined; }} It is basically a wrapper of the previous ANTLR parser code we had. However, it merges the result and the issues found in a nice ParsingResult object. We can remove the error listeners to no longer show them in the developer console. Instead, we will visualize them with Monaco right over the code, using the position information automatically provided by Tylasu:

editor.onDidChangeModelContent(() => { const code = editor.getValue(); const parsingResult = parser.parse(code); visualizeIssues(parsingResult.issues);});function visualizeIssues(issues: Issue[]) { const diagnostics: monaco.editor.IMarkerData[] = []; for (const issue of issues) { if (issue.position) { diagnostics.push({ severity: monaco.MarkerSeverity.Error, message: issue.message, startLineNumber: issue.position.start.line, startColumn: issue.position.start.column + 1, endLineNumber: issue.position.end.line, endColumn: issue.position.end.column + 1 }); } } monaco.editor.setModelMarkers(editor.getModel()!, "json", diagnostics);} This could already work as a JSON validator website, but thanks to Tylasu, we can do much better. How about coloring each object member depending on its type? For that, we need a deeper understanding of the code.

1. Chisel the parse treeWe start by modelling our ideal tree structure. In JSON, there are six types of values:

2. 4 literal values: numbers, strings, booleans and null

3. 2 collection types: arrays have ordered values and objects have pairs of name-values

We can create a class that extends the Tylasu Node class for each concept:

export class JSONValue extends Node { }export class JSONNumber extends JSONValue { value: number; constructor(value: number) { super(); this.value = value; }}export class JSONString extends JSONValue { value: string; constructor(value: string) { super(); this.value = value; }}export class JSONBoolean extends JSONValue { value: boolean; constructor(value: boolean) { super(); this.value = value; }}export class JSONNull extends JSONValue { } The four literal value classes probably contain no surprises. They hold a value and provide a constructor for convenience. The array and object nodes are a bit more interesting:

export class JSONArray extends JSONValue { elements: JSONValue[]; constructor(...elements: JSONValue[]) { super(); this.elements = elements; }}export class JSONObject extends JSONValue { members: JSONMember[]; constructor(...members: JSONMember[]) { super(); this.members = members; }}export class JSONMember extends Node { name: string; value: JSONValue; constructor(name: string, value: JSONValue = new JSONValue()) { super(); this.name = name; this.value = value; }} The main difference is they have fields that are nodes, also known as child nodes. In the current version of Tylasu, child nodes need to be initialized in the constructor. This requirement will be relaxed in the upcoming version of Tylasu.

Next, we define the parent-child relationships between nodes. Arrays and Objects may have multiple children, while Members have an individual child:

constructor() { super(new ANTLRTokenFactory()); this.createNodeDefinitions(); } createNodeDefinitions() { registerNodeChild(JSONArray, "elements", true); registerNodeChild(JSONObject, "members", true); registerNodeChild(JSONMember, "value", false); } Finally, we need to transform the ANTLR tree to our AST model. Thankfully, Tylasu includes helper classes to facilitate the conversion from ANTLR:

export const transformer = new ParseTreeToASTTransformer();transformer.registerNodeFactory(ValueContext, (x: ValueContext) => { if (x.NUMBER()) return new JSONNumber(parseFloat(x.NUMBER()!.text)); if (x.STRING()) return new JSONString(x.STRING()!.text.substring(1, x.STRING()!.text.length)); if (x.text === "true") return new JSONBoolean(true); if (x.text === "false") return new JSONBoolean(false); if (x.text === "null") return new JSONNull();}); As before, the four literal cases are simple. For nodes with children, we need to specify how to convert their children too:

transformer.registerNodeFactory(ArrContext, (context: ArrContext) => new JSONArray()).withChild( x => x.value(), (node: JSONArray, elements?: JSONValue[]) => node.elements = elements || [], "elements", ArrContext);transformer.registerNodeFactory(ObjContext, (context: ObjContext) => new JSONObject()).withChild( x => x.pair(), (node: JSONObject, members?: JSONMember[]) => node.members = members || [], "members", ObjContext);transformer.registerNodeFactory(PairContext, (context: PairContext) => new JSONMember(context.STRING().text.substring(1, context.STRING().text.length - 1))).withChild( x => x.value(), (node: JSONMember, value?: JSONValue) => node.value = value || new JSONValue(), "value", PairContext); With this information, the transformer will take care of walking over the entire tree and converting all the nodes. We just need to invoke it in the parseTreeToAST method of the parser:

protected parseTreeToAst(parseTreeRoot: JsonContext, considerPosition: boolean, issues: Issue[], source?: Source | undefined): Node | undefined { return transformer.transform(parseTreeRoot);} Now, parser.parse produces a tree with our ideal structure. Much nicer! It also includes types, position information, and child relationships. Also, each node has:

• an “origin” that remembers where the node comes from – in our case, an ANTLR parse tree node corresponding to a portion of source code. In other cases, nodes may originate from other Tylasu nodes after an AST-to-AST transformation, or may have been created programmatically;

• an optional “destination” node. When we apply multiple transformations, each intermediate node will also record what it’s been transformed into.

• Colorize the membersNow that we have an AST, analyzing and processing the code becomes a breeze. To colourize the members, we can:

• Walk the root node

• Filter the JSONMember nodes
• Pick a CSS class name depending on the type of its value
• Create a Monaco decorator in the node’s name’s position with the selected class name.

We can realize the above steps with the following function:

function colorizeMembers(root?: Node) { if (!root) return; const newDecorations: monaco.editor.IModelDeltaDecoration[] = []; for (const node of root.walk()) { if (node instanceof JSONMember) { if (!node.position) continue; let className: string = ""; if (node.value instanceof JSONNumber) { className = "number-member"; } else if (node.value instanceof JSONString) { className = "string-member"; } else if (node.value instanceof JSONBoolean) { className = "boolean-member"; } else if (node.value instanceof JSONNull) { className = "null-member"; } else if (node.value instanceof JSONArray) { className = "array-member"; } else if (node.value instanceof JSONObject) { className = "object-member"; } newDecorations.push({ range: new monaco.Range( node.position?.start.line, node.position?.start.column + 2, node.position?.start.line, node.position?.start.column + 2 + node.name.length), options: { inlineClassName: className } }); } } decorations = editor.deltaDecorations(decorations, newDecorations);} Now every object member in the editor will have a CSS class assigned depending on its type. We can customize their look in index.css:

.number-member { color: plum !important;}.string-member { color: burlywood !important;}.boolean-member { color: cadetblue !important;}.null-member { color: gray !important;}.object-member { color: lightgreen !important;}.array-member { color: lightsalmon !important;} Here we have just modified the color, but nothing stops you from changing fonts, size, or anything really. In fact, in the following, we’ll give the user the ability to customize the fonts and the colors of the editor itself.

1. Let’s add a little secretProgramming should always stay fun. Let’s take this chance to make the editor configurable on the spot by interpreting the inputted JSON as Monaco configuration.

If the input JSON is an object and contains the "applyMonacoConfiguration": true member, our secret feature will activate:

• If the input does not match the expected members or types, we will show validation errors
• Otherwise, we will apply the given configuration to the running editor:

function applyMotypescriptnacoConfiguration(root?: Node) { let diagnostics: monaco.editor.IMarkerData[] = []; const configuration = { applyMonacoConfiguration: JSONBoolean, fontSize: JSONNumber, theme: JSONString, lineNumbers: JSONBoolean }; if (((root as JSONObject)?.members?.find(x => x.name === "applyMonacoConfiguration")?.value as JSONBoolean)?.value) { for (const member of (root as JSONObject).members) { if (!(member.name in configuration)) { diagnostics.push({ severity: monaco.MarkerSeverity.Error, message: "Unexpected configuration member", startLineNumber: member.position!.start.line, startColumn: member.position!.start.column + 2, endLineNumber: member.position!.start.line, endColumn: member.position!.start.column + 2 + member.name.length }); } else if (!(member.value instanceof configuration[member.name])) { diagnostics.push({ severity: monaco.MarkerSeverity.Error, message: "Unexpected member type", startLineNumber: member.value.position!.start.line, startColumn: member.value.position!.start.column + 2, endLineNumber: member.value.position!.start.line, endColumn: member.value.position!.start.column + 2 + member.name.length }); } } monaco.editor.setModelMarkers(editor.getModel()!, "json", diagnostics); if (!diagnostics.length) { editor.updateOptions(JSON.parse(editor.getValue())); } }} ``` In this case, we only allow configuring the theme, the font size, and whether the line numbers are displayed. But of course, we could take this idea much further. Maybe even support the entire JSON schema specification. Or maybe create completely different validation rules.

The point is, that once we have an AST that fits our mental model, we can create any tool on top of it.

ConclusionWe’re done! We’ve gone over every bit of code present in the Github tutorial repo. But this is just the beginning.

ANTLR can parse all kind of languages. Monaco has all the features of the most used code editor in the world. Tylasu offers facilities to test and serialize ASTs to create professional and interoperable applications. The space of potential applications is huge.

Want more ideas on how to apply these new techniques? We have many articles on the topic.

The post Build a rich JSON validator website with Tylasu appeared first on Strumenta.

View Details

In this article, we introduce the Chisel Method, an open-source method for parsers and transpilers; an approach in the field of software engineering to address the challenges in developing parsers for modern, complex programming languages, by focusing on clear, objective goals and measurable outcomes.

The article provides detailed information about the development lifecycle under Chisel, from initial development to maintenance and expansion, highlighting its innovative features like robust connectivity, comprehensive documentation, and the importance of making the method teachable.

IntroductionIn the landscape of software engineering, the development of parsers remains a cornerstone, critical for interpreting and processing the programming of languages that power the digital world. Despite the well-established theories underpinning parsing, the actual process of building parsers often treads a precarious path, mired in unpredictability and a trial-and-error approach.

The Chisel Method attempts to turn the often chaotic process of building parsers into a streamlined, rule-based journey, ensuring reliability and efficiency. Whether you’re a seasoned developer or new to the field, by reading this article, you will learn how the Chisel Method attempts to turn the approach in parser construction, turning a complex process into an achievable, predictable task.

Parser Development ChallengesIt is a common misconception, in the context of parsing programming languages, that creating parsers is an already solved problem.

While the basic principles of parsing are well-understood, the practical application of these principles presents ongoing challenges.

Modern programming languages are becoming increasingly complex with rich syntax and semantics. This complexity means that creating parsers that can accurately and efficiently interpret these languages is an ongoing challenge. Each new language feature or syntactic sugar can introduce parsing ambiguities and complexities that need to be addressed.

Programming languages are not static, they evolve over time, new versions of languages often come with changes or additions to the syntax, which parsers need to adapt to.

Maintaining parsers for evolving languages is a continual process, not a one-time solution.

The Chisel MethodThe Chisel Method addresses the challenges in parser development through its innovative principles, offering a solution that is both efficient and effective.

The diagram below represents an overview of the process.

Clear Goal DefinitionAt its heart, Chisel establishes a clear, objective goal that is mutually understood and agreed upon by both the user of the parser and the Language Engineering Team.

The Chisel Method deliberately avoids using language specifications as the primary means of setting goals for parser development. This choice is based on several practical considerations:

1. Absence of Written Specifications: Many languages, especially domain-specific ones, may not have formal, written specifications. Relying solely on specifications to define the goals of a parser can be impractical or impossible in these cases. The Chisel Method needs to be adaptable to a wide range of languages, including those without formal documentation.
2. Difficulty in Measuring Progress: Measuring progress based on specifications can be misleading. For example, quantifying progress by the number of covered chapters or sections of a specification does not accurately reflect the actual work completed. Some sections may be more complex or time-consuming than others. This approach can give a false sense of progress and does not provide a clear indicator of how close the parser is to being functional or complete.
3. Verification Challenges: Ensuring that a parser fully “covers” a specification is a manual and subjective process. It’s challenging to automate this verification, making it inefficient and prone to errors. Without automation, the process becomes labor-intensive and can significantly slow down the development cycle.
4. Subjectivity and Interpretation Issues: Specifications can be open to interpretation, leading to disagreements between the Language Engineering team and the Client. What one party considers to be in compliance with the specifications may be viewed differently by the other. This subjectivity can lead to conflicts, revisions, and delays in the development process.

Instead, the Chisel Method focuses on concrete, measurable goals, such as the ability to parse a pre-selected set of example files and validate the correctness of the generated ASTs. This approach offers a more objective, quantifiable, and automated way to gauge progress and success, leading to a more efficient and harmonious development process. By sidestepping the ambiguities and limitations of language specifications, Chisel provides a more pragmatic and effective pathway to developing robust and reliable parsers.

DevelopmentChisel predicates its approach on having a well-defined, objective, and measurable goal. This clarity is crucial in aligning the efforts of all team members and ensuring that every step taken is towards a common end.

Validation Checks

The primary goal of the parser, revolves around two essential validation checks on a pre-selected set of example files:

1. Parsing Capability: The first check involves the parser’s ability to parse all the example files. Successfully parsing these files means that the parser can construct an Abstract Syntax Tree (AST) for each example. This is a fundamental requirement, as the AST is a critical component in understanding and manipulating the structure of the source code.
2. AST Validation: The second check focuses on validating the AST built for each language construct within the example files. This step is crucial to ensure that the ASTs are not just formed but are also correct and accurately represent the intended structure of the parsed language.

In the development phase it is crucial to collect a significant amount of examples for a given programming language. Github is a good starting point to search if you don’t have licensing concerns. Google hosts a selected public dataset Google BigQuery4, which contains more than 2.8million open source GitHub repositories. For example, you can extract the available Java files running the following query. The LIMIT 1 enables you to limit the number of extracted files as Google charges for the BigQuery usages.

The number is crucial, it is safe to start with a reasonable number and then add more examples to detect edge cases.

At Strumenta, we have developed tools and plugins integrated with the IDE that assist and streamline the process.

These tools allow monitoring the progress in the parser development, providing information about the Parsing Capability (number of files successfully parsed) and AST Validation (language construct within the example files). The process is not linear, in our experience the most of the effort goes into the parser development and tends to slow down when most of the remaining constructs are edge cases.

Completion and Delivery

Once the parser passes these two checks, it can be considered complete for its initial version. This milestone marks a significant achievement in the development process, allowing for the delivery of the first version of the parser.

Maintenance and ExpansionThe maintenance phase begins post-delivery, where the parser goes continual refinement and improvement. During this phase, additional files can be added to the validation set, effectively setting new goals for the parser. This continuous improvement cycle ensures that the parser remains effective and up-to-date with evolving language specifications and use cases.

AdoptionA parser’s true value is realized only when it is effectively integrated and operational within a system, this phase is dedicated to ensuring that the transition is smooth, efficient, and avoids unnecessary complications.

The Chisel Method provides three key features to achieve this goal:

1. Providing Good ConnectivityThe use of StarLasu open-source libraries ensures that the parser offers robust and user-friendly APIs for native integration. These APIs are crucial for allowing the parser to communicate effectively with other components in ensuring data is passed and processed efficiently.

StarLasu supports the following programming languages:

• Kolasu, for the implementation on the JVM (and in particular with Kotlin and Java)
• Tylasu, for the implementation on Node.js and on the browser, using Typescript or Javascript
• Pylasu, for the implementation with Python
• Sharplasu, for the implementation with C#

StarLasu supports the Adapter architecture for cross-language integration which enables the parser interaction between components written in different programming languages.

The adapters are not an alternative to the native libraries, they enable reuse of existing components such as a parser written in Java in the development of a transpiler in Python.

The serialization/deserialization available in StarLasu makes it possible to exchange the ASTs between components written in different languages organized as a Language Engineering Pipeline.

A typical example is the Language Engineering Pipeline which is a structured approach to language processing and translation. This approach is rooted in Model Driven Development, where models (in this case the Abstract Syntax Trees) play a crucial role.

In this architecture, the output of a component is used as input for the next component in the pipeline.

For example, an RPG to Java transpiler would be composed by the following components:

• RPG Parser: Transforms RPG code into a Plain RPG AST. This step is foundational, as it converts the source code into a tree structure that represents its syntax.
• Semantic Enricher: Enriches the Plain RPG AST by resolving symbols and calculating types, resulting in an Enriched RPG AST. This enrichment adds semantic context to the syntax tree, making it more meaningful for subsequent transformation.
• AST Transformer: Converts the Enriched RPG AST into a Java AST. This transformation is the core of the transpilation process, mapping the constructs of one language to another.
• Java Code Generator: Takes the Java AST and generates executable Java code. This final step turns the abstract tree structure into concrete, runnable code in the target language.

The diagram below provides a graphical representation of the pipeline:

(In this example, it is important to notice that every component could be written in any of the languages supported by StarLasu, for example the RPGParser and the Semantic Enricher could be in Java due to the performance and the AST Transformer and code generation could be Python.)

The Language Engineering Pipeline architecture can be applied in many use cases such as interpreters or code editors.

2. Providing DocumentationThe use of an internal documentation tool automates the generation of comprehensive, clear, and up-to-date documentation. This documentation is essential for users and developers to understand how to integrate and utilize the parser effectively. Good documentation reduces the learning curve and speeds up the integration process.

3. Making the Method TeachableEnsuring the method is teachable is key to its adoption. By training both the internal team and potentially the client’s team, the Chisel method establishes a knowledgeable base of users who can maintain and evolve the parser.

Automation plays a crucial role in simplifying the learning process. By automating tasks that are prone to human error, such as AST validation and parsing capability checks, Chisel reduces the likelihood of common mistakes that can be discouraging for learners. This automation not only speeds up the development process but also allows learners to focus on understanding the core concepts and methodologies, rather than getting bogged down in tedious, error-prone details.

Furthermore, Chisel’s systematic approach, which breaks down the development process into clearly defined stages, provides a structured learning path. This structured approach makes it easier to teach and learn, as it organizes the process into manageable segments, each with specific objectives and outcomes. Learners can focus on one aspect of the process at a time, building their knowledge and skills incrementally.

This approach significantly reduces the risk for the client, as they are not solely reliant on external support for maintaining the system.

ConclusionThe Chisel Method attempts to innovate in the realm of software engineering, in the specialized area of parser and transpiler development. Chisel offers a structured, pragmatic approach that prioritizes clear objectives, measurable goals, and practical solutions.

Its focus on parsing capabilities, AST validation, and the use of modern tools and resources marks a significant departure from traditional methods, positioning it as a versatile and efficient method for developers.

The Chisel Method not only simplifies the parser development process but also ensures adaptability and sustainability in a field characterized by continual evolution, proposing a more streamlined and effective approach for developers and engineers in Language Engineering.

More resourcesBuilding Advanced Parsers in Kolasu

The post Chisel: an open-source method for parsers appeared first on Strumenta.

View Details

Markus Vöelter interviewed Federico Tomassetti, one of the people behind the LIonWeb Initiative.

An initiative to create an ecosystem of interoperable components for building language-oriented modeling tools on the web.

We believe that a lively ecosystem will facilitate the adoption of language engineering and modeling solutions by reducing vendor lock-in. Development of advanced solutions will be accelerated by mixing and matching components, potentially sourced from different vendors or open-source initiatives. It will also foster innovation, as incremental improvements can be defined on top of the existing libraries and components.

Markus Vöelter

Hello Welcome everybody. This is another interview about LionWeb or with LionWeb participants or members about their involvement and why they are involved. My name is Markus Vöelter; I was the interviewee last time this time I’m the interviewer, because Federico is the interviewee, and since he can’t interview himself, I guess we needed to find somebody and that somebody is me. Backgrounds is a bit strange, because I’m still more or less in bed with my broken leg, but I have lots of time because I can’t do anything. So it’s a great opportunity to do the interview. Hi Federico.

Federico Tomassetti

Hi, Markus.

Markus Vöelter

So let me introduce you a little bit. Federico has been a longtime member of the language engineering and MPS communities starting, I guess with your work in 2011 at Fortiss on a mbeddr right?

Federico Tomassetti

Yes, for which, by the way, I should thank you for all the support that you gave me at the time.

Markus Vöelter

There was no point, but since then, you have founded Strumenta. I called it a boutique consulting and development company, right? Small but beautiful! Focused on all things language engineering. And you’re also one of the more regular conference speakers and you publish tutorials and articles and stuff, mostly on LinkedIn. And of course, in your website, and you do a lot of stuff in the open source community, which is a nice segue to LionWeb. So why did you decide to join LionWeb?

Federico Tomassetti

Well, I think, for me, the LangDev conference was a very important moment where I noticed there were a lot of very interesting ideas, like people working on typesystems, people working on interpreters, people working on code generators, on editors on the web. And I like a lot of these things. I mean, a lot of these solutions, add a nice bit, and not so nice rest. I really thought if I could take the best of everything it would be great; and so on a more practical level, I thought about that, on the other end, I remember about my time as a PhD student, and how frustrating it was that, you know, if you wanted to make any meaningful contribution, you had to do a lot of legwork to create some basic stuff, and I didn’t really have the time, so it was really impossible, right? So I was thinking if we have this common shared base on which people can plug in meaningful contributions that will be really a totally different world.

Markus Vöelter

Yeah, that’s something that I was also annoyed by, especially when I was still more involved in academic conferences, where every university that dealt with language engineering, built their own complete stack from the ground up, usually worse than what was already available. And then at the time, when they could add something like meaningful, then, you know, the PhD student left, so they just build another base stack instead of doing interesting stuff. So yeah, I agree the infrastructure should be commodified. Right. So is that the main way how you think that LionWeb will benefit the community acting as this kind of foundational layer on top of which to innovate?

Federico Tomassetti

Yes. I think it will lower the barrier for innovation but also permit to make more valuable solutions, because, you know, if I could combine the best interpreter with the best code generator with a great web editor, and so on, I can put together solutions that, at this stage, no one can put together also considering that most actors are relatively small, and there have maybe the resources to do one thing decently, but not a lot of things.

Markus Vöelter

Yeah. Obviously, totally agree. So in addition to working with MPS, and all its projectional magic, you also work a lot with parser based tools in the context mostly I guess, of legacy modernization and for building kind of smallish utility style DSLs. How do you see LionWeb helping there?

Federico Tomassetti

Well, there are different ways in which I think LionWeb can help what but first of all, when when we perform legacy modernization, typically we want to use some parser and combine it with some module for model- to-model transformation and some code generators. These same modules could be useful also, for other projects, for example, a DSL could want to generate Java or Kotlin to be later executed. In some other cases, we also use interpreter for implementing legacy modernization, for example, you need an interpreter for RPG. So this module could be reusing in that context. And then I forgot what else I wanted to say.

Markus Vöelter

So, but you would basically use the LionWeb meta-meta-model and meta-models based to build on top of that, to represent the result of the parsing, for example, RPG. And then assuming there is lots of LionWeb based transformers and generators and stuff, then, of course, from there on, it is much less work.

Federico Tomassetti

Yes, and I remember now, what I wanted to say that we can also have a lot of facilities tha can that can work on all sorts of modules, like models repositories, but also tools to perform analysis on large models. For example, in the context of legacy modernization, it’s interesting to be able to pass a large application with all the ASTs in a model repository, and then run some statistics to find which statements are more frequently used or verify if there are certain patterns, so group of statements used together in a certain way. And these kinds of facilities can be used for all sorts of languages. So would it make sense to have them built by someone and reused by everyone else.

Markus Vöelter

Yeah. So have you already integrated some of your parsing and text language transformation, utility things that you built at Strumenta with LionWeb?

Federico Tomassetti

Yes, in two ways. First way, I tried something more experimental. So I tried to make all of our system extend LionWeb itself so that they were natively supporting LionWeb. But then I thought that maybe we will still evolve a little bit LionWeb in the coming months. So I added an other mechanism to have compatibility with LionWeb that is an export and import facility so that we can use it with the current version of our frameworks.

Markus Vöelter

And by the way, you can definitely mention the name of your frameworks. I didn’t because I couldn’t remember but lasu, StarLasu, PyLasu.

Federico Tomassetti

The name is terrible. But we originally started with a one in Kotlin. So created KoLasu for KOtlin LAnguage SUpport. But then we needed one for Python, one for TypeScript, and one for C#. In the end, we call them StarLasu. So like, in the sense of an *Lasu, all sorts of Lasu. Yeah, I probably should not be authorized to give names to things.

Markus Vöelter

You can ask me for name suggestions, I came up with a great names like mbeddr, and IETS3. So I’m great at naming. Okay, this is a joke, for the non involved people. So what are you working on? What are you working on at the moment in the context of LionWeb? What’s your personal, most relevant or interesting aspect?

Federico Tomassetti

I don’t know if it’s interesting but the main I am working on is the implementation of the boring layer that we need for Java. And as we said before, there is a lot of boring legwork that all people need to replicate every time they start a modeling solution. So we’re trying to write these boring things here has a library that people can reuse. And so that is what I’m focusing on the most. Other things that I and my colleague, Alessio will work in the future could be the support for Python, for example; and maybe a toy model server that we use just to store models, to then perform statistics without versioning, without support for multiple user at the same time.

Markus Vöelter

Yeah, because your use case is basically import from Legacy system, do analysis, throw it away, import another one, so you don’t have these repositories, the core lifecycle supporting thing; it’s just a temporary storage for the past result.

Federico Tomassetti

Exactly. Yeah. So we started using MongoDB for now, we then started experimenting with Neo4J; and we save stuff do analysis, or attach some web application to visualize something, and that’s it.

Markus Vöelter

By the way, dear audience member who is watching this, if you notice, or wonder why these questions that we’re asking each other are kind of repetitive and always the same. That’s intended, right? The idea is to basically have the same framework for these short conversations and get everybody’s perspective on these questions. So continuing in that in that vein, what do you think we need to make LionWeb successful?

Federico Tomassetti

That’s a good question.

Markus Vöelter

You came up with it?

Federico Tomassetti

Okay. No, I think that we need more people on board and using this thing, because it’s an ecosystem. So it’s valuable only if people are on it; is like, the network effect, like, Facebook: if everyone is on Facebook, then it’s valuable, otherwise, it’s not. So I think that having a couple of good tools compatible with LionWeb could be a good start. And then maybe collaboration with universities, because if a bunch of master’s students, PhD students start using it and maybe find some bugs, or maybe write a tutorial, or maybe showcase some interesting solution, then I think they can help us refine the solution, but also, you know, having people start using it and providing feedback.

Markus Vöelter

Yeah. Obviously, you already had at least one good idea already regarding how we can quote advertise, which is these videos made this where your idea, right, but we have any other ideas, what we can do to kind of broadcast the message further to get more people to join?

Federico Tomassetti

Well, besides advertising this at that next LangDev and maybe reach out to people in the EMF community, that because in a way, I see LionWeb as sort of a new version of EMF built 20 years later with support for the web, and we support four different languages. So not just Java. So these are the main differentiators, but if people are willing to consider moving from EMF to LionWeb, or use them together, because there is an import and export facility, also, we should test it if it works, but there is one; so maybe that could be one thing. And for the rest, probably reach out to some professors and have them join. Right. And perhaps write an article for modeling-languages.com.

Markus Vöelter

Yeah, that’s one thing we talked about last time, specifically, right? And then we could probably actually talk to, we both have kind of friends and acquaintances in the academic world. Maybe 20, kind of usual suspects, kind of language engineering professors around mostly Europe, we could systematically ping them and make them aware of this thing. So that would probably also be a concrete step. Cool. Anything else you want to add?

Federico Tomassetti

No, just that, I think this is has potential to be quite exciting, because could maybe lower the entry barrier for a lot of people, putting our energies together, so I have big hopes for LionWeb. So let’s hope the rest of the world agrees.

Markus Vöelter

Yes, I’m gonna join you in that hope. So that’s a hopeful message to the end, towards the end of this interview as always good. All right, Federico, thank you very much for being a guest here. All right.

Federico Tomassetti

Thank you.

The post Interview with Federico Tomassetti appeared first on Strumenta.

View Details

It all started after attending LangDev 2022. We were watching a lot of great presentations about language engineering and realized that none of those tools were compatible. There were great editors of different kinds, model repositories, parsers, code generators, interpreters, symbol solvers and much more. Each one of them was incompatible, with the other tool […]

The post LionWeb: Building great Language Engineering applications appeared first on Strumenta.

View Details

We interviewed Markus Voelter, one of the people behind the LIonWeb Initiative. An initiative to create an ecosystem of interoperable components for building language-oriented modeling tools on the web. We believe that a lively ecosystem will facilitate the adoption of language engineering and modeling solutions by reducing vendor lock-in. Development of advanced solutions will be accelerated […]

The post LIonWeb Initiative – Interview with Markus Voelter appeared first on Strumenta.

View Details

We interviewed Jos Warmer, who is taking part in the LIonWeb Initiative. An initiative to create an ecosystem of interoperable components for building language-oriented modeling tools on the web. We believe that a lively ecosystem will facilitate the adoption of language engineering and modeling solutions by reducing vendor lock-in. Development of advanced solutions will be […]

The post LIonWeb Initiative – Interview with Jos Warmer appeared first on Strumenta.

View Details

Introduction In this article, we will show you how to use Sharplasu for building advanced parsers for the Python language. We choose Python 3 as a language to be parsed for the developers to see a real-world example they are familiar with – involving imports, function definitions, and parameters –, and for them to understand […]

The post Building advanced parsers using Sharplasu appeared first on Strumenta.

View Details

OverviewThis article will provide insight into the Strumenta approach in migrating an ERP system from IBM RPG to Python based on our experience with a real project in the manufacturing industry.

Migrating a legacy system from an outdated language to a modern programming language can be a daunting task, but it can also be a necessary step for companies to stay up to date with the latest technologies and to improve their overall efficiency.

RPG is a programming language that has been around since the 1960s and has been widely used in the manufacturing and distribution industries.

However, it is becoming harder to find experienced developers, and the language suffers from the lack of some of the modern features and functionalities that are available in newer programming languages like Python, which has gained popularity in recent years due to its ease of use, readability, and versatility.

What is an ERP System?An ERP (Enterprise Resource Planning) system is a software application that integrates and manages a range of business processes such as finance, human resources, procurement, inventory management, customer relationship management, and more. It provides a single platform that streamlines these processes and allows information to be available between different departments in an organization.

The architecture of an ERP system usually consists of a central database that stores all relevant information about the organization’s operations. This includes customer data, inventory levels, suppliers information, financial data, and so forth. Users can access this information in real-time and use it to make better decisions that drive productivity and efficiency.

Typically the system has several modules (or applications), each designed to handle specific business processes. There may be a finance module that handles accounting and financial reporting, a supply chain module that tracks inventory levels and manages supplier relationships and so forth.

The system allows to collect data from users or other systems such as an e-commerce website, which is stored in the central database. This data is then processed and analyzed by the system, which generates reports and provides information that help users make decisions. The system can also automates many routine tasks, such as data entry, invoicing, and order processing, reducing the workload and increasing productivity.

Many organizations have developed custom programs over many years that are used to analyze data and automate tasks, representing the business logic of the company. These processes are highly efficient in handling a wide range of business use cases specific to the company’s business and structure.

These programs represent a valuable asset for the organization, and it is critical to ensure that translation in another language will preserve the same exact functionality.

The MigrationOur migration process starts with an analysis phase organized in workshops with the customer’s developers and business analysts. The migration process involves several steps, the objective is to identify how the available functionalities need to be migrated, the scope, the complexity and the critical part of the system.

The analysis of the current system typically includes the following steps:

1. Identify the key functionalities The first step is to identify the key functionalities of the current ERP system. This involves understanding the business processes that the system supports, and the features and capabilities that are required to support those processes. This information is collected during the workshops with developers and end-users and by reviewing the code, system and business requirements.

2. Create an inventory Once the key functionalities have been identified, the next step is to create an inventory of the current system. This includes all the programs, databases, and interfaces that are currently in use. The inventory includes details such as the purpose of each program, the data it processes, and the interfaces it uses.

3. Evaluate the current system With the inventory in hand, the next step is to evaluate the current system. This involves reviewing the current code base, identifying any inefficiencies, code pattern, areas for improvement, and determining how well the current system will be translated in the target language. At Strumenta we make use of custom tools based on our parsers to process the source code. For example, to analyze the complexity and the dependencies of an RPGLE program, we make use of an internal tool called RPG2SEQ which generates a sequence diagram representing the code execution, the files it accesses, and the external dependencies. The diagram below is an example of such a diagram.

From the diagram it is possible to extract useful information such as the complexity of the program, the files involved (FILE1,FILE2,FILE3,FILE4), the operation of the files (READ,UPDATE), the invocation of subroutines ($Subroutine1,$Subroutine2,etc) and an external program (PROGRAM2.rpgle). In this particular project we have also to migrate the existing DDS (Data Definition) into SQL Tables. We developed a DDS2SQL transpiler to convert the Logical and Physical files into SQL statements to create the tables and the views required.

Our RPGLE parser is able to parse the DDS and create the AST representation of the data, the transpiler implements a AST transformation to translate the DDS AST into a SQL AST and a code generator walks the SQL AST to produce the output code.

A physical file in RPGLE corresponds to the SQL CREATE TABLE statement that has the same fields and attributes as the physical file. Here’s an example SQL statement that creates a new SQL table based on a physical file MYFILE.

| DDS | SQL | | A R MYFILEA FIELD1 10AA FIELD2 5S 0A FIELD3 D | CREATE TABLE MYFILE ( FIELD1 CHAR(10), FIELD2 NUMERIC(5,0), FIELD3 DATE) |

The Strumenta’s DDS2SQL transpiler automates this job resolving all the references to other files and creating a single DDL file in several SQL dialects, which is a very effective solution when there are hundreds of files that must be processed.

In addition, we make use of our RPG parser to perform static code analysis, to ensure that the codebase is well-structured, efficient, and secure, and that the migration process goes smoothly. These tools can help to identify potential issues early in the process, reducing the risk of bugs or other problems down the line, and ensuring that the new system is of high quality and meets the needs of users.

Code statistics refer to the use of statistical methods and analysis to evaluate the codebase of a software project. It involves collecting data on various aspects of the code, such as the number of lines of code, the number of functions, the number of files, and other code metrics, and then analyzing this data to gain insights into the quality and maintainability of the code.

Code statistics provide a wealth of information about the codebase, including:

1. Code complexity: Code statistics can be used to measure code complexity which is a measure of how difficult the code is to understand, translate, and modify.
2. Code quality: Code statistics can be used to measure the quality of the code, such as the number of bugs or defects in the codebase. This information can be used to identify areas of the code that need improvement and to prioritize bug fixing efforts.
3. Code maintainability: Code statistics can be used to measure the codebase’s maintainability, such as the number of comments in the code, the ratio of code to comments, and the average length of functions or methods. These metrics can provide insights into how easy it is to maintain and modify the code.
4. Code patterns: Strumenta’s code statistics tools allow to identify patterns present in the code and perform refactorings to eliminate code duplication.

By analyzing code statistics with the customer during workshops, language engineers, developers and project managers can gain insights into the complexity of the codebase, identify

the areas of the code that need improvement, and prioritize transpilation efforts. This information can be used to ensure that the codebase can be translated into the target language.

The table is below an example of statistical analysis.

| Statement | Percentage | CumulativePercentage | | Calculation | 37.94% | 37.94% | | File Operation | 22.69% | 60.63% | | If | 22.46% | 83.09% |

This example aggregates the RPG statements in categories to evaluate what statements are the most frequent and how much are they used in the code base. In this case translating the three statement categories will cover about 83% of the total code base.

1. Identify potential issues Based on the evaluation of the current system, potential issues can be identified. These can include outdated technologies, inefficient code, unsupported hardware or software, and security vulnerabilities. In this particular RPGLE to Python project we identified cases in which the code referred to platform specific system information of the IBM RPGLE runtime environment. That information is not available in the Python runtime, at least not in the same way, in conjunction with the customer we designed the solution of how the code should be translated in these particular cases.

2. Develop a migration plan Based on the inventory, evaluation, and identification of potential issues, a migration plan can be developed. The plan outlines the steps required to migrate the system from IBM RPG to Python, including timelines, budget, and resources required. It should also identify any risks or issues that may arise during the migration process, and outline strategies for mitigating those risks. All information is collected into the report that Strumenta delivers to the customer at the end of the analysis phase, which is the migration plan.

3. Code Translation Migrating RPGLE code to Python can be a challenging task for developers, especially when dealing with complex business logic and database integration. Transpilers are software tools that can convert RPGLE code into Python code by analyzing the syntax and semantics of the original code and translating it into Python. These tools can save a lot of time and effort for developers who would otherwise need to manually rewrite the code, and they can also help to reduce the risk of errors and inconsistencies in the migration process.

One of the benefits of using automatic migration tools is that they can preserve the functionality and business logic of the original RPGLE code, while also taking advantage of the modern features and libraries available in Python. For example, Python has a rich set of data analysis and visualization libraries that can be used to process and present data in ways that were not possible in RPGLE.

Code that relies heavily on platform-specific features or external dependencies may require significant modifications or manual rewrites. Additionally, the quality and readability of the resulting Python code may vary depending on the complexity of the original RPGLE code and the quality of the transpiler.

1. TestingTesting is an essential part of any software development process, it helps to ensure that the new system functions as intended, is free from bugs and errors. There are several types of testing that can be conducted during the migration process, including unit testing, end-to end testing, system testing.

Unit testing involves testing individual units or components of the system, such as functions or modules, to ensure that they function as intended. This type of testing is typically automated and can be conducted using testing frameworks like PyTest or unittest.

End-to-end testing involves testing the interaction between different components of the system, such as different modules or databases, to ensure that they work together as intended. This type of testing is typically conducted manually and may involve creating test cases that simulate real-world scenarios.

System testing involves testing the system as a whole to ensure that it meets the requirements and specifications of the project. This type of testing can involve both automated and manual testing and may include stress testing, security testing, and performance testing.

During the testing process, it is important to document any bugs or errors that are discovered and to track them through to resolution. This helps to ensure that the system is fully tested and provides the same functionality of the original RPGLE code.

The Strumenta’s RPG to Python transpiler can automatically generate the skeleton for the end- to-end testing. The diagram below illustrates the sequence of the execution steps.

The test case loads the database with the test data and verifies the precondition before running. A common precondition for all tests is to verify if the database has been loaded successfully and data is available for processing. The next step is to execute the program under test, which updates the database. At the end of the execution stage, the control goes back to the Test case that verifies the postconditions. The postconditions can include performance requirements, in that case the execution time of the program under test is recorded and compared to the performance requirement.

Challenges in Real World ProjectsAlthough the process takes in consideration many of the common challenges, every project is unique and unexpected cases must be properly managed.

For example one of the steps the analysis performs on the RPG code is the identification of unstructured code such as GOTO statements. In this particular project the RPG developers have been very disciplined and we found very few cases. The Strumenta’s RPGLE to Python transpiler can automatically identify code patterns involving GOTO statements.

The example below is a common case handled automatically by the transpiler.

| $LABEL: Read File; If …; GOTO $LABEL; Endif; If …; GOTO $LABEL; Endif; … | while True: file.read() if … : continue If … : continue |

The RPG code is a continue-goto pattern which means that there is a single label at the beginning to jump in certain conditions.

In this case, the goto statements are replaced by the transpiler with a Python while loop and a continue statements.

Although the Strumenta Transpiler can automatically eliminate GOTO statements the customer team decided to eliminate the statements from the original RPG code.

As mentioned earlier another challenge was how to implement the platform specific system information used by the programs. In real word projects it depends on the target frameworks and deployment environment, and configuration chosen by the customer. In this case, the code referred to the program status data structure (PSDS) that makes system information available to an RPG program. On the target platform those information could be stored in environment variables and instruct the transpiler to generate the Python statements required to read the variable value at runtime; another option would be to store those values in a configuration file. There is not one right solution for this, it really depends on the target environment and customer preferences.

Another challenge identified during the code analysis was the use of the READP RPG statements (read previous) which allows reading a file backwards. As mentioned before files have been translated to SQL tables and the Python Database API Specification v2.0 does not include a method to read the previous record. As part of the RPG-runtime for Python classes to provide such functionality has been implemented using the pagination and implementing the transpiler to invoke the proper method when positioning and reading the database table.

Conclusion This article highlights the challenges companies face when migrating a legacy system from an outdated programming language to a modern one. It emphasizes the benefits of such a migration, including staying up-to-date with the latest technologies and improving overall efficiency. The article specifically focuses on Strumenta’s approach and tools to migrate an ERP system from IBM RPG to Python, based on a real project in the manufacturing industry.

The post Migrating an ERP system from IBM RPG to Python appeared first on Strumenta.

View Details

In this article, we will reflect on how to choose the target for a migration from RPG. RPG is a programming language that has been successfully used for decades to create a lot of software. Nowadays there are a large number of very large and valuable applications written in RPG, and their owners are wondering: should we move to another language?

There are several good reasons to consider this: in particular the lack of RPG developers and the interest in moving to the cloud. Yet, this is not an easy choice to make, so in this article, we’ll study the problem from different perspectives and share our findings. Hopefully, this will give concrete arguments and guidance to the many considering how to keep their software valuable for the coming decades.

Why migrate from RPG to another programming language?We have already listed in the introduction the two more common reasons for considering migrating from RPG to another language: the lack of RPG developers and the interest in moving to the cloud. Let’s look at these two reasons individually.

The lack of RPG developers. If finding developers is always difficult, finding developers who know RPG is extremely difficult, as everyone trying to hire them would have noticed. There are way fewer RPG developers than, let’s say Javascript or Python developers, but they also tend to be older. While experience is a great quality for developers, this makes us think that 5 years from now some of them will have retired, and finding RPG developers will be even harder. To get some data, according to the TIOBE index, RPG is the 40th most popular language. Looking for RPG developers on LinkedIn we can find 73,000 of them in the entire world. For comparison, we can find multiple millions of Python developers or JavaScript developers. In other words, RPG developers are over 100 times rarer than Python or JavaScript developers, and the trend seems to be against RPG, with the average age of RPG developers being high.

RPG does not run on the cloud. If we look at AWS, Azure, and Google Cloud, none of them supports natively RPG applications. This means that RPG code, as-is, cannot be easily run on the cloud. In other words, by sticking to RPG one has to own, operate and maintain their own hardware. Scalability is more difficult as it requires buying individual IBM i series machines. The problem is when you would need one machine and a half, or when you would need a few more machines but just for a short period of time. Without the cloud, you do not have access to that flexibility.

There are additional reasons for wanting to migrate from RPG, like the uncertainty about the future of the IBM i Platform, but we think it makes sense to focus on these two.

How to migrate your codebase? Manual vs Automatic MigrationMost companies think that migrating from RPG can be done only by rewriting the code into another language, line by line. For large applications, this is simply unsustainable, and therefore many companies do not even look into migrating to another language.

We think more companies should be aware of the fact that it is instead possible to perform automatic or semi-automatic migrations to translate the business logic expressed in RPG into another language. This removes the possibility of human errors and greatly reduces the effort needed, bringing down the time necessary for the migration and the costs.

Automatic migrations can be performed by using a transpiler. We know about this topic as we have written several transpilers, including transpilers from RPG to other programming languages. If you want to learn more about how we design transpilers you can find more in this article: How to write a transpiler.

What is written in RPG?RPG has been used to write mostly ERP or enterprise software. According to our study, two-thirds of the applications written in RPG belong to this category.

This is something to take into account when considering the language to migrate to: it should be a language well suited for writing ERP software.

RPG is also commonly used to write financial software, with roughly 30% of RPG applications falling in this category.

What characteristics should a candidate programming language have?

When picking a good candidate language for migrating away from RPG we should, first of all, consider a language that allows us to solve the two main challenges we have with RPG: lack of developers and lack of support for deployment in the cloud.

Let’s see which languages we can consider based on these two factors.

PopularityIf the reason for migrating from RPG is the lack of developers, we should consider only languages that have at least 10 times the number of RPG developers and languages which are ideally gaining popularity, so that the trend indicates that we will be able to find developers familiar with that language in the midterm.

To find popular programming languages we proceeded in this way:

1. We first of all compiled a list of candidates. To do this we looked at several resources, including the TIOBE index.
2. We then excluded the languages which were clearly not suited for the task as they were not general-purpose languages (for example, we excluded SQL, HTML, CSS).
3. Finally we estimated how many developers were familiar with that language. To do that we used various sources: from LinkedIn to surveys and articles. Assessing the number of developers who are familiar with a certain programming language is very difficult, but for our needs, it is sufficient to get the right order of magnitude, and that is much easier.

The result of this process is the following table:

| Millions of developers | At least 10 times more developers than RPG? | | Python | 6.0-6.6 | Yes | | C | 2.4-2.5 | Yes | | Java | 4.1-6.0 | Yes | | C++ | 2.8-3.1 | Yes | | C# | 2.9-3.5 | Yes | | Visual Basic | 0.5-0.6 | No | | Javascript | 7.7-8.5 | Yes | | Go | 1.4-2.4 | Yes | | Ruby | 0.6-0.8 | No | | Rust | 1.0-1.1 | No | | Swift | 0.6-0.9 | No | | RPG | 0.06 – 0.08 | No :D |

Ability to run on the cloudThe second aspect to consider is which languages can “run in the cloud”. To do this we looked into which languages are supported by the three main cloud providers: AWS, Azure, and Google Cloud. We then picked the languages which are supported by at least two cloud providers, as this gives us some reassurance that at least one major provider may offer support in the mid-term period. I.e., even if one provider removed support, we will still be left with at least another provider.

This is the result we got:

| AWS | Azure | Google Cloud | Supported by at least 2 cloud providers | | Python | Yes | Yes | Yes | Yes | | C | Yes | No | No | No | | Java | Yes | Yes | Yes | Yes | | C++ | Yes | No | No | No | | C# | Yes | Yes | No | Yes | | Visual Basic | Yes | Yes | No | Yes | | Javascript | Yes | Yes | Yes | Yes | | Go | Yes | No | Yes | Yes | | Kotlin | Yes | Yes | Yes | Yes | | Ruby | Yes | No | Yes | Yes | | Rust | Yes | No | No | No | | Swift | Yes | No | No | No |

Which candidates are worth considering as a replacement for RPG?Based on what we have seen, let’s see which languages are both popular and have sustainable support for the major cloud providers.

Here there are our results:

| Popular | Cloud support | Worth considering? | | Python | Yes | Yes | Yes | | C | Yes | No | No | | Java | Yes | Yes | Yes | | C++ | Yes | No | No | | C# | Yes | Yes | Yes | | Visual Basic | No | Yes | No | | Javascript | Yes | Yes | Yes | | Go | Yes | Yes | Yes | | Kotlin | No | Yes | No | | Ruby | No | Yes | No | | Rust | No | No | No | | Swift | No | No | No | | RPG | No | No | No :) |

So these are the languages that we believe it makes sense to consider when migrating away from RPG:

• Python
• Java
• C#
• Javascript
• Go

In the next section we will discuss each of these languages individually.

How to pick the right target language?We have seen that there are 5 general-purpose programming languages that are used by millions of developers and supported by cloud services providers. This however does not tell us anything about them being a good fit for replacing specifically RPG. This is what we are going to dive into in this section.

PythonThe Python language has a vibrant community, and, despite being over 30 years old, it is growing in popularity.

What are some of its strong points?

First of all, it is reasonably easy to learn. We think this in particular a good factor for it to be used as a replacement for RPG. The reason is that many RPG developers can be considered good analysts who also double as developers. This is to say that their domain knowledge, that is, their ability to understand requirements and provide business solutions, is often more important than their knowledge of the latest programming paradigm. For this kind of user, Python would be a great fit. We expect people who are today defining business logic for their ERP written in RPG to prefer Python with respect to the alternatives in this list.

A second advantage of Python is the rapid development and good productivity that comes from it. This is particularly useful when working in close contact with clients and having the need to prototype solutions quickly to quickly obtain feedback from the clients. Again, a great fit for developers writing business logic inside ERP systems.

As a testament to the good fit of Python for writing ERP software, we can point to existing ERP platforms being written in Python, notably Odoo.

A weakness of Python is instead absolute performance. In general, programs written in Python are not as efficient as programs written in Java or C#. That said, this problem can be circumvented through the adoption of appropriate C extensions, and in this way, libraries like Numpy allow us to obtain good performance also in Python.

JavaJava is a very stable language, created in 1995, and used by millions of developers. It has a very large ecosystem: there are many different great IDEs for Java, and an incredible amount of open-source libraries and frameworks. All of these characteristics make Java a very solid and reliable choice.

Java has arguably been the primary language for writing enterprise applications in the last decades. It is a language that helps prevent errors and offers good performance. Late innovation, like GraalVM and Quarkus, permits to make Java even faster.

In general, we would suggest moving from RPG to Java if you are writing applications in the finance domain, where catching errors and having good performance can be particularly important. In that field the need for rapid prototyping can not be as important, so the fact that Java requires more “ceremony” for writing applications compared to Python would not be a hurdle.

C#C# is arguably a very well-designed language, which has learned from Java and improved upon it. It has also been a language that evolved more rapidly than Java, until recently and the result is brilliant. C# has also a good ecosystem. It may not have as many libraries as Python, Java, or Javascript, but quality libraries and frameworks are available, and the ecosystem seems stable.

C# is not a language on the rise, but at this stage, there are definitely many young developers learning it.

Compared to other languages C# has traditionally been connected to one particular operative system: Windows. This has changed in the last few years.

A platform that demonstrates the ERP solutions which can be written with C# is Microsoft Dynamics 365 Business Central (formerly Microsoft Dynamics NAV).

Despite its advantages, probably in many circumstances, Java would be a better choice than C#, unless your organization is specifically oriented to the Windows OS, or other Microsoft solutions, or if you have already a large number of C# developers in-house. If that is your situation, you can move from RPG to C# confidently.

JavascriptJavascript is probably the most popular programming language in the world. Another huge advantage of Javascript is the incredible number of libraries available.

Another important advantage is that, if developers learn Javascript to write your business logic, they will be able to use the same language also to write web applications that may act as a frontend. For companies with very small development teams, this may be a big advantage, while it could be less relevant for larger organizations we may have different teams for writing the business logic and the frontend.

The main disadvantages of Javascript are:

1. The instability of the ecosystem: new building tools and frameworks emerge and are abandoned, and the best practices vary quickly. This is not an ideal environment for companies looking for stability. The companies that today are considering moving from RPG are typically favoring stability over being constantly on the edge of new innovations.
2. The language is not designed with solidity in mind, but more for quick prototyping and for building applications where errors have limited consequences. In other words, if a button does not display in the right place it is not a big deal, while if an invoice cannot be generated that would be an issue.
3. Javascript is not designed for parallel computations and large calculations. This would make it a terrible implementation language for large batch jobs currently being executed on the IBM i Series.

All in all, we believe that the disadvantages make Javascript not a good fit for replacing RPG. It could make sense to consider it for small organizations who would appreciate how easy it is to recruit developers (and possibly also developers looking for lower salaries) and to hire a few developers who could both write the business logic and the frontend.

GoGo is a language with growing popularity, and used by a large number of developers, especially in Asia.

Go is a system language, typically used to write efficient code with high parallelism. We did not find many examples of enterprise software being written using Go. Go is also a rather low-level programming language.

Go does not seem to have such a strong ecosystem as the other languages considered, and we cannot find a reason to recommend migrating from RPG to Go. We believe that, of the candidates we considered, Go would be the worst choice in most cases.

What to do next?If you are considering migrating from RPG to another language, we hope that the information presented was useful to you. We would love to hear your feedback.

At this stage, you may be able to make a choice and decide to migrate from RPG to Python or to Java. In that case, great! You may be interested in learning how we typically plan for language migrations. If that is the case you may want to read this article: Audit & Analysis for Language Migration Project.

If you have restricted the choice to a few candidates and you would like to discuss the alternatives with us, feel free to reach us and we will help you reach a decision.

SummaryIn this article we have seen to which language it makes sense to migrate our RPG codebases. We have considered the main reasons for migrating away from RPG (lack of developers and cloud support) and based on that we identified the 5 best candidates for replacing RPG: Python, Java, C#, Javascript, and Go.

We then looked at each of them and we can conclude that in most cases Java or Python would be the best choices. In some specific cases also C# could make sense, while Javascript and Go are typically not the best choice, unless there are very specific reasons.

The post How to choose the target for a migration from RPG appeared first on Strumenta.

View Details

When we are tasked with performing a Language Migration we start with planning it. In this article, we are going to show our process for Audit & Analysis for Language Migration Projects, describing each single step and explaining the reasons why we move in this way. Hopefully, you can benefit from our experience with transpiling from one programming language to another and avoid the scars we collected over the years. If you are reading this, you may be looking into your first or second language migration as soon you may not have a lot of experience with this specific problem. We are here to give you a shortcut to get that experience.

What goes into planning a Language Migration?Planning a Language Migration basically means understanding the existing codebase, defining how we want the new system to look like and then defining a plan to get there reliably.

Sounds simple, right?

We call this process Audit & Analysis, because we start with an audit of the existing codebase and we analyze the problem of converting the code, and producing a plan, described in a shiny report.

What is the goal of the Audit & Analysis?The goal of the Audit & Analysis is to take time to reflect on what we are aiming to do and anticipate as many problems as possible so that we can correct the course now, while we have just a report and not a few hundred classes to change.

Once you get the plan, obviously you can just feed the report to Chat-Gpt and you will get a fully working, reliable transpiler. Right? Or, for the old-fashioned, you can build your own transpiler or work with Language Engineering craftsmen like us.

Why start with the Audit & Analysis?As developers, we love coding. Oh boy, how difficult it is to resist jumping into the comfort of our IDE and start producing code, crunching method after method, class after class. One feels so productive; one feels the pleasure of crafting something, the excitement of seeing the first results.

And yet, even if it feels great, it is so very wrong to start coding without a plan when dealing with something as complex as a Language Migration.

You can think of moving a codebase from a language to another as similar to a heart transplant. Before you get started, you want to be really sure that the heart is compatible with the receiver, that the tools are clean and in the right place, that the receiver is sedated and so on. I think your patients would genuinely expect you to anticipate all possible problems and not just react to them as they appear. Shouldn’t we strive to be as professional? We think so because a failure in a language migration can have significant economic and organizational consequences for the people involved.

So we start by figuring out what we are dealing with, defining a plan we are confident in, and then, when we know where we are doing we can, finally, open the IDE and let the fun part begin.

How is the Audit & Analysis structured?The Audit & Analysis is composed by five steps:

• Audit: In this step, we understand the existing system: its architecture, its size, the built-in functions that are used, the statements that are used, and the pattern that appears more frequently and is idiomatic to this codebase.
• Target Architecture Definition: At this stage, we want to define the architecture of the target system. Which version of Python or Java are we going to use? Which frameworks? Generating code for Odoo or Flask, for Spring or Vaadin are very different scenarios. We may also want to pick a database, and perhaps define integrations with existing applications already written. Here we define what we would like to get in the end.
• Risk Analysis: We look for risks. What could go wrong? What are the features that are difficult to replicate on the target system and important for us? Does our application employ AS/400 activation groups in complex ways? Is the usage of logical files that we find complex to replicate? Or do we need to emulate the same arithmetic rounding? Are performance expected to remain comparable? Here we look at the problems and write down a mitigation strategy for each of them.
• Mapping Definition: Here we discuss how to map complex idioms or complex functionalities of the source language into the target language.
• Testing Strategy: We define a testing strategy. How will we verify that the system works as expected?

Let’s now take a look at each step individually.

Step 1: AuditThe first step in an Audit of an existing codebase is… to get the whole codebase. And that could be more difficult than you think. Sometimes files are spread in different directories and no one knows exactly where all of them are.

We solve this by parsing the files we get and verifying if we can find all the symbols they refer to. Does program A call program B and we cannot find program B among the source files we have? We add it to the list of the missing files. When the Client delivers them to us we process also those, and the dance repeats until we have all files.

To perform a proper audit you will need a parser for the language you are migrating away from. You could build it, or you could license one from us.

Once we have the entire codebase it is time to start with producing some statistics:

• How many lines of code we have (with or without whitespace and comments)
• How many files we have
• How many statements we have
• How many expressions we have
• What is the frequency of each statement
• What is the frequency of each expression
• What built-in functions are we calling and how many times

For example, we use an internal tool to perform this statistical analysis. In the screenshot below you can see that we show how many times each built-in function is used:

We then discuss with the owner of the system which are the most important entry points and the most important procedures. At this point, we take a look at the sequence diagrams that we generate for each procedure. So that we get a sense of how the application works.

In case of general purpose languages that support embedded SQL, we may also want to trace access to the tables.

Finally we look for recurring patterns, using a mix of experience and statistical analysis.

Things we are currently not doing but we plan to do in the future are:

• Identifying unreachable/dead code
• Identifying code clones

Below you can find a video a simple product we use internally to navigate the code:

While we built this tool specifically for this job you may instead use an IDE to perform these analysis. It will typically require more manual work, as the tool is not specifically designed for that, but it can be doable.

Step 2: Target Architecture DefinitionIf we go somewhere, we should first of all figure out where that somewhere is. In the case of a migration, we need to define how the new system is expected to look like before we can define a plan on how to generate code for that new system.

Among other things we need to understand:

• On which platform will the new system run? Will it run on the cloud? On the Intel architecture? On Apple Silicon? On which operating systems?
• What is the target language? Do we want to use Java? Or Python? Which versions? Do we want to use multiple languages? For example, is there a scripting part that we want to define in Ruby? Or maybe generating bash scripts? What about SQL?
• Are we going to interact with a database? Which one? SQL based? No-SQL?
• Do we need to integrate with certain services or APIs?
• Does the generated code need to follow certain guidelines in use at the company?

These are questions that the future owners and maintainers of the translated system should be able to answer.

If you are translating your own codebase, it means that you have to answer these questions. Or some Architect in your team has to.

In our case, we ask these questions to our clients, as they will need to use and maintain the code we are generating. While we can help in making some of these decisions it is really important that the company is involved in the discussion and they are convinced of every decision made at this stage.

Step 3: Risk AnalysisIt will not come as a revelation that it makes sense to spend some time trying to anticipate all problems that we expect to encounter in the migration process before jumping head first.

We typically start by asking the client what they have already identified as problems, as they typically contact us after they have been considering doing a migration for a long time and have reflected on the problem. If you are performing a migration in-house, that translates to talking with both the colleagues that are maintaining the existing system and the ones that are in charge of maintaining the new system once it is deployed. Now, in many cases, these are two sets of developers, with the old RPG, Cobol, SAS, or Visual Basic developers about to retire and be replaced by the enthusiastic Java, Python, or Typescript developers who are going to take over. Here we are lucky when there is some overlap between the two teams and at least one person is familiar with both the old language and environment and the new one, as they are aware of the differences among the two and they can more easily spot the problems. If there are no developers available who are familiar with both languages, then the language engineers will need to act as intermediaries between the two teams and familiarize themselves with both languages.

You may also want to speak with colleagues that know their domain very well as they can underline what are the critical parts of the system and the areas with are more fragile or more important to get right.

For us, another resource to identify risks is our experience with migration. Given we have years of experience translating between several languages, we have a pretty good list of things to consider.

If you are doing this on your own for the first time, then I am afraid you have a bit higher chance to run into issues you have not anticipated. I wish I could share the list of 7 tips to identify all potential risks in a language migration, but I am afraid there is no such list.

These are some examples of problems we could identify during the risk analysis:

• The original language has low-level constructs with no equivalent in the target language (someone said Goto or Pointers?)
• The arithmetic is implemented differently in the original system and in the target system. I.e., expressions could produce slightly different results
• The original system is using extensively features of a standard library or third-party libraries for which there is no equivalent in the target system
• Significant differences in user interfaces and user interaction patterns available

Once we have identified these problems we discuss them until a solution or a mitigation strategy emerges. Once we are confident we can move forward we start our journey. Difficulties can always arise in similar projects, but this work typically helps arriving prepared to face most of them.

Step 4: Mapping DefinitionNow that we know where we are (Audit), we know where we are going (Target Architecture), and we know the pitfalls we want to avoid (Risk Analysis), it is time to define a plan to get there. This is exactly what we should do as part of the Mapping Definition.

In this phase, we look at the different components of our system, and we define what we will map them in the new system.

For example, in RPG, we store data on physical and logical files, while in the target system, we typically want to use a database. A typical mapping that we may want to go for is to map physical files to tables and logical files to views.

We may also decide to map an RPG program to a Python class, or a procedure to a Python method.

In this phase, the mapping is defined at a rather coarse level: we are defining what each component will be translated, but we typically do not go deeper than the class or method level.

Step 5: Testing StrategyOne question that will always be in the air is: how can we be sure that the new system will behave as the old system?

This is an important question to answer. On one hand we know that, given the translation will be automatic, we may have systemic errors but not one-off errors that appear just on one occasion. Typically if things are translated in an incorrect way, they are translated incorrectly very consistently, and so we have many occasions to notice that.

We off course define unit tests for our transpiler, and you should do so if you are writing one. But that alone is not enough to reassure that the new system will behave correctly.

In the rare case in which there are tests for the original system, we will celebrate and then translate those tests. We will then ensure that the translated tests are all green and that will give us reasonable confidence, provided the tests have a certain amount of coverage.

In most cases however, there are no tests for the existing system, and in that situation, we consider writing end-to-end tests the best course of action. These are difficult to write, and typically, writing them requires understanding the functionality. For this reason, collaboration from the maintainers of the existing solution is very valuable. End-to-end tests are also expensive to run, and not just to write, but a bunch of them can give us confidence in the new system working correctly.

A typical end-to-end test is created in this way:

• In the original system, we set up the initial state of the system
• We run a program we want to test
• We capture the end state of the system
• We translate the initial state of the system and the final state of the system in the equivalent of the new system. For example, we may need to translate a file into data inserted in a SQL database
• We execute the translated program, passing the initial translated state, and we verify to obtain the end translated state.

We typically have clients defining the end-to-end tests. We give them guidelines and a template to fill out. In some cases, clients like to use Gherkin to define these tests. We do that because they know their infrastructure, and it is not always convenient for them to provide access to their data. In case you are performing the translation in-house, you may want to involve a separate team to test the results of the translation.

What next?Ok, you have now completed your Audit & Analysis. You have discussed this with all people involved, and everyone nodded, and the thing can be crossed off as done.

Now what?

Now it is time to move forward with the implementation. And for that, we have some advice defined here: ​​https://tomassetti.me/how-to-write-a-transpiler/

SummaryMigrating your codebase from one language to another is a complex process, especially if you have never done it before. So you can do your best to ensure that you create the best conditions possible to make this successful. Part of this is getting some preparatory work done before getting started. Our way to get this done is going through our Audit & Analysis process. We hope this process will help you too.

The post Audit & Analysis for Language Migration Projects appeared first on Strumenta.

View Details

IntroductionIn this article, we’ll present a new take on editors for domain-specific languages (DSLs), that combines features of traditional text editors and a more projectional approach: bridging the gap between textual and projectional editors.

We’ve already talked about this topic, once at the LangDev conference in 2022, and once during the Strumenta Community Virtual Meetup – the video of the latter talkis publicly available. The present article expands on those talks.

We’ll go over a recap of textual and projectional editors, and their differences, to lay out a map of the existing territory. Then, we’ll present our ideas to bridge the gap between the two and draw some conclusions.

While we usually include links to the accompanying code with a permissive license, this time we cannot do so because the results that we’ll present are early research work at Strumenta and are currently proprietary. Instead, we’ll focus on the high-level concepts rather than their implementation as code.

Text EditorsLet’s start with defining what a text editor is. Let’s keep in mind that we’re in the context of editing models, not any kind of document. In that context, by “text editor” we mean any library or application that presents and stores models as text. Examples include VSCode/Monaco, Emacs, Vim, and the editor in your favorite Integrated Development Environment (IDE).

A text editor in an IDEIn a text editor, the model could be code in a programming language, but also, for example, a piece of configuration, a SQL query, a Markdown document, a model in a domain-specific language – anything with a purely textual, formal representation.

Specifically, when we say text, we mean a linear sequence of characters. Actually, text editors present the model as a sequence of lines, possibly decorated e.g. with line numbers, markers and messages of various kinds, as illustrated in the image above. So, there is some difference between the presentation of the model and its internal, stored format.

Still, text editors store models as a flat sequence of characters, and most of the operations that they provide (such as moving the cursor, cutting and pasting, and so on) are based on characters.
These editors may provide more advanced operations, such as refactoring options (e.g. renaming a variable in every place where it’s used). To enable these features, the developers of the editor must program some “language intelligence” into it – an understanding of the language in which the model is written. And, to obtain such intelligence, the editor must use a parser; this is a component that analyzes the text and builds a rich model out of it, a tree, or a graph, known as an Abstract Syntax Tree (AST).

Strengths and Limitations of Text EditorsText editors are a technology that we’re very familiar with; both as users and as developers of these kinds of editors. We have mature libraries for building them and established applications that we can extend. We have well-known algorithms and ample literature on how to write text editors, and code editors in particular. We also have good parser generators such as ANTLR and libraries to build models that are easy to navigate and transform.

Surely a good part of the success of text editors is due to the fact that text is very versatile. Even without specific support, it is possible, even if probably not comfortable, to read and edit any model encoded as text. It’s also possible to apply generic text-based tools, such as grep (regular expression search) and sed (advanced search and replace). These tools don’t understand the syntax of the model, and thus cannot be as accurate and easy to use as an IDE’s “find references” and refactoring commands. Nevertheless, they can offer good results when more sophisticated alternatives are not available.

On the other hand, relying on text also has quite some limitations. Developers have devised syntaxes to express many different concepts as text. However, the need to encode information as text limits expressiveness. For example, it’s awkward to represent a table as text, and it’s especially hard to edit it. For some uses, though, tabular notation is ideal; for example, certain procedures are best explained with a decision table. Or, imagine encoding a graph as text. At best, it stops being practical once the graph grows bigger than a handful of nodes.

There are other drawbacks besides limited expressiveness. While we can use a parser to build a model that is easier to traverse and manipulate than text, parsing has a cost. Advanced IDEs offer powerful analysis and refactoring tools, but they consume quite a lot of resources to build and maintain their internal models from the source text.

Not just that; even the best parser will sometimes break on malformed code. Surely we’ve experienced a situation where our IDE has gotten confused, maybe after we pasted a good chunk of text. Inevitably, when writing in a text editor, our code is frequently malformed. That’s because we can only insert bits of text – single characters, or pasted snippets – that most often aren’t well-formed code. Even if when we stop typing the code will be perfectly valid, in the interim, it’s often not. Also, not every syntax is equally well-suited for analysis while typing. This is an issue especially with legacy languages, that weren’t designed with advanced IDEs in mind.

Finally, a textual editor can be a bit hard for novices to approach because it’s so free-form that it gives little guidance. You get a blank page, where you have to type something, and nothing tells you where to start. Of course, that’s mitigated by tutorials, online documentation, IDE wizards, and code generators that can automatically set up the skeleton of a code project from some initial description. But still, in a text editor we have limited room for guiding novices – and in general for interactivity.

Projectional EditorsOn the other side, we have projectional editors, also sometimes called structural editors. The most famous example is Jetbrains MPS, but we also have Freon (was ProjectIt) and others.

These editors store the model as structured data (e.g. as XML, JSON, or proprietary formats). Then, they present a projection of the model to users. So, when using a projectional editor, we don’t see the raw structure of the model (e.g. XML nodes with attributes and children); rather, we may edit a diagram, or a table, but often just a textual document resembling code or prose. Some editors actually go to great lengths to ensure that the experience is as close to editing text as possible when the projection is mostly textual.

However, no matter how text-like a projectional editor is made to be, the underlying paradigm is very different from a text editor. In fact, projectional editors don’t use parsers and cannot edit “generic” code they can’t understand. Every element on the screen – a word, a phrase, an icon, a table cell, … – matches an element in the model, or is a synthesis of several elements. There’s no parsing phase that reconstructs the model from the source code every time the user modifies it. The model is the source, and the editor is a tool to view and edit it.

As a consequence, projectional editors are able to provide operations that don’t work on sequences of characters, but rather modify the structure of the model directly. For example, replacing a node with another, where a node could be anything from a single expression to a whole function body or query clause. Visually, that node may correspond to a block of text, or to a row in a table, or a box in a diagram.

Strengths and Limitations of Projectional EditorsSo, projectional editors allow greater expressiveness because they’re not constrained by the limitations of text and parsers, and greater opportunities for interactivity because the available operations are not limited to text manipulation. Often they also allow combining different syntaxes together, so for example we could embed diagrams or tables, or other kinds of domain-specific representations, into a more straightforward textual syntax.

Also, the problem of having to work on a potentially broken model while the user is editing is vastly mitigated in projectional editors. It’s not completely eliminated; in part because to construct a valid model we may have to pass through an incomplete model, in part because parsing is a bit like email – something that likes to creep into any kind of software application. There will be maybe micro expressions or small pieces of text that are still easier to parse than to model as a projection. For example, we may prefer to treat a date, or a very simple arithmetic expression, as text, even if it would perhaps be more conceptually correct to represent it as a composite object. However, most of the time, no parsing is involved in editing a projection.

Furthermore, a more interactive experience means that we can also offer more guidance to novices. Common approaches include having a new editor start from some pre-populated nodes, rather than a blank page; and offering a simplified view of the model, that hides some of the complexity. Users may then toggle it to “expert mode” when they’re confident enough. It’s also easier to display contextual help, not limited to the inline documentation hovering over some code elements in popular IDEs. So, we can say that projectional editors in some aspects could be easier to learn.

With all of that in mind, of course, there are also drawbacks to projectional editors. For example, they’re unfamiliar. These kinds of editors have been in existence for quite a long time, and there’s literature on them. However, outside academia and specialized niches, projectional editors are still pretty much unknown, and for many users, the learning curve could be quite steep, especially if we didn’t spend many resources on improving the learning experience as we’ve mentioned before. Text is simple, in part because of its characteristics, and in part, because we’re so well accustomed to working with text. It’s considered an essential aspect of using computers, that we learn early on.

Not just that, projectional editors are unfamiliar to us developers, too, should we want to build one. Not “us” at Strumenta of course, because we’re specialized in that, but the average developer will probably find it easier to build a VSCode extension than an MPS language. For example, the libraries and tools that we have at our disposal are still somewhat less mature than what we have for text. Often for these editors, MPS included, the text editing experience out of the box is not great; we’ve got to do some development work to bring it more or less on par with the standard experience of a text editor. Without any development effort, an editor made with MPS won’t allow you to insert anything just by typing; instead, it will require that the user chooses what to insert from a menu. Only after having inserted a node, MPS will allow us to “fill in the blanks” by typing, for example, to give a name to our new class, function, or database table.

Finally, although work is being done in that area, the most well-established projectional editors are only available as heavyweight desktop applications. While this is hardly an issue when users are developers, it hinders adoption among other professionals. Users who aren’t developers but that would otherwise benefit from domain-specific languages and tools that allow them to represent their knowledge, and allow software to understand and act upon that knowledge. These users may find a desktop IDE or language workbench too intimidating, while a lightweight web-based editor would be more accessible.

Bridging the GapCan we somehow find a middle ground between textual and projectional editors? Is there a solution that has all the pros and none of the cons? Well, that’s impossible, probably. Still, maybe we can bring some features from projectional editors together with some from text editors and obtain a new kind of tool that is better than the alternatives in some aspects.

We want an editor that:

• is more expressive than plain text, i.e., it allows us to insert notations that are not easy to represent with text;
• provides a nice editing experience out of the box, so that novice users can just start typing without learning too much;
• is based on mature libraries and tools;
• is ready for deployment on the web.

Well, maybe we can have such an editor. Read on.

Word ProcessorsAt some point, we came up with the idea to take a word processor and improve it in some way, to get closer to that sweet spot. That may sound strange. We know word processors as tools that we may use to write many different kinds of documents, but certainly not code or structured data. Actually, it’s one of the facts that we learn early on as computer users, that you write some stuff in Word and other stuff in Notepad, and they’re not the same thing.

However, crucially, word processors are already a sort of projectional editors. When you use Word or Google Docs, even though it appears that you’re writing plain text most of the time, you’re actually editing a rich document model, that is both stored as structured data and also somewhat presented as structured data. Think about the DOM of a web page.

This model consists of paragraphs, tables, images, and other elements. And the operations that we can do on such a model are not limited to text manipulation; for example, we can indent a whole paragraph or change the bullet style of a list, or even cut and paste a slice of a table. Also, as anyone who tried opening Word documents in OpenOffice Writer or Pages will know, word processors don’t store documents as text, either. These days, they read and write compressed XML, mostly; in the past, it was proprietary binary formats.

Now, of course, the limitation of a word processor is that the metamodel, or the “schema” of the document, is fixed. It’s only meant to represent “rich text” documents, not other structures. Potentially, it may have a few extension points. For example, in Word, you can enter mathematical formulas, or insert other kinds of external objects. But usually, from the point of view of the editor, these are black boxes. They’re like images; you double-click on a formula, and another window opens, so you can edit it with a dedicated interface. When you close it, the editor updates with a new image. It’s not really an integrated experience. However, we can improve on that.

ProseMirrorAfter some research, we identified a promising project that could work for our use cases. It’s called ProseMirror and it’s a toolkit for building web-based word processors. So it’s not a prebuilt editor component, it’s a set of modules that work well together. Of course, it comes with built-in features that we can reuse as-is or adapt, such as copy-paste, keyboard shortcuts, undo/redo, etc. Even if it’s not a finished editor, and requires some setup, the text editing experience out of the box is ok.

Some other key features of ProseMirror include:

• it’s open-source and has a vibrant, helpful community
• it’s built to be extensible
• it has a transactional design, which for example makes it relatively easy to integrate collaborative editing
• while it focuses just on the editor, and by itself it doesn’t come with a full UI with, for example, a menu bar, it’s been integrated into all sorts of component libraries and frameworks, commercial and open-source.

Also, crucially for our goals, ProseMirror has a free schema. That is, it doesn’t come with a fixed metamodel of the structure of the documents. We can define our own blocks and nesting rules, either starting from scratch or extending a predefined schema.

Thus, we can combine standard rich-text blocks with other kinds of structured blocks. Those will be well integrated into the same editor, unlike what happens in Word or Google Docs. That is, we can define how our new blocks are presented and we can define how they interact with the rest of the editor. That’s the “projectional” aspect of our editor in a nutshell. Also, as we’ll see, we can incorporate parsing in some of these blocks, so as to have potentially a mix of rich-text, projectional, and traditionally parsed code – all in the same document, with the same editor.

Our Experiment in ActionLet’s look at some of our work, in pictures, since we cannot share the code.

Here we can see an instance of the editor. It may look underwhelming if you expect a full word processor like Google Docs, because we can only see very few controls here. But this is on purpose. The ProseMirror toolkit only deals with the contents of the editor. Then, it’s up to the developer to build the UI around it – menus, a toolbar, and everything else. So, to keep things simple for demos, we only built a very limited editor experience, just to show some rich text that we can style as bold and italic. We also chose to color the free-form text using CSS, to distinguish it from more structured code sections that we’ll insert later.

Under the editor we show the underlying document model as JSON:

So we can already see that even though it looks like we’re editing simple text, actually we’re acting upon a structured model. The structure of this model is defined by the metamodel, which we can also see as JSON. We’ll delve deeper into the topic in the following sections.

Introducing Code BlocksNow, let’s look at some of the “magic” that we’ve added. After the rich-text block, we have placed a special block. That’s a “parsed text” block that we defined on top of the basic ProseMirror schema. Any text we write there is passed through an ANTLR-based parser. In particular, we chose to integrate a Kotlin parser that we generated from the official open-source Kotlin grammar.

As we can see, we report parse errors directly in the editor, to showcase the integration. Now, of course, this is not a full code editor; it’s a very simplified proof of concept. The idea here is to show that we can mix rich text with something that is already a little more structured.

Of course, a parser’s output isn’t just a list of errors. Indeed, the main result of parsing is a parse tree – a tree representation of the grammatical structure of the code. We can see that, in our editor, we seamlessly integrated the parse tree into the document model:

Thanks to ProseMirror, we get features like undo and redo, that work with our custom block from the start. We even get primitive copy-and-paste out of the box – however, everything becomes plain text when pasted. To have a more integrated copy-paste implementation, so that, for example, pasting “code” in the middle of a text block will split the text and insert a code block, we’ll need more work. We won’t be doing that for the time being, but the possibility is there.

Going ProjectionalSo far, we’ve shown how to mix rich text editing with light “code” editing – where code is plain text and we use a parser to reason about its structure. We can see both these aspects as “projections” of the underlying document model, but that’s not exactly what we mean by a projectional editor. In fact, even though we store the model as structured data, we can only edit the document as text. In other words, the only editing we can do is by means of string manipulation operations, like adding a character (typing), inserting a substring (pasting), etc.

Such an approach, if properly developed, can address interesting use cases, for example, editing code and documentation together. However, we want to go a step further and introduce a new block to our editor that is more clearly projectional. That is, we want to have editable fields in the middle of other content that is not editable as text.

Let’s see this in action so it will hopefully be clearer. Let’s say that we want a block to represent a Kotlin class. So, instead of writing the text “class” followed by a space, the class name, braces, and all the syntax that Kotlin requires, we’ll just press Ctrl+Space:

Let’s select “class” from the context menu; that will insert a “Kotlin class” block:

This is an excerpt of the underlying model:

{ type: "block", attrs: { kind: "class" }, content: [ { type: "line", content: [ { type: "keyword", attrs: { keyword: "class " } }, { type: "field", attrs: { name: "name", placeholder: "name"… So, this is a kind of block that has some fixed portions – such as keywords and punctuation – that we cannot edit as text, and some fields that we can write into, such as the name of the class in the example. From ProseMirror we also inherit operations on whole blocks, such as selection:

In the picture, the entire class block has been selected. You can do that in ProseMirror for any block by clicking on it while holding down the Ctrl key (or Cmd key on OSX). In addition, we set up projectional blocks so that, when the user clicks on a non-editable element (such as the keyword “class” in the example), the entire block is selected. Once a block is selected, we can for example delete it by pressing the backspace key, or move it around by clicking and dragging with the mouse.

As with parsed text blocks, a lot of work is needed to make these projectional blocks usable in a real-world editor. However, the basic idea is hopefully evident enough.

Mixing and MatchingSo far we’ve seen a class block and how we can change the class name by typing in a field. However, the most interesting part of a class are its contents, such as properties and methods. Let’s see what we can offer to edit the body of a class.

One alternative is to insert another parsed block:

Then, we’ll edit the body of the class as text, but the model will contain a tree structure for it, as we’ve seen before.

The interesting thing about this approach is that the parsed block is now constrained to recognize only the elements that can actually be in the body of a class. So this is a viable path if we have a textual language with a parser, and we want to gradually build a projectional editor for it.
Another alternative is to create projectional blocks for the elements in the class body. Here we can see a projectional block representing a method declaration, that in Kotlin is introduced with the fun keyword:

In the end, we can decide where it’s convenient to use projectional editing, and where it’s more convenient to use a parser-based approach. For example, editing mathematical expressions in many projectional editors is awkward; even in an advanced tool like MPS, a lot of customization is required to allow users to just type in the expression (and edit it afterwards). In those cases, it may be preferable to stick to a text editor with a parser. That provides a nicer editing experience out of the box, and since the language of arithmetic expressions is much more constrained than a full programming language, the drawbacks of a parser-based approach are usually greatly mitigated.

ConclusionsWe’ve shown a very early research prototype that combines aspects of traditional text editors and projectional (or structural) editors, based on the ProseMirror editor toolkit. Hopefully, this article will spark a conversation on the approach we’ve taken. Also, if you’re interested in developing the concept further and applying it to your use case, please contact us. We’re also experts in traditional code editors such as Monaco and Visual Studio Code, as well as DSLs with MPS and other technologies.

The post Bridging the Gap between Textual and Projectional Editors appeared first on Strumenta.

View Details

In the world of transpiler development, a type of compiler that takes source code written in one programming language and converts it into another programming language, the ability to navigate between the original code and the generated code can greatly enhance the developer’s experience. In this article, we will explore a solution of using the Language Server Protocol to bridge the gap between two distinct programming languages; IBM RPG and Python programming language. The article will showcase the benefits and practical applications of this solution, and provide a comprehensive guide on how to implement it in your own projects.

This article presents a basic implementation of a Visual Studio Code extension that makes use of a Language Server and a Language Server Index to track the relationship between code generated by an RPG to Python transpiler based on the RPG parser and Python code generator. Both components, the parser, and the code generator have been developed using Kolasu, a Kotlin open-source framework created and maintained by Strumenta, designed to support the implementation of Abstract Syntax Trees (AST) providing functionality to define, navigate and transform the AST.

Kolasu is part of the StarLasu project that provides the same functionality in other languages such as Typescript and Python. It should be noted that the code presented is this article is not a full implementation of a compliant LSIF, and a more comprehensive implementation is beyond the scope of this article.The Language Server Index Format (LSIF, pronounced “else if”) is a file format for describing the structure of a codebase and the relationships between different elements within it. It is designed to enable developers to quickly navigate and understand large codebases, and to support features like go-to-definition, find-references, and so forth.

The Language Server Index FormatThe main idea behind LSIF is to have a common language-agnostic format to represent code information that can be used in different programming languages. This allows for different tooling to be built on top of the same data, making it easier to share and reuse code intelligence across different IDEs and other tools. LSIF also provides a way to handle the scalability problem, The language servers need to analyze (parse) the codebase to provide code intelligence. Having a pre-generated index file allows for skipping the process of analyzing the codebase every time, which could be slow and resource-intensive. In short, LSIF aims to improve the developer’s experience by providing easy access to code intelligence information across different tools and languages and making it more efficient by using pre-generated index files. The following diagram provides an overview of the overall architecture presented in this article, in particular, the translation from an IBM RPG source code (.rpgle) to a Python source code (.py).

The IDE implements an LSP Client to communicate with the LSP Server. The message exchanged between the client and the server follows the standard LSP protocol. In addition, the IDE allows users to read and display the Original Source code and the Output source code. The transpiler is responsible for the source-to-source conversion and in addition to the output code it generates an index file (.json) which stores the cross-reference information between the original source code and generated code. In this particular case where an RPG program is translated into a Python program, the index file contains information about what Python statements have been generated from a particular RPG statement (or statements) and vice versa.

The Language Server ProtocolThe main benefit of the Language Server Protocol is that it enables the implementation of cross-platform and cross-language development tools for code editors and integrated development environments (IDEs). This standardized protocol allows seamless integration of language-specific tools, such as code completion, go-to-definition, linting, and hover information, into a common development platform. It also defines a standard way for the client to provide context information to the server, such as the location of the cursor in the code of the currently open file. Additional information about the Language server protocol is available in this article.

The diagram below illustrates the message exchanged between the client and the server when a user opens a file in the IDE, then positions the cursor on a symbol and requests the symbol definition.

The initialization of the LSP is the process by which the language server and a client establish a connection and exchange information to prepare for communication.The initialization process has two steps:

1. The client sends an initialize request to the language server, providing information about the client’s capabilities and the current state of the code editor, such as the location of the cursor in the code and the URI of the currently open file.
2. The Language Server responds with a server capabilities response, providing information about the server’s capabilities, such as which languages it supports, additional information that the client needs to know, such as the version of the language server, the messages supported and any additional settings that can be configured.

Once the initialization process is complete, the client and server can begin sending and receiving standard LSP messages, such as requests for code completions, definition requests, or error diagnostics. In the second interaction, the user opens a file (foo.rpgle) and the IDE Language Client sends a didOpen message to the Language Server. The didOpen message contains information about the file that has been opened, such as the file’s URI. The Language Server uses this information to analyze the file and to compute code intelligence features. It’s important to note that the didOpen message is not used to open the file on the server side, it’s used to notify the server that a file is opened on the client side so that the Language Server can run the appropriate language-specific code analysis algorithms and provide the code intelligence features. In the next interaction with the IDE, the user positions the cursor in the code and selects the Goto Definition menu item. This action triggers the definition messagewhich is sent by the client to the server to request the definition of a symbol at a specific location in the code. The request contains the file URI and the position of the cursor in the code. The server processes the request by analyzing the code and providing a response with the location of the symbol definition. The location can be in the same file or in another file, and can also contain multiple locations if the symbol is defined in multiple places. The code analysis required to compute the location of the definition may result in traversing many Abstract Syntax Trees which can be not very efficient. The use of pre-generated Language Server Index files can improve the performance and scalability of the language server. Additional information about the messages supported by the LSP can be found in the official documentation.

Cross-Language Code NavigationIn addition to the common code intelligence that can be implemented within the Language Server for development purposes (such as autocompletion and error checking), working with transpilers to migrate legacy code involves a lot of back-and-forth navigation from the original and the generated code. The LSP protocol does not include a specific message to map a source code to another source code, it would be necessary to develop a custom message. Although it is possible to implement custom LSP messages, to add new functionality to the language server or customize the behavior of the client it is required that the custom message follows the LSP message structure and format and ensures that both the client and server are compatible with the custom message. Another option would be to reuse an existing and already defined message adding additional parameters to trigger the new functionality, which is much simpler to implement. For example, to navigate from the RPG source code to the corresponding Python-generated code it would be nice to point to the statement of interest, right-click, and select Goto Generated Code.

The same functionality would be applied to the generated Python code to jump back to the Original RPG code.

To implement such functionality we can make use of the standard definition message appending an additional parameter to the file URI. There is a second advantage to this solution: when dealing with a Python file the IDE would interact with the Python Language server which does not support the cross-navigation functionality. I would be required to modify the Python language server, which is not an option. The simple way to implement this feature is to create a custom menu entry that sends both requests to our language server implementation.

The extension implementation, based on the context will append a postfix “/py” or “/rpgle” parameter to inform the language server that the functionality requested is the cross-navigation between the source code. Based on the param the Language Server will retrieve the location and the ranges using the entries within the index, if the parameter is not specified it means the client requested a standard definition and the Language Server will return information about the symbol.

Structure of the Index In order to understand the structure of the index it is necessary to provide a little background information about how transpilers work. The two main components are the Strumenta RPG Parser and the Python Code Generator, both based on the Strumenta open-source Kolasu framework. The Kolasu framework provides support for the AST definition, navigation, and transformation.

The RPG parser processes the input RPG source code, including the DDS required for symbol resolution and produces the AST representing the code. The generated AST is submitted to the transpiler that performs the necessary transformations to create the AST representation of the target code in Python. The structure of the Python AST depends on runtime, which Python libraries will support the runtime environment and it is usually a custom development. Optionally, in the case of DDS it is possible to generate the corresponding SQL for logical and physical RPG files. The transpiler is also able to generate the Index File for the cross-language reference leveraging on the features of the Kolasu. In Kolasu, each node of the AST contains position information about the position and the range of the statements in the original source code. This position information is populated during the parsing phase. The transpiler, while computing the target AST, which is also based on Kolasu, populates the origin property of the target AST node creating a link between the two AST.

After the code generation, in the target Python AST both the position and the origin are populated. The origin contains the position on the original RPG code and is simply copied for the original node; the position is calculated during the Python code generation. Now that the target AST contains all the required information it is possible to traverse the target Python AST and generate the cross-reference between the RPG and the Python code. The physical file structure of the cross-reference is basically a map that contains the URI of the file and a map of line numbers and source code ranges. The diagram below shows the structure of the index file for the previous example.

The repository contains a map of the URI of the files pointing to the Cross Reference Data which contains the name of the corresponding generated file URI and the map of the corresponding positions. The position map consists of the line in the original code as key and the position in the target code as the corresponding position. For example, by positioning the cursor at line 1 in the foo.rpgle file the Cross Reference Data contains the URI of the target Python file (foo.py), and line 1 corresponds to the code between line 4 column 1 to line 5 column 6.

LSP ClientThe source code of the project is available on GitHub and includes the Visual Studio Code extension source code and the Language Server in Kotlin. The first step in the extension implementation is to set up the Language server client.

export async function activate(context: vscode.ExtensionContext) : Promise<void> { const initTasks: Promise<void>[] = []; /* Language configuration */ vscode.languages.setLanguageConfiguration("rpgle", {}); const lsConfig = vscode.workspace.getConfiguration("rpgle"); initTasks.push(withSpinningStatus(context, async status => { await activateLanguageServer(context, status, lsConfig); })); await Promise.all(initTasks);} The activateLanguageServer function performs the initialization of the LSP client The Kotlin LSP server is written in Kotlin it is launched using the sample-xcode-server script located in the languageServer.path specified in the configuration. When the server is executed it performs the initialization sequence as mentioned early.

async function activateLanguageServer(context: vscode.ExtensionContext, status: Status, config: vscode.WorkspaceConfiguration) { console.log("Activating Language Server..."); // The language server must be in the bin directoryconst languageServerPath : string = config.get("languageServer.path")!; // The language server must be in the bin directory const startScriptPath = path.join( languageServerPath , "bin", "sample-xcode-server" + ((process.platform === 'win32') ? '.bat' : '')); console.info("language server path: " + startScriptPath) if (isOSnix()) { child\_process.exec(`chmod +x ${startScriptPath}`); } const outputChannel = vscode.window.createOutputChannel("RPG"); context.subscriptions.push(outputChannel); let env: any = undefined; const options = { outputChannel, startScriptPath, env }; languageClient = createLanguageClient(options); // Create the language client and start the client. let languageClientDisposable = languageClient.start(); context.subscriptions.push(languageClientDisposable); await languageClient.onReady();} The next step is to implement the two commands for the cross-code navigation, Go to Generated Code and Go to Original Code in the context menu available when right-clicking on the source code. To add a context menu item in a Visual Studio Code extension, you need to create a new command and register it with VS Code. Once the command is registered, it will appear in the context menu when the user right-clicks on the editor. In the package.json file of your extension, add the command name to the contributes.menus section. This will make the command appear in the context menu.

"commands": [ { "command": "language-server.origin", "title": "Go to Original Code" }, { "command": "language-server.target", "title": "Go to Generated Code" } ], "menus": { "editor/context": [ { "when": "resourceLangId == rpgle", "command": "language-server.target", "group": "navigation" }, { "when": "resourceLangId == python", "command": "language-server.origin", "group": "navigation" } ] } The when clause allows specifying when the option in the context menu’ should be visible, in our case we want the Go to Generated Code available on RPGLE files and Go to Original Code when editing a Python file. In the extension code, we create a new function that defines the behavior of the command, in the activate function we register the command with VS Code by calling the registerCommand function and passing in the command name.

export async function activate(context: vscode.ExtensionContext) : Promise<void> { ...let gotoGeneratedCode = vscode.commands.registerCommand('language-server.target', async (files?: vscode.Uri[]) => { let fext = extractExtension(vscode.window.activeTextEditor?.document?.uri.fsPath!);if(fext == "rpgle") { executeCrossReferenceRequest("file://" + vscode.window.activeTextEditor?.document?.uri.fsPath+"/py") } } ); let gotoOriginaldCode = vscode.commands.registerCommand('language-server.origin', async (files?: vscode.Uri[]) => { let fext = extractExtension(vscode.window.activeTextEditor?.document?.uri.fsPath!); if(fext == "py") { executeCrossReferenceRequest("file://" + vscode.window.activeTextEditor?.document?.uri.fsPath+"/rpgle") } }); context.subscriptions.push(gotoGeneratedCode); context.subscriptions.push(gotoOriginaldCode);//context.subscriptions.push(disposable);} The executeCrossReferenceRequest perform the request to the language server to obtain the cross-reference data and to open and select the generated code. The client sends a textDocument/definition message to the server, with an additional py or rpgle parameter to specify the action required. The language server will access the index file to retrieve the location of the corresponding generated code returning the URI and the position within the code.

async function executeCrossReferenceRequest(uri : string) { let curPos = vscode.window.activeTextEditor!.selection.active; console.log(uri); console.log(curPos); let param = { textDocument : TextDocumentIdentifier.create(uri), position : curPos, partialResultToken : undefined, workDoneToken : undefined }; languageClient.sendRequest("textDocument/definition", param) .then( (data : any) => { console.log(data); let fileUri = vscode.Uri.file(data[0].uri) let start = new vscode.Position(data[0].range.start.line - 1, data[0].range.start.character) let end = new vscode.Position(data[0].range.end.line - 1, data[0].range.end.character) vscode.window.showTextDocument(fileUri, { preview: true }).then((editor) => { var range = new vscode.Range(start,end); editor.revealRange(range) editor.selection = new vscode.Selection(range.start,range.end); let timerId = setInterval(() => { clearInterval(timerId); editor.selection = new vscode.Selection(range.start,range.start); }, 1000); }) } ) .catch( reason => console.log(reason) ); } LSP ServerThe Kotlin language server is launched by the extension when it started and as described earlier in this article performs the initialization. During this phase, the server searches for the index files (with an extension idx) and submit the task to a worker pool.

class RPGLanguageServer : LanguageServer, LanguageClientAware, Closeable { ... private val workerPool : ExecutorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()) override fun initialize(params: InitializeParams): CompletableFuture<InitializeResult> = async.compute { LOG.info("initialize") val extensions = listOf("idx") params.workspaceFolders.forEach { ws -> val directory = File(URI(ws.uri)) val files = directory.walk().filter { it.isFile && extensions.contains(it.extension) }.toList() LOG.info("Adding workspace ${ws.uri} to source path ${directory.exists()} ${files.size}") if(directory.exists()) { files.forEachIndexed { i, file -> val progressPrefix = "[${i + 1}/${files.size}] ${file.name}" val progressPercent = (100 * i) / files.size progress?.update("$progressPrefix: Updating source path", progressPercent) workerPool.submit(RPGParserRunner(file,this)) LOG.debug("Adding file ${file.absolutePath} to workspace") } } } progress?.close() val clientCapabilities = params.capabilities InitializeResult(serverCapabilities) } ...} The indexes are .json files that contain the cross-references between the source files and are loaded by the RPGParserRunner and stored within the Repository. The code presented in this example is loads only the index files, but in this phase, it would be possible also parse the source code, create the AST, resolve the symbols and add to the repository additional information to provide code intelligence such as hover, goto definition and so forth.

class RPGParserRunner(val file: File,val server : RPGLanguageServer) : Runnable { override fun run() { server.LOG.debug("Parser started : ${file.name} ${file.lastModified()}") val elapsed = measureTimeMillis { try { if ("idx".equals(file.extension)) { val xref = Klaxon().parse<CrossReference>(file) xref?.references?.forEach { k, v -> Repository.addCrossReference(k, v) } } } catch (e: Exception) { server.LOG.error(e.message) } } server.LOG.debug("Parser completed: ${file.name} in $elapsed ms") }} The Repository contains the cross-reference map which contains a map of CrossReferenceData for each document URI and the ranges relative to the code as explained in the section relative to the index structure.

data class RefPosition( val startLine : Int, val startCol : Int, val endLine : Int, val endColumn : Int)data class CrossReference( val references : MutableMap<String,CrossReferenceData> = mutableMapOf())data class CrossReferenceData( var uri : String = "" , var crossReference : MutableMap<String,RefPosition> = mutableMapOf())object Repository { private val crossReferenceMap: MutableMap<String, CrossReferenceData> = ConcurrentHashMap<String, CrossReferenceData>() fun addCrossReference(name : String, xref : CrossReferenceData) { crossReferenceMap[name] = xref } fun getCrossReference(name: String): CrossReferenceData? { return crossReferenceMap[name] }} When the user selects one of the options on the context menu the server receives a textDocument/definition message containing the URI of the file and the position of the cursor. This information is used to look up the cross-references and returns the location of the original or generated code. The py and rpgle postfixes indicate that a cross-navigation request has been received.

class RPGTextDocumentService (val server : RPGLanguageServer) : TextDocumentService, LanguageClientAware, Closeable { override fun definition(params: DefinitionParams?): CompletableFuture<Either<MutableList<out Location>, MutableList<out LocationLink>>> { val line = params!!.position.line + 1 val column = params!!.position.character var uri = normalizeUri(params.textDocument.uri) LOG.info("definition $uri $line,$column") val locations = definitionAt(line,column,uri,LOG) locations.forEach { it LOG.info("definition $it") } return CompletableFuture.completedFuture(Either.forLeft(locations)) }}fun definitionAt(line: Int, column: Int, uri: String, LOG: Logger? = null) : MutableList<Location> { val locations : MutableList<Location> = mutableListOf() when { uri.endsWith("/py") || uri.endsWith("/rpgle") -> { val xuri = uri.substring(0,uri.lastIndexOf('/')) LOG?.info("xuri: $xuri") val xref = Repository.getCrossReference(xuri) xref?.apply { val targetURI = xref.uri LOG?.info("xuri Found") val elapsed = measureTimeMillis { xref.crossReference.get("$line")?.apply { val range = Range(Position(this.startLine, this.startCol), Position(this.endLine, this.endColumn)) val location = Location(targetURI, range) locations.add(location) } } LOG?.info("index search time: $elapsed ms") } } } return locations} SummaryWe went through a simple implementation of an index to navigate the relationship between the source and the generated code of an IBM RPG to Python transpiler.

Although is not a full implementation of a compliant LSIF, the index is accessed in the Language server and provides a very useful functionality to navigate the code. It is particularly useful if you are already familiar with VS Code extension development, and Language server development but it can be a good choice even for people interested in transpiler development using the Kotling programming language.

You can find the soruce code for this article on GitHub.

The post Editor support for cross-language navigation appeared first on Strumenta.

View Details

In this article, we are going to present you our SAS parser, a commercially-licensed parser for SAS. We are going to see how and why you can use it. What is SAS? SAS is a programming language used for statistical analysis. It is a part of the homonymous system used by large organizations to perform […]

The post How to Use the SAS Parser appeared first on Strumenta.

View Details

For our series of interviews “Strumenta Talks”, we host Adolfo Ochagavìa. Adolfo Ochagavìa is a freelance software developer with an interest in Domain Specific Language. He will explain his project and share with us his experience of building DSLs for the chemical industry. […] There are also some tools to report this as a Word […]

The post Strumenta Talks: in conversation with Adolfo Ochagavìa appeared first on Strumenta.

View Details

Spring Boot Migrator is a tool to migrate applications to Spring Boot. In this article, we are going to see how it works and how you can use it. This may seem a narrow topic, but it is an interesting example of an application that can manipulate your code. The reason is that the tool […]

The post Automatic Refactoring With Spring Boot Migrator appeared first on Strumenta.

View Details

A new major release for ANTLR with version 4.11.1. Technically we are at release 4.11.1, but this is the same as 4.11 apart from correcting the internal version name. Excluding the Go target, this is not a major release in terms of features, but there are significant improvements for the ANTLR project itself. As a […]

The post We better Go with ANTLR 4.11 appeared first on Strumenta.

View Details

Introduction In this tutorial, we’ll give an introduction to Peggy. As usual in our articles, all the code is on GitHub. Peggy is the successor of peg.js, an open-source library for writing parsers in JavaScript. It’s based on parsing expression grammars or PEGs, hence the name. PEGs are a powerful parsing formalism with a gentler […]

The post A Peggy.js Tutorial appeared first on Strumenta.

View Details

Today we are taking a look at Spoofax, a Language Workbench, an environment that permits quickly creating DSLs, and in this tutorial, we will show a concrete example of that. We think that anyone seriously interested in Language Engineering should be familiar with all the main platforms. Some of them could become useful tools in […]

The post A tutorial on Spoofax, a Language Workbench appeared first on Strumenta.

View Details

In this article we will talk about parsing log files: how it is different from parsing your average programming language and how to accomplish it. Parsing log files is a common need. Big companies like Microsoft release tools just for that purpose and there are even entire companies built around the task of parsing and […]

The post Parsing Log Files appeared first on Strumenta.

View Details

Why this interview? Kevin is a Consultant Solution Architect with a long experience, he worked for Intel, eBay, Realtor.com, Microsoft and now is working for Data View LLC. He has a lot of very interesting experiences and we talk with him focusing in particular on migrations and, in particular about his specific experience with migrations. […]

The post Interview with Kevin Mackey appeared first on Strumenta.

View Details

One of the most productive features of ANTLR is the support for easily processing the result of parsing. In this article, we are going to talk about the two ways to handle the result of parsing: listeners and visitors. We are going to see how they differ and what are the best use cases for […]

The post Listeners And Visitors appeared first on Strumenta.

View Details

Every day we get closer to death, but on the other hand we also get closer to a new ANTLR release. All in all a fair trade, maybe. My point is we are all going to die soon that we have a new major release of ANTLR: 4.10. The previous one was in the most […]

The post A Shiny, New ANTLR Release: 4.10 appeared first on Strumenta.

View Details

The code for this article is on GitHub: getting-started-blockly In this article we are going to talk about Blockly, a library to create visual programming languages. The user writes code using blocks and the software transpiles it to your traditional programming language of choice. It gives the power of programming to people without any experience in […]

The post Anybody Can Code With Blockly appeared first on Strumenta.

View Details

Why this interview? For our new series of interviews “Strumenta Talks” we host Rob Wunderlich. Rob Wunderlich provides tools and training for users of Qlik software. He is the creator of QSDA Pro, it is a code profiler and code analysis tool for QLIK Sense Applications, so for applications that are written using the Qlik […]

The post Strumenta Talk: in conversation with Rob Wunderlich appeared first on Strumenta.

View Details

Introduction Parsing is the process of turning a linear sequence of characters into an organized, tree-shaped structure. Then, a computer program can further elaborate on such a structure to produce useful results. For example, it could tell us that “cool” isn’t a valid data type, even though we’d so much want it to be. Jokes […]

The post Challenges in Parsing Legacy Languages: The Case of SAS Macros appeared first on Strumenta.

View Details

The code for this article is on GitHub: getting-started-antlr-php PHP had always been a popular language, but for a long time it had less than a stellar reputation. Well, all of that is in the past. And what better proof that PHP is again a serious language than ANTLR support? In this article we are […]

The post Getting Started With ANTLR in PHP appeared first on Strumenta.

View Details

More and more people are looking into adopting DSLs. While they see the general benefits, and enthusiastically learn the technicalities there is one recurring issue many are facing: figure out on which problem to apply DSLs.  For this reason, I think that sharing experiences and reflections is useful. Today I am very happy to share […]

The post Experiences of Practical DSLs usages: a talk with Glen Braun appeared first on Strumenta.

View Details

Today we see more and more interest in low-code and no-code solutions. One may wonder how they relate to DSLs. Aren’t they all solving the same problem: democratizing software development? Well, yes, and no. In this article, we would like to discuss what these solutions are useful for, how they compare, and what you can […]

The post When you need low-code or no-code and when you need DSLs appeared first on Strumenta.

View Details

Introduction Or, is writing this tutorial a good idea? We’ve written about Monaco and, separately, about TextX, in several past articles. However, we’ve received enough requests for a TextX + Monaco integration tutorial, that we really had to write one. The thing is, combining TextX with Monaco doesn’t really make much sense. Or does it? […]

The post Integrating TextX and Monaco – A Non-Tutorial appeared first on Strumenta.

View Details

DSLs are relevant only to companies that build software by making developers and some kind of non-developers collaborate. This is the situation in which DSLs shine. If this is not your case then probably adopting a DSL should not be among your priorities. Some examples: You build software for tax calculations. You have tax specialists […]

The post Which CEOs should care about DSLs? appeared first on Strumenta.

View Details

DSLs are relevant only to companies that build software by making developers and some kind of non-developers collaborate. This is the situation in which DSLs shine. If this is not your case then probably adopting a DSL should not be among your priorities.

Some examples:

• You build software for tax calculations. You have tax specialists collaborate with developers
• You build digital therapeutics software. You have healthcare practitioners collaborate with developers
• You build business automation software. You have business analysts collaborate with developers
• You build actuarial software. You have actuaries collaborate with developers
• You build software to control engineering processes. You have engineers collaborate with developers

There are no other single factors that make a huge difference. Regarding the size of your company, if you can afford to invest tens of thousands of dollars or euros to get the DSL designed and implemented for you, then you are not too small. The field you operate in is irrelevant: there are applications of DSLs in all sorts of verticals. I can just say several examples seem to work particularly well in regulated fields, probably because those companies hire many specialists.

Why should a CEO care about DSLs? Because:

• There is a cost in making your specialists and your developers communicate. They spend a lot of time explaining what they do to each other and there are misunderstandings. DSLs can bring down those costs
• Because they can help specialists do their job more efficiently. Often specialists have limited tools to check their reasoning. Many errors are found only late in the process when developers start writing code. That is terribly inefficient. When we use DSLs we can give tools to specialists that help them do their own thing and find many errors themselves, early in the process
• Because it can reduce time to market. In each case we have ever seen, once you get a DSL people can do their own stuff more efficiently and more quickly. As a result, things take weeks instead of months and you can put your products out there much more quickly

Ok, what are DSLs? In essence, think of super-smart editors that can help specialists write things down correctly.

Imagine these editors have a “reasoning spellchecker”, which points out errors.

And imagine these editors being able to generate automatically useful stuff: software, documentation, configuration files, and more.

No, there is no AI involved in the process. What makes it possible for these smart editors to be constructed are DSLs. The technology behind it is actually quite boring and reliable, but let’s keep that between you and me.

That’s great, but what could go wrong? Now, there are challenges and risks when adopting DSLs:

• DSLs designers with the right skills are rare. You may need to wait to get the right people available to get started or to scale up your project. You do not need a ton of people for creating DSLs, but you need the right ones and you cannot find them that easily
• Adopting DSLs changes how things work. It does not matter that things are changed for the better, someone will be resistant to change. It could be the manager leading the testing department who all of a sudden needs fewer people, or it could be a long-time employee used to get things done in a certain way. In our experience, while people can consider themselves to be open to change, they are not always so ready to adapt their way of working, when needed
• Developers may be against DSLs if they are not involved right away in the discussion. DSLs make the life of developers easier, but they frequently misinterpret the idea behind DSLs, so they may think it is not possible to achieve what DSLs promise. It should be clear that DSLs are not about getting rid of developers. It is about removing the boring back-and-forth dance to implement, correct, and revise once more details in the code based on how specialists change their minds. This is done by making the specialists become much more independent. Developers will still be very important for the development of the core platform. Typically developers, in the end, see how DSLs benefit them too, but their concerns must be addressed by involving them properly, or they could resist and even sabotage the project

Ok, this seems to make sense. How can I learn more? * You can throw some technical material to your CTO. For example, this guide on DLSs. * You can read some case studies, like this one: Voluntis Case Study

• And you can, of course, ask us to explain more, and more importantly discuss if and how this can work for your specific case. We could start with a 45 minutes call to discuss your case. You can book it by filling out this questionnaire.
• You can also write at info@strumenta.com

If you are interested in this movement, you may also want to take a look at the Subject Matter First Manifesto.

The post Which CEOs should care about DSLs? appeared first on Strumenta.

View Details

Hedy is a pedagogical language designed to simplify learning programming. In this article we are going to see how it can improve teaching programming to students and can inspire language designers to create languages that are simpler to learn. It is a language that proposes a new approach to teaching programming. It is really interesting […]

The post Teaching Programming With Hedy appeared first on Strumenta.

View Details

We have built numerous transpilers for our customers and we know how challenging it can be. This is why we started exploring machine learning to implement our transpilers using examples provided by our clients. The challenge was to find an approach that would work with a limited set of samples and we believe our approach […]

The post Language2Language Transformers: machine learning to build transpilers. appeared first on Strumenta.

View Details

WASI: how to run WebAssembly code outside of your browser? WebAssembly started as a technology for writing applications running in the browser without using JavaScript. Now it can also be used to run applications outside of the browser, thanks to WASI. In this article, we are going to see what this is useful for, how […]

The post WASI: how to run WebAssembly code outside of your browser appeared first on Strumenta.

View Details

Why this interview? Peter Strong is a software engineer, architect, and he spent most of his career in the telecoms space, with companies like Nortel Networks, Ericsson, Arista Networks, and in a lot of smaller companies, startups in the communication space, like Meriton Networks, BelAir Networks, Lumina Networks. We talk with him about his project Dark Matter focusing on the problem that he was trying to solve when he created Dark Matter.

So, the basic idea is having a Java-based framework where you could describe schema of all of the data that you wanted to manage, and then to be able to easily translate that into an interface through which you could manage it.

So the basic concepts have gone from strictly modeling data more towards being able to define concepts associated with anything. So as time went on, moving into different domains, what I found was that the same problems occur again and again. You come across a particular domain, you see people struggling to represent the concepts in that domain, and so the basic concepts behind Dark Matter, like defining attributes, types, classes, and then generating code from it, or even just representing configuration data associated with different domains, it occurred again and again.

So basically, from the Dark Matter site, we can provide a link later, you can download this starter project. And the idea here is that we have no real code associated with your DSL at the starting point, just a framework. So from here, you will do basically a bootstrap of the project, and it’s simply asking you for a package associated with this. So if we do something like, we have example, and that example DSL, xDSL. And then it will ask you for, basically, a file extension or an abbreviation of your DSL. So in this case, we’ll do xDSL. And it will generate a variety of things.

So basic concepts, and this is the thing. It comes from an object oriented perspective. It’s based on Java.

Federico Tomassetti:
Hello, Peter. Thank you for joining us today.

Peter Strong:
Federico. Likewise. Good to see you.

Federico Tomassetti:
Good, good. I really appreciate the possibility to interview you. We have a lot of questions that I want to ask you, so I will start right away. And the first thing that I will like to ask is to tell us a bit about yourself.

Peter Strong:
Right. Well, I’m a software engineer, architect, and I spent most of my career in the telecoms space, with companies like Nortel Networks, Ericsson, Arista Networks, and in a lot of smaller companies, startups in the communication space, like Meriton Networks, BelAir Networks, Lumina Networks. So I come from a networking background, and for me, a lot of the aspects there have to do with the configuration data associated with networks and understanding it, and thus the reason I started this project.

Federico Tomassetti:
Good, good, good, and indeed we will talk about this project, Dark Matter, today. And well, I would like to start a discussion about Dark Matter, focusing on the problem that you were trying to solve when you created Dark Matter.

Peter Strong:
Right, right. So within the networking space, of course, like I said, it’s very data-intensive. And so one of the first things was trying to come up with a relatively straightforward way to represent the concepts of configuration data associated with networks, but also then to allow it to be managed through a web-based interface. And when this started back in 2010, the choice then was to use Google Web Toolkit.

Peter Strong:
So, the basic idea is having a Java-based framework where you could describe schema of all of the data that you wanted to manage, and then to be able to easily translate that into an interface through which you could manage it.

Federico Tomassetti:
Okay. And in this case, the intended users of your system would be a network engineer, right?

Peter Strong:
Yes. In that case it would be users who were configuring their network data, and being able to basically understand how all that data fitted together. So things like relationships between objects and things was very, very important, and Dark Matter made that a very clear process.

Federico Tomassetti:
Good, good. And okay, you say the project was started over 10 years ago using a technology that at the time was very interesting. Today is less common. So can you tell us a bit about the portability of this solution, and how the technological stack evolved over time?

Peter Strong:
Right. So the basic concepts have gone from strictly modeling data more towards being able to define concepts associated with anything. So as time went on, moving into different domains, what I found was that the same problems occur again and again. You come across a particular domain, you see people struggling to represent the concepts in that domain, and so the basic concepts behind Dark Matter, like defining attributes, types, classes, and then generating code from it, or even just representing configuration data associated with different domains, it occurred again and again.

Peter Strong:
So as time went on, I basically began to build pieces that were more along the lines of a domain-specific language, to represent things, and provide behavior associated with those concepts.

Federico Tomassetti:
Good, good. So I imagine that given you have created essentially a DSL, you were able to capture what was relevant to you initially. So a network configuration where you instruct from the technologies that we’re using for the optimum implementation. In your case, for code generation. So make this logic survive all these technological transitions, right?

Peter Strong:
Right. So that is one of the benefits of having a base model for whatever it is that you’re doing. Now, still, most of my work has been in Java, but I’ve also used these mechanisms to generate Python, to generate Golang, to generate other forms of schema that are required. And in some cases, I guess, augmenting existing DSLs in a way that adds more descriptive aspects to them.

Peter Strong:
One area where I’ve done that was with Google Protocol Buffers. So we were using Google Protocol Buffers to transmit the status information, performance data, and things like this, from devices. And Google Protocol Buffers is an incredible compact form of sending data of one place to another, also makes it so that if there are changes in the data from one version to another, it seamlessly handles those things. Fantastic mechanism.

Peter Strong:
What was missing was a way to be able to document that data, which was very important for the end users of it. So in that particular case, I simply modeled the Google Protocol Buffer concepts in Dark Matter, but then allowed the engineers on the device to extend it, with definitions like succinct descriptions of what each of the fields meant, the units involved, and so forth, and then use that to generate the Google Protocol Buffer stack but also generate FrameMaker documentation.

Peter Strong:
Once again, that’s an older form of things, but the documentation guys love this because there was a single source of truth for what the data was. You were able to have the Google Protocol Buffer specification for the device and for the clients of that data, but then also get the documentation from it, which was being built as part of the build process. So, it basically helped everyone communicate and get useful information to make using the data easier.

Federico Tomassetti:
Yeah. I imagine that the documentation for this kind of system is very important, because probably they’re going to last for a while.

Peter Strong:
Well, it lasts for a while but then it also changes on a fairly frequent basis. So having that as a clear mechanism to say, you know, we’ve added new fields in here, or more importantly, perhaps, we’ve changed the meaning of something, and it’s immediately reflected in the documentation, and in the API that’s been generated from this code. So yeah, documenting things, very, very important.

Federico Tomassetti:
Good, good. So we talked a little bit about the advantages, we gave an idea to our listeners of the main goals of the project. Maybe this is the right time to show a little bit how it looks like.

Peter Strong:
Sure. I can just give you a short demo of basically a starter project. So let me just see here. Share screen. And now, I’m not sure if this will… Shift. Okay. Okay. So can you-

Federico Tomassetti:
Yes, I can see it. I can see Eclipse.

Peter Strong:
Okay. So basically, from the Dark Matter site, we can provide a link later, you can download this starter project. And the idea here is that we have no real code associated with your DSL at the starting point, just a framework. So from here, you will do basically a bootstrap of the project, and it’s simply asking you for a package associated with this. So if we do something like, we have example, and that example DSL, xDSL. And then it will ask you for, basically, a file extension or an abbreviation of your DSL. So in this case, we’ll do xDSL. And it will generate a variety of things.

Peter Strong:
So now, if we refresh our project, we’ll get some interesting pieces out of it. So from this it creates a file structure that provides you with what I would call the meta-schema of your DSL. So I’ve broken it out here. All of this can go into a single file, but I tend to break it out into various pieces.

Federico Tomassetti:
Sure.

Peter Strong:
So basic concepts, and this is the thing. It comes from an object oriented perspective. It’s based on Java. So you will see things like class definitions. These wind up being turned into actual Java classes of different types, for instance, abstract classes, structural classes. And in this particular case, I’ve just had it generate a couple of basic concepts to show you the idea here. So you have your class definition, and then you have references to must-have, or mandatory attributes. So in this case it is a named concept, and it has a variety of other attributes that it may have.

Peter Strong:
Your attributes are defined, likewise, as attribute definitions, with names. So something that looks a little bit like Google Protocol Buffers, although this was invented prior to that time, is unique identifiers for each of these things. This allowed for very succinct, compact transmission on the wire for Google Protocol… Or, not for Google Protocol Buffers, but for Google Web Toolkit.

Peter Strong:
So the idea was, you weren’t constantly transmitting strings all over the place, you wound up with this. Now, at some point we may talk about the future of things. I may wind up changing this so that you don’t have to provide this. But at the moment, it is the way that everything is represented within the framework. So you have to provide a unique identifier within a given module for each of your different pieces.

Peter Strong:
So you have your classes, you have your attributes. You can also define types, so things like your enum definitions, you can have. And then, there’s something else which basically allows you to define complex types with required parts and optional parts, and come up with basically very succinct microgrammars for particular attributes. So in this case, you have a reference to a Concept B, some strange things to say why you’re doing this, this is to provide, say, an alternate name for something.

Peter Strong:
And then, so you can specify it in a single line without very much syntactic cruft, as I would say, so that you can simply say, an example of this attribute would be… And I’ll show you the instances of these objects. William prefers this, Bill… And then you can put in an optional note or something else like this. But from this, you can build up fairly complex structures that are very easy to write into a file.

Peter Strong:
Those are the basic pieces. From that, like I said, it generates a series of code that you can then extend with your own behavior. For instance, for each of the concepts there there’s a separate file that’s generated, and you can go ahead and extend this with behavior. Anyway, there’s a lot of different aspects associated with that that we won’t have time to cover, at least here.

Peter Strong:
But then, the actual example of the data associated with this DSL, it would be you have a module. Basically, the whole thing is set up in terms of being able to specify modular chunks of specification. Modules allow you to include other modules. And then within this, you always start the type of object that it is that you’re defining. If it’s a named object, you have its name, and then the attribute values associated with that.

Peter Strong:
So, in this particular case, we have the alternate specification and this is an example of how you actually specify this in the grammar. The nice thing about all of this is that you don’t have to do anything special in terms of parsing or figuring this out. This is all built into the framework, so you can get started very, very quickly, and then if you actually run, in this case, the test associated with this, it will parse this file.

Peter Strong:
And in this case all it’s doing is it’s finding all of the Concept As and printing them in this format, which is simply called Object Instance Format. And then, it’s also showing you the same thing dumped in JSON format, which obviously very familiar to everybody, but it’s the way that the object instance format works is that it has a lot less in terms of quotes, and colons, and braces, and so forth. So it’s cleaner for a lot of users to be able to specify. But if you want the data in JSON you can have it in JSON as well.

Peter Strong:
So from there, you can also just do… One of the other things that I always find starting a project myself, is that it takes time to figure out how to build the project and deliver it. And with this starter project, all of that stuff is taken care of for you. So you can do a Maven build, it will go ahead and create what’s called a shaded JAR, which basically has the entire set of code built into it. It’s self-contained. Now, what I’m not sure is if I switch windows, can you still see my screen?

Federico Tomassetti:
I still see Eclipse.

Peter Strong:
Okay. Right. Anyway. Suffice to say, you can set up an alias that points to that shaded JAR, basically Java minus JAR and your shaded JAR, and then it will start this base application that you have now built, and you can start going from there. So you don’t need to try to figure out, well, how do I actually deliver this to somebody? It’s done for you.

Federico Tomassetti:
I can see you have put a lot of effort on engineering this, and make it really usable for your colleagues.

Peter Strong:
Right. It’s one of those things that’s always a pain. You can see a very simple example of something, but when you actually go to deliver it to somebody, that can be a bit of a challenge. So I’m trying to make it as straightforward as possible.

Federico Tomassetti:
So if I understand correctly, using Dark Matter, someone can define some data structures, but also a lot of constraints for this data structure, and then someone else can actually define the data that respected that structure, right?

Peter Strong:
Correct. Yes. Now, for instance, I was going to show you, because I’m not as… New share, let me just see here. Yes. So let me just try this. Ah. Okay. So for instance, this is in another project called Polymer, that was built on top of an open source project called Plastic, that came out of OpenDaylight, which is an open source project for an SDN controller.

Peter Strong:
So in that particular case, there were a lot of very powerful concepts for doing model to model mapping and so forth. So just to show you, it isn’t just very simple things that you can do with this in terms of an attribute and a value. For instance, here, there was a concept of having reusable patterns, which is a fairly common thing when you are doing work with data models, and being able to translate or recognize particular things.

Peter Strong:
So in this particular case, the idea was that you could define a pattern that you could then refer to in another location. Often, you find in different grammars that because they’re hierarchic, often things are defined in JSON or in XML. They’re hierarchic. And so if there’s a piece of information in there that you want to use over and over again, there’s often not an easy way to do it. With the Dark Matter setup, you can actually have it so that you refer to this as a first class thing, and then in another piece of the grammar, you can simply say, “I’m going to make a reference to a Plastic pattern.”

Peter Strong:
And the pattern is defined in this case in JAVA regular expression format. It allows you to specify groupings so that you can easily reach into the pattern and pull parts out of it. Once again, this is something that you can refer to in another aspect of the grammar to say, “Match this pattern and pull out Part One, and then insert it in some other structure or something else like that.” And it also has the concept of defining a test for this, so that you could easily say, “I want to use the Flash separator pattern,” so this is a reference attribute to this pattern, and give it some input, and say, “I expect that Part One’s going to be one, two, and Part Two is going to be three, four.”

Peter Strong:
And this way you can actually set up automated testing of your patterns that you’ve defined, to make sure that you don’t break them. Somebody might come in and make a change to this, but then you can run an automated test and go, “Nope. Somebody has broken this. Everybody expects this to behave in this way, and it doesn’t.”

Peter Strong:
So just in terms of expressibility of things like patterns, things like embedding JSON within the actual Dark Matter grammar as well, you can drop in JSON, you can drop in JavaScript, you can drop in Python. The way that the attributes are defined is pretty powerful in terms of defining the type. So it allows you to combine many different things within the structure.

Federico Tomassetti:
Okay. Well, I understand the embedding JSON. I’m not sure what the embedding JavaScript or Python are useful for.

Peter Strong:
Right. Well, in the particular case of JavaScript, I was working on, basically, an extract, transform, load pipeline, and we were doing model-to-model translation of things. And it turned out that DSLs are very powerful, but sometimes it comes down to, if you want to cover everything in the world, it becomes unwieldy, and it’s just too much. So you fall back to a programming language. And in this particular case it turned out that the network engineers, who were doing the model-to-model translations, were familiar with JavaScript. So basically I made it so that you could create, for instance, small, reusable functions, where you wanted to alter an attribute that was going from one model into another, but you could define it in JavaScript and execute it, because Java actually has the Nashorn implementation of the JavaScript engine in it.

Peter Strong:
It was very trivial to say, “I can define a JavaScript function,” which takes an input and gives you an output, and build it right into the language so that you could use it to define basically a function-oriented programming to simply say, “Here’s a reusable function that I can use to manipulate this,” and then the various network engineers could build a little library of these things and express what they need them to do. So, just another possibility. Once again, depends on the domain, and what your users want.

Federico Tomassetti:
So if I understand correctly, with your language, you have the possibility of representing directly a lot of data processing features that you may want, like recognizing patterns, but you also have the possibility, basically, of scripting, by using JavaScript or Python, so that you can cover those corner cases, and this is useful because if you didn’t do that, the alternative would be evolving the DSL to be as powerful as a general-purpose language but at that point you lose all the advantages of having the DSL?

Peter Strong:
Yes. So one of the other things is, I guess, a combination of things across domains. One of the other projects that I’ve worked on more recently had to do with managing instances of AWS, compute instances, so basically setting up virtual machines and things like that. And that got to be somewhat challenging because sometimes you would have to do basic bash scripting, for instance, to configure various things. And then in other cases you also had to use things like Ansible, an existing very powerful mechanism for doing configuration.

Peter Strong:
I was able to combine those different things to say I can define the instance of a virtual machine that I want, and then have various, what I would call initializers. And the initializers were an abstract concept, but you could then embed bash scripts within one type of initializer, Ansible scripts within another type of initializer, and use them interchangeably so that you were able to combine concepts across domains to achieve a particular result, which was basically configuring these virtual machines for use. So it helps, definitely.

Federico Tomassetti:
Wicked. So we have seen that Dark Matter is very flexible and so can be applied to many different domains, but I think there was one particular project that we discussed, initially by email, and was one project where your view is Dark Matter, and maybe we could talk a little bit about that project, if that works for you?

Peter Strong:
Sure. I’ll just… I guess I can stop sharing here, now, perhaps.

Federico Tomassetti:
Ah, yeah.

Peter Strong:
We can come back to sharing if required. Yeah. Okay. So, which particular piece were we referring to?

Federico Tomassetti:
I think it was a project about writing an NMS project using Dark Matter that previously was written in Clojure.

Peter Strong:
Oh, right. So this is basically what I would call a life cycle service orchestration product that required information from a disparate set of network devices. So we’re talking about integrating data from Cisco, Juniper, Huawei, many different network management devices. But to pull them together and represent them in such a way that the users could visualize the entire network from end to end.

Peter Strong:
So the challenge there was primarily that you had network engineers who understood, for instance, the command line syntax of a show run config on all these different types of devices, but they’re all subtly different. And the process for doing this in terms of integrating the data was originally that the network engineers would go through a particular data dump, try to understand it, and then write up stories that would then be passed on to the development team to say, “Okay, for this type of device I want you to extract this data.”

Peter Strong:
So it was a very labor-intensive process, as it were. So when I came to this and looked at it, the first thing was, how do I understand what the data is coming from these devices? So I created, basically, a configurable parser for this form of data. So command line syntax is very, very common in network devices, but they’re all subtly different. So the first thing that I gave the network engineers was a mechanism to say, “You can run this parser over the data and get a summary of the structure of the information.”

Peter Strong:
So how they configured it was defined in Dark Matter. So it was simply a concept of a indent-delimited parser for this data. From this, they got a data catalog which showed them the entire structure. So that was immediately useful because it allowed them to communicate to the development team exactly what data it was, and give examples of what it was. So we got that far-

Federico Tomassetti:
Yeah, if I can interrupt you, I imagine that in this situation it’s very important to be precise, and I think that what I can imagine happening is that when network engineers could only describe in words the format of the data, they inevitably miss some details. So they didn’t have a way to, let’s say, find that the bug is in the description.

Peter Strong:
Absolutely. And the thing was, is that each of these different forms of command line syntax can have different forms, with basically optional pieces in them, mandatory pieces. But the optional pieces would often change, and so it was important to recognize the keywords that were used to indicate pieces of optional data.

Peter Strong:
And of course, these dumps from particular devices, tens of thousands, or hundreds of thousands of lines long. So the network engineers, they tried to cover the use cases that they were thinking of, but often they wouldn’t catch everything simply because there was so much of it. So cataloging the data and giving a very clear indication of all forms of the data was a real bonus, because they would go, “Oh, I didn’t realize that you could also have this form of the data, or that form of the data. Okay, so I need to put that in my story as well.”

Peter Strong:
So it helped with the communication, just to start with. But then, it came down to, well, what’s the model into which we wish to turn this data? What do we want to turn it into? So I worked very closely with the CTO, director of standards and the director of solutions architecture in this case, and together we came up with an abstract model of what networks look like, in terms of their different layers, the dependencies between the layers, and that was actually put together into a patent for the company.

Peter Strong:
So that was the other aspect, and likewise, that was modeled in Dark Matter. Now, it comes down to we have what’s on the device, what’s the abstract model that we want to work into, so the network engineers were trained on what that model was, and then was, now, how do I go from one to the other? And that was where having the DSL that allowed them to do the model-to-model translation came into play, and coming back to, now we need reusable functions, because a certain form, say, of an IP address, or a site name, or something else like that, had to be normalized or whatever.

Peter Strong:
So now it came down to, we could extract the data, provide a catalog for it, we have a model to which we wish to translate it, and now to join the things together, to connect the dots, we come up with a DSL that lets them express how to pull the data out and turn it into the abstract model.

Peter Strong:
So now we go from, we have to write everything up in a very long, tedious error-prone format, into here’s a clear definition of what we need to be able to do. We can run it basically to process a particular configuration file from a device. They can easily look at the output in the object instance format. It’s cleaner to look at than JSON. JSON is, once again, very, very useful, much better, for instance, than XML, but XML has its purposes as well. But it’s a very clean format. They can look at it. They can eyeball it very quickly and go, “Okay, now I’m going to turn this into another format,” which wound up in a graph database. And that was based on the schema designed for this abstract model of what the network looked like.

Peter Strong:
So it significantly shrunk the delivery timeframes from, here’s a new device that needs to be integrated into the overall view of the network, to now we actually have the data for it extracted, and represented in a model, and available for use by the product.

Federico Tomassetti:
Okay. So if I understand correctly, before using Dark Matter, basically the network engineers had to look at logs, understand the format, describe it, give it to developers, the developers were trying to implement it. Eventually they will recognize corner cases for which there was not a description, or maybe the description wasn’t absolutely precise, maybe some details were overlooked so they had to go back with the network.

Peter Strong:
Yes. Right. And so the iteration time in there was quite long. So any situation where you can increase the level of communication and clarity and reduce the iteration time for whatever it is you’re doing, I think that’s one of the benefits of a DSL. Now, that being said, it does take more study, upfront knowledge of the domain, in order to create the DSL. So this is one of the things that sometimes people will look at, “Well, why is it taking this long?”

Peter Strong:
Well, there’s what I would call a potential hump that you have to overcome. It’s the learning of the domain that takes the time. But then, once you’ve learned it and you’ve communicated the way of how to express it to your end user, then you see the benefits of it, and I think in any organization, overcoming that initial resistance… It took you six months to create this thing, but that’s a very long time. But if you then compare that to the effort involved and the time involved and the errors that you eliminate from whatever process it is, I think there are tremendous benefits there. But it’s always, I find, an education process in terms of the benefit that you get out of some forms of DSL versus continuing to do it the way you’ve always done it.

Federico Tomassetti:
I think it would be interesting to discuss more about the resistance that the different authors may have, because I can imagine that the domain experts, in this case the network engineers, could resist, for example. They could say, “I don’t want to become a developer, so I will just give you the document, please figure it out.” Or maybe the developers could resist, say, “Okay, it’s our job to write the system, why these engineers are trying to do that?” Or maybe the organization could resist because they could think, “I don’t want to make the investment in creating the DSL because I don’t know how soon I will recover my investment.” So these are ideas that comes to mind. I don’t know if any of these happen, and I would be interested to hear what really happened.

Peter Strong:
Yeah. Strangely enough, from the user perspective, I find less resistance, because if you successfully… You sit down with them, they’re your customer, you sit down with them, you understand their pain points and what causes them grief, and what causes them to be frustrated with how long it’s taking to do something. If you deliver something that actually solves those things, or lessens those problems, I think they’re quite receptive.

Peter Strong:
Now, strangely enough, you tend to get more resistance, I’ve found, say, from development communities and from management along the lines of what you were talking about. There seems to be… In some cases it’s simply momentum, as in, an object at rest has to stay at rest. “This is the way that we’ve done it. We’re familiar with these mechanisms. This is new and we don’t want the change.”

Peter Strong:
Then, it could also move into, “This is new, and it actually eliminates part of the work that we need to do.” And again, this depends on organization. Sometimes cultural background. I’ve run into that as well, where the amount of work that people do, if you try to reduce it, they resist that, because they think eventually they’re going to be out of a job. But in actual fact, what you’re doing is you’re making them more productive so that in the end the company is more productive. They’re getting more value from the people that they have on staff, and that’s a good thing, because now the company is just spending less time doing one thing over and over again. They’re actually increasing their revenue, as it were, because now they can take on more work, get in more customers, end customers, and go that way.

Peter Strong:
So, there are so many different social landscapes into which you have to sell these ideas. But I think in the end, when it comes down to the organizational resistance, it has to come down to, yes, there’s an upfront learning cost here, but once you have actually overcome that, you are going to be able to do more work, get more things done for your end customers, and therefore you’re going to be more profitable if you take this kind of approach.

Peter Strong:
But it can be a tough sell, to come into a new domain, understand it well enough to give a very good proof of concept. It can sometimes be a challenge. But I guess the one thing that is good there is that so many domains exhibit the same kinds of patterns in terms of what they’re attempting to do. Once you’ve got enough of these things under your belt, you begin to be able to take advantage of those patterns, and go, “Okay, I’ve worked on a project like this previously, I can dust off a piece that I already have, drop it in, and get moving much more quickly.”

Peter Strong:
But for anything, there’s always that learning curve, and explaining the benefits to the developers involved, to the organizations involved. Sometimes you’re successful, other times you’re not.

Federico Tomassetti:
I think this is why these kind of discussions are very important for the community, to show examples of where this work and what kind of benefits it brings, and also discuss the fact that yes, resistance is normal. We experience it. There are reason for it, but in the end, if organization are willing to work on that resistance, there is the possibility to improvement. I would like to talk exactly about the advantages that you think this solution has brought in practice, compared to the previous system.

Peter Strong:
Right. In this particular case, it was really the network engineers, or actually, it was more of a pre-sales engineer that I helped in this case. The particular company that we were doing this work for was a new potential customer, and they needed to see results in a relatively short period of time. And this person had basically come into the company, really only maybe a month or so prior to my having put this entire mechanism in place.

Peter Strong:
And he was able to, within a month of coming in, he was able to integrate new devices that this customer had been dealing with, that we hadn’t even seen before. So the existing network engineers hadn’t been dealing with this device before, had no experience with it, but this gentleman came in, I showed him the basic concepts, he grasped them very, very quickly, and was able to do the integration and do a proof of concept for the end customer within a very short period of time. And in the end, we wound up securing that customer, and it was a big win for the company.

Peter Strong:
So it was basically, to be able to bring somebody in, and within a month make it so that what had maybe in the past taken six months for an experienced network engineer to accomplish, going through the previous development approach was pretty satisfying, I guess I’ll put it that way.

Federico Tomassetti:
We can say that speeding up development by six times is quite an interesting achievement, probably. Many companies are interested in that kind of improvement.

Peter Strong:
Indeed. Well, I would hope so, because in the end, it means you’re bringing in more revenue because you’re able to do more work with the same amount of people. Some people will look at it you can do the same amount of work with fewer people, but I don’t think generally companies will do things that way. They have employees who’ve been working with them for a long time. They understand the landscape of what it is the company is trying to do, and I don’t think many companies will actually look at it as an opportunity to get rid of people. I think they will more look at it as, “We have this backlog of potential customers that we haven’t been able to deal with. Now we can actually start dealing with them and expanding the amount of work that we can do.”

Peter Strong:
So anywhere where you can reduce the amount of development effort to satisfy an end customer I think you’ve got a tremendous business advantage there, and it’s just once again, you do have that potential hump you have to overcome. You have to understand the domain well enough to do something that’s useful enough to get that reduction in the iteration time, to perform whatever task it is that you’re doing.

Federico Tomassetti:
Yeah. And I also think that from the point of view of the domain expert, in this case the network engineer, it must be very satisfying to be able to achieve more in the same time. In a way, you compress the boring, mechanical part, but your actual contributions stay the same, you’re just able to produce it as a much smaller time.

Peter Strong:
Yeah. Absolutely. And I think they found that… I actually had some comments or recommendations on my LinkedIn from the point of view of the people who were my internal customers, who were just so happy that they were able to do this. Because it’s tedious and time-consuming, especially within these complex of networks, to go through all of this data. So anything that lets them understand and grasp the data more quickly, and then actually see the results of their effort for their end users, I think is very satisfying to them, and very energizing for them. Because it was like, “I don’t have to do all of this work up front and then wait several weeks for some result to come out.”

Peter Strong:
And then that brings to mind another thing which is context switching. The more time you leave between various parts of whatever it is you’re doing, you lose your context, and then two weeks later, somebody comes and says, “Hey, I’ve got that data. It’s been extracted into the database. Can you check it?” And you go, “Well, now I’ve forgotten what that is, and now I have to go back.”

Peter Strong:
And so context switching is a cost as well, and I think that’s one of the things… I don’t know if I’ve ever really put it that way, but I find myself, if I’m working on a project, if I can work that project through to completion without then being interrupted by many other pieces, I tend to be more productive. And I think that that’s what they found, was that they had the context, they understood the customer problems they were trying to solve, and the data associated with this, and that they were able to go from, “Here’s the data,” to, “Here’s how we actually want to present it to the users so that we can see how their network is connected, and do it in a week as opposed to several months.” I think, “That’s great, now we’ve done that, and now I can move onto the next customer and solve their problems.” So yeah, context switching, the more of it that you can reduce I think the better off you are.

Federico Tomassetti:
Okay. We are running out of time in a few minutes, so I want to be sure we cover a few more points. One thing that… Well, I think that you occurred many times in discussion about domain-specific languages and model-driven development in general is that they seems to be used mostly in Europe and less in North America. I think you’re based in North America, right?

Peter Strong:
Yes.

Federico Tomassetti:
So can you comment on this?

Peter Strong:
Yeah. It’s more of a feeling. I can’t point to substantive data on this, but I tend to find that most of the blogs that I read, and people that I follow, tend to be more based in Europe than in North America. And the only really big code generation conference I’ve ever been to was actually in England, the code generation conferences that were there. I was only lucky enough to go to one of them.

Peter Strong:
And it was one of the most, I guess, professionally satisfying conferences I’ve ever been to. Simply to be surrounded by people who, I think, think the same way in terms of what the benefits of domain-specific languages, model-driven engineering, code generation, and these kinds of things, are. And once again, I’m thinking back, this was quite a while ago now, but I would say basically all of the people that I’ve met there were from Europe. And it’s quite puzzling. I don’t know if it’s a cultural thing, if the background of people in Europe tends to be more formal from the point of view of software development, not so much the move fast and break things which tends to be maybe more of a Western… Well, when I use it in that term, Europe would be involved in Western civilization, but I’ll say North American view of things.

Peter Strong:
I’m really not sure what the difference is, but it just seems to be that way. And I’ve worked with people in different organizations in large telecom companies, for instance with Ericsson, and even within Ericsson I found that there was a core set of people in Stockholm who were very aligned to model-driven engineering and code generation and things like that. Whereas, the local office was not nearly as aware of these kinds of things. And I’m not sure. It’s an oddity. But it’s good. I appreciate the fact that there is a community there, and you are part of that community, and educating people on these things and helping us with different approaches to this. And yeah, I appreciate that you are all there.

Federico Tomassetti:
Yeah. Unfortunately, I never attended a code generation conference, but I heard many people talk about, so I wasn’t that lucky. But it’s true that if you’re based in Europe, maybe it’s easier to get in contact with some of the people working the area. For example, in my case, during my PhD I was able to spend six months in Germany, working with Markus Völter and his team that had a lot to learn about this work.

Peter Strong:
Yeah. Indeed. I mean, yeah, geographically traveling within especially the mainland of Europe and everything else is considerably easier than in North America where things are fairly distributed in terms of distances, cost of getting to different places, and things like that. And being in Canada, at least staying in Canada, it’s even worse, because we’re much more sparsely populated. And things like rail travel and whatnot, it takes a long time to get to different places. So you’re pretty much, you have to go with air travel, and yeah, it’s definitely much more expensive to get around. So yeah, definitely could be.

Federico Tomassetti:
Good, good. And I think that before we close, one thing that we need to discuss is the future of Dark Matter. So if you can tell us what you’re planning for the project?

Peter Strong:
Well, I think in some regards this opportunity that you’ve given me has really wanted me to do a lot more in terms of proselytizing, presenting this, and educating about the benefits of this. So I may wind up taking it in the direction of calling it something like Dark Meta, because most of the information… There’s not really Dark Matter data so much as it’s the meta information around different domains, and being able to express it, being able to combine them, and this kind of thing.

Peter Strong:
I’ve been using this in basically every company that I’ve been in for the past decade, but I haven’t really had the time to put into educating people about it, documenting all of the things that are there, putting together a weekly or a monthly blog about how to approach different things. So I think this is good impetus to start looking at that, and investing a bit more in making it available to more people.

Federico Tomassetti:
Good, good. And is there a website where people can find out about Dark Matter, or a link we can share?

Peter Strong:
Yes. I can provide you with those links. It’s available on GitHub. The DM Start Project, which I think right now is going to be the best place to go. It’s quite small at the moment, but I will basically extend the documentation there to go through some of the things that I showed you there today, just in terms of here are the basics of how you get started, and then the idea would be, I can expand on that to say, “Okay, here are some examples of more real-world things, instead of just a Concept A and a Concept B. Let’s actually walk through creating something from scratch that is real.” So I’ll provide you with the GitHub link.

Federico Tomassetti:
Good. We will add the link there

Peter Strong:
… and we can go from there. Okay.

Federico Tomassetti:
Okay, my final question is this, is there anything you would like to add and I forgot to ask you?

Peter Strong:
Good question. I think we’ve covered most of the things that we had initially talked about. Once again, I would definitely like to express gratitude from the point of view of the work that you have been doing. I came across you because I was doing some work with ANTLR, with which I have never really had much experience. I came across your site, I got a lot of very useful background information that basically allowed me to move forwards with the work that I was doing, and just your commitment to educating people in this area and all the rest of it, once again, I’m very much indebted to you, and keep up the good work. I think there’s a lot of benefit of this for a lot of people. They just need to know about it.

Federico Tomassetti:
Thank you. Thank you a lot for your kind words, and thank you for taking the time for sharing your work, and yeah, it was great. And thank you very much.

Peter Strong:
Thank you, Federico.

Federico Tomassetti:
Okay, bye.

Peter Strong:
Take care.

Link:

GitHub DM Start Project

The post Interview with Peter Strong on Dark Matter DSL project appeared first on Strumenta.

View Details

In this article we are going to talk about Racket, a Language for Creating New Languages, the programming language designed to create new programming languages quickly. Finally, a racket worth your time.

Racket started as a pedagogical tool, but it has evolved into a complex and productive platform. It is a language and an ecosystem of tools, all aided by excellent documentation.

What Is Racket Racket is a complex project, with a solid theoretical foundation, it even has a manifesto. It is also the product of more than 20 years of practical experience.

It started as a tool to help students learn programming languages. From that perspective using Scheme as the foundation of Racket was probably a terrible idea, but on the other hand it forced the project to evolve, because:

we understood that nobody could teach Scheme in an hour and then focus on the essence of computing and programming (from the manifesto)

What they end up with is Racket as we know it today.

The creators of Racket believe that programming is about solving problems in the correct language, so Racket is fundamentally a tool to create new programming languages quickly.

Racket Is More Than a Language Racket has a family of programming languages that provides ways to safely interconnect different languages. It includes protection mechanisms to safely use low-level (think C) features of programming languages and more high-level languages.

It also includes an IDE, Dr. Racket, that is designed to be an integral part of the process, including resource management features to help design effective languages. Although it also has support for other common IDEs, such as VS Code or Emacs.

In short, Racket is a whole set of tools that provide a coherent experience to design programming languages. Mind you, it is not a language workbench: it does not help you in creating editors or other supporting tools for your new language. Racket is an ecosystem of independent tools, all catered to the creation and execution of the language.

The advantage of Racket is that you can start from zero and end up creating a whole set of languages, aided by excellent and abundant documentation. The disadvantage is that with such a tightly designed environment, you might have to start from scratch. That is because it is harder to reuse previous tools or environments that you are familiar with. It is kind of its own little world. For example, Racket has its own build tools that manages everything from installing packages to compiling code into executables.

Despite its academic beginnings, it has been used in notable applications like the creation of Arc, the language of Paul Graham, that powers Hacker News. It has also been used for scripting languages in some Naughty Dog (a videogame developer owned by Sony) games.

Learning Racket Given its origins, it is not surprising that Racket has good documentation. However, it is surprising how good and in-depth it is. Even just looking at their website, you can find a tutorial, a complete guide, a complete reference and also specific guides to use Racket in systems programming or web development.

Obviously, there is also documentation for using DrRacket, for other tools, like the package system, and a myriad of Racket libraries. Their documentation page lists 994 different manuals. They are so many that I had to make a script to count them.

There is even more than that. If you need more formal educational materials, you can read books teaching you:

• how to learn programming with Racket in different ways
• how to design programs (using Racket, but with a general application)
• varying level of computer science books that use Racket to teach the foundational concepts

You can also find courses, summer schools and training programs for learning Racket.

Racket did not take the world by storm in 20 years, but it has developed a strong and solid following. There is a stable and lively community that can help you using Racket to achieve your goals.

The best way to start is, of course, the quick introduction to Racket.

The Power of Racket There is so much clear and complete information about Racket that it is hard to add anything new about it. In fact, there is so much material that you might not know where to start. We believe we can say something useful about Racket by providing a clear introduction to the subject. We can show you how it feels working with Racket, so you can understand if it can work for you.

The first and fundamental point to understand is that Racket is based on Scheme. You can immediately recognize that by the liberal use of parentheses.

``` (define (my-length lst) (cond [(empty? lst) 0] ))

```

If there is one thing that everybody knows about Scheme is that: parentheses. The rest is lesser known. In fact I would say that Scheme is the perfect cross between an esoteric language and a normal programming language. It is a famous, influential and respected language, but it kind of lives in a separate world. Its syntax is complex and looks foreign, unrelated to other family of languages influenced by C or Python. And the syntax is just one part. The philosophy behind the language is quite different from what you are used to.

Scheme, and therefore, Racket, are based on the idea to give you just simple tools and let you run with them. This is why it is complex and powerful at the same time. There is no syntactic sugar in Scheme, but there is much power.

The Amazing Things You Can Do Racket gives you the full power of metaprogramming. This means that you can use code itself as data to be manipulated. In Racket you can do things like changing the way the code itself is interpreted, to achieve things that would be impossible in most other languages.

```

lang honu

function fib(n) { if (n == 0) 0 else if (n == 1) 1 else fib(n-1) + fib(n-2) }

fib(30)

```

This is a valid Racket program. The first line, #lang honu, sets the language, that is to say the rules, that will be used to interpret the rest of the code. Honu is just a language created by a user of Racket, that allows you to write code in a C-like style. As you can see, it does not look like a Scheme program.

It has the flexibility and power of Lisp, but you can also intervene on the syntax. For instance, you can make the language look completely different and remove the massive use of parentheses.

This makes Racket an ideal tool to create technical DSLs. You can create any language with Racket, even a markup language for publishing, like Pollen: the book is a program.

The Racket World You can create all sorts of programs with Racket. It comes with a score of libraries for supporting common needs like network, parsing and 3D libraries. There are also numerous examples of tools created with Racket, like one to synchronize a local directory with an AWS S3 bucket.

This is great, until you are fully in the Racket ecosystem. If you step outside of it, you might find some problems.

To see what I mean, let’s start with the good part. Look at this example program run in Dr. Racket.

The IDE does not look exceedingly professional but it is great for learning. It provides interactive help, that makes it easy to understand what is going on and what happens when you use any function Racket. You have a simple and easy-to-use IDE with an included interpreter. What more can you ask for?

The end result is a beautiful image of a checkerboard. Racket looks great: powerful and easy to use.

Now, look at what happens when you try to run the same code in Visual Studio Code, with the Racket extension. Can you spot the difference?

This does not look as nice. You cannot actually see any image. The program that works great in Dr. Racket do not work at all outside of it.

This is a common occurrence when dealing with integrated ecosystems. Sometimes things break when you are using something not specifically designed for that ecosystem. Mind you, other kinds of software that do not rely on the language slideshow works perfectly. This is a small flaw, but it can be jarring for developers used to recent programming languages, that the same code works differently when run in different environments.

Setting aside this flaw, there is another thing to remember when working with Racket.

Racket Is Easy to Learn… Have you ever met someone who said something like: “it is very easy to learn, just read this book about it”. And you immediately think that if you need to read a whole book about it, it cannot really be easy to learn. This is how you are going to feel with Racket.

There is extensive documentation for everything, from the syntax of the language to the computer science concepts behind it. This is not just because of the pedagogical origin of the language. You are probably going to need to spend a good amount of time reading the documentation. There is a lot of exciting information about programming and Racket in there. However, there is also just stuff that you need to learn and you are going to find it only there.

Many people, when they learn their first programming language, might be ready to read a tome with 1000 pages about it. For the second language they learn, they might read a small book to understand how the language differs from what they know. For every other new language they learn, they just look up some references and a bunch of tutorials. That is because they now get programming and assume that they kind of know how all programming languages work. For sure, a lot of people did that for JavaScript.

…And Yet It Is Hard You cannot use this approach for learning Racket. You cannot just slap a lot of parentheses around and assume that this means that you now know how Scheme works. Racket is quite different from every other language you know (unless, of course, you know another Lisp or Scheme language). So, you are going to need that documentation to take advantage of Racket.

So it is easy to learn, because there is a lot of documentation, but it is also hard, because you need to actually read it.

You cannot pick up Racket in a few weekends. The good news is that some people will find that exciting, a perfect chance to learn a new family of languages. Everybody else will have to decide if the usefulness of Racket is worth the effort.

Once you do that, though, You can really do anything with Racket. How many languages do you know that allow you to change the very way the compiler understands your code?

Summary Racket is a lot of things: a language, a family of languages and a powerful set of tools. All of this supported by in-depth documentation to guide you in all your endeavors. This makes it a good tool to build DSLs for small communities.

You can create a Prolog-like language or a Java-like one. And you can mix them in the same codebase. So, it is also the ideal choice for learning programming, improving your understanding of computer science concepts and experimenting with them.

The post Racket a Language for Creating New Languages appeared first on Strumenta.

View Details

Introduction Converting from JavaCC to ANTLR: JavaCC was the first popular parser generator (or compiler-compiler, hence the name) for the Java platform. It originated at Sun Microsystems and was later open-sourced.

In this tutorial, we’ll show how to convert a parser from JavaCC to ANTLR. Indeed, there are legacy parsers that are now dated; those would benefit from using ANTLR instead, which is more actively maintained and has other advantages, as we’ll see.

We’ll show a semi-automatic translation path, using a software tool that can quickly migrate the most common/boring parts of a grammar leaving only the more interesting bits out.

Why Converting to ANTLR Is a Good Idea Let’s set this straight first: converting or migrating some technology, library, language, etc. just for the sake of it, or to be “more modern”, is generally not beneficial. In fact, widely-used tools such as the aforementioned JavaParser and JSqlParser are based on JavaCC to this day.

Still, there are several reasons to migrate – some are generic, some are specific to JavaCC and ANTLR. So, let’s look at various aspects in which ANTLR is indeed “better” than JavaCC.

Like ANTLR and other similar tools, JavaCC takes a formal description of a language as input and then outputs a parser; in this case specifically, a piece of Java source code that implements the parser. The generated parser is capable of recognizing the language described by the grammar. As part of the process, it builds a structured representation of the source code you invoke it on – a parse tree. Application code can then further process the tree.

JavaCC is Java Only JavaCC only generates parsers in Java. Instead, ANTLR can generate parsers in several programming languages. In principle, we could have a single grammar from which we generate a parser in Java, another in C++, another in JavaScript, and so on.

Of course, that’s only relevant if we’re writing a library or component that we want to consume in-process from several languages or platforms. That may not appear to be an everyday requirement – unless your job is writing and selling parsers, that is. That’s part of our job, as you can see at Parser Bench, where we showcase some of the multi-platform parsers that we’ve built. However, that’s not the case in most software companies.

Still, nowadays it’s quite common to have some logic replicated on the backend (e.g. coded in Java or C#) and the frontend (JavaScript). Think of a domain-specific language that is compiled on the backend, but that users may edit on the frontend with a web code editor. Or even in a desktop application made with Electron, such as Visual Studio Code. The editor may use a parser to implement better language support, such as code completion or semantic checking.

If we generate the client-side parser from the same grammar used in the backend (or parts of it), we’ll ensure that the language is consistent between the editor and the compiler, and we’ll reduce effort and errors resulting from maintaining two unrelated parsers.

Now, not all supporting ANTLR runtimes have the same level of quality and performance; also, an ANTLR grammar can contain language-specific elements (predicates and actions) that prevent it to be reused as-is to generate a parser in another language. And, anyway, we could implement the client-server scenario above by delegating all the “language intelligence” to the server. Or, in the case of VSCode, to a separate “server” process communicating with the Language Server Protocol.

Still, that doesn’t come without drawbacks and added complexity. So, being able to reuse the entire grammar, or parts of it, in multiple environments, is an appealing feature of ANTLR, that JavaCC lacks.

ANTLR Has a More Active Community Community in open-source is also a matter of personal preference, since an important part of fitting in has to do with human relationships and affinity of values. Still, if we look at easily measurable parameters, ANTLR has more active committers, more commits, and a bigger ecosystem of supporting tools, libraries, and runtimes.

Also, ANTLR has a single reference implementation. Another performance-oriented implementation exists which is a superset in terms of syntax and features.

In contrast, JavaCC’s development has been stale for many years. Nowadays, JavaCC has both a “legacy” implementation, which receives some limited maintenance, and a newer fork/implementation called JavaCC 21, which is more actively developed. However, at the time of writing JavaCC 21 doesn’t seem to have gained a lot of traction. That’s proably at least in part due to its quite litigious author (we’ve been bitten ourselves, so beware should you inquire about JavaCC 21). So, choosing the “right” version of JavaCC today would require some study. That said, since we’re talking about conversion, the used JavaCC version is a given, not something that we can choose.

Anyway, both tools are used in popular open-source libraries. For example, JavaCC in JavaParser and JSqlParser, ANTLR in Hibernate 5+ and Groovy3, just to name a few.

ANTLR Is Simpler The grammar language used by ANTLR is definitely more readable. Compare:

literal: INTEGER\_LITERAL | LONG\_LITERAL | FLOATING\_POINT\_LITERAL | CHARACTER\_LITERAL | STRING\_LITERAL | booleanLiteral | nullLiteral;

With the equivalent JavaCC:

Expression Literal(): { Expression ret; } { ( <INTEGER\_LITERAL> { ret = new IntegerLiteralExpr(tokenRange(), token.image); } | <LONG\_LITERAL> { ret = new LongLiteralExpr(tokenRange(), token.image); } | <FLOATING\_POINT\_LITERAL> { ret = new DoubleLiteralExpr(tokenRange(), token.image); } | <CHARACTER\_LITERAL> { ret = new CharLiteralExpr(tokenRange(), token.image.substring(1, token.image.length()-1)); } | <STRING\_LITERAL> { ret = new StringLiteralExpr(tokenRange(), token.image.substring(1, token.image.length()-1)); } | ret = BooleanLiteral() | ret = NullLiteral() ) { return ret; } }

Also, ANTLR allows us to split a grammar over multiple files. That way, we can further reduce complexity by grouping related rules together while isolating them from unrelated rules.

To be fair, JavaCC 21 comes with much leaner syntax, which is close to ANTLR’s, while still supporting the legacy syntax. So, it offers a more gradual migration path. Also, it supports grammar includes, to split the same grammar across multiple files. However, it doesn’t appear to be used much in the wild yet.

Also, JavaCC generates self-contained parsers that need no external dependencies, while ANTLR requires a runtime support library. This could be a valid concern in the past. However, it’s unlikely to be significant today. Nowadays, the average application includes dozens of library dependencies already and we have powerful tools to manage them.

ANTLR Is More Powerful Technically speaking, ANTLR4 employs an adaptive LL(*) parsing algorithm, while JavaCC generates recursive-descent LL(1)/LL(k) parsers. So, JavaCC is roughly in the same league as ANTLR3. Instead, ANTLR4 is strictly more powerful in that it can operate with unbounded, adaptive lookahead.

In practice, this means that JavaCC recognizes fewer languages than ANTLR4. Also, it requires certain manual interventions that ANTLR4 supports automatically, such as choosing the lookahead depth and refactoring left-recursive rules. The latter applies in practice to expressions in most languages, where the naive and intuitive description looks like the following:

expr: expr (PLUS | MINUS) expr | /* other cases omitted */ | variable | NUMBER;

That’s a left-recursive rule, and neither ANTLR3 nor JavaCC support that; we’d have to rewrite the rule in a more convoluted way.

To be fair, these points mostly matter if we’re designing a new parser. When we’re dealing with a conversion, the JavaCC grammar already exists and works within the constraints of the parser generator. Since we’re talking about conversion, ANTLR4 being more powerful doesn’t seem to matter much.

However, with ANTLR4 we could then refactor the grammar to be more in line with ANTLR’s idioms and possibilities. We may also gain the possibility to evolve our language to support a new construct, that would have been hard to parse with JavaCC.

On another note, ANLTR4 generates a parse tree out of the box, with no special instructions. Instead, in JavaCC we must use the JJTree preprocessor – another tool to learn with its syntax and idiosyncrasies. On the other hand, with JJTree one may construct an abstract syntax tree directly from the parser – as much as it’s feasible and convenient; it was the solution used in ANTLR3 but it was abandoned in ANTLR4. With ANTLR, we have to transform the parse tree into an AST in a separate step. Fortunately, libraries such as Kolasu greatly help in that regard. Also, JavaCC 21 does not require JJTree (but it supports the same syntax as JJTree in the core).

Automatic Conversion We can go quite far with automatic conversion from JavaCC to ANTLR. Indeed, our own Federico Tomassetti wrote a proof of concept implementation. The context was to evaluate the migration of JavaParser from JavaCC to ANTLR. Thus, Federico’s tool was tested primarily against the Java grammar used in JavaParser at the time.

Let’s give a brief look at how the tool works. The basic principle is:

1. (Parsing) Read the text of the JavaCC grammar and parse it into an in-memory representation (abstract syntax tree)
2. (Code generation) Walk the AST to generate ANTLR4 code from it.

This is the minimal process to implement source-to-source transformation, or transpiling. In the real world, a transpiler performs one or more tree-to-tree transformations (also called model-to-model transforms) before finally generating code. That breaks down the generation process into more manageable steps. It is, for example, the strategy that JetBrains MPS extensively employs for code generation.

Loading a JavaCC Grammar We can use JavaCC itself to parse a JavaCC grammar. Of course, JavaCC (like ANTLR) needs that capability to do its job. However, ANTLR for example is separated into two components. We have the tool, which reads a grammar and generates a parser, and the runtime, which supports the execution of the parser. So, an application built with ANTLR usually can’t parse the ANTLR grammar language. To enable that, we need to include the ANTLR tool as a dependency. Of course, that’s only possible on the JVM, since the tool part of ANTLR is written in Java. JavaCC, instead, comes in a single monolithic library comprising the tool and the runtime.

So, loading a JavaCC grammar amounts to the following code (in this case, Kotlin, but it would look similar in Java):

fun loadJavaCCGrammar(javaCCGrammarFile: File) : JavaCCGrammar{ val javaccParser = JavaCCParser(FileInputStream(javaCCGrammarFile)) Options.init() javaccParser.javacc\_input() return JavaCCGrammar(JavaCCGlobals.rexprlist, JavaCCGlobals.bnfproductions) }

Note that JavaCC uses antiquated practices such as saving mutable data in static fields. Today we know that to be a poor design choice in most cases. This is just an example of JavaCC’s aging codebase. It dates back to the earliest versions of Java (before generics, collections, and most of the stuff we take for granted today).

Code Generation Superficially, there’s not much to say about code generation, either. In fact, most JavaCC concepts map to ANTLR concepts with only differences in syntax.

One evident difference is that, in ANTLR, parser rules are distinguished from lexer rules using the case of the first character in their name: parserRule vs LexerRule. By convention, lexer rules are typically written in all caps, as in “SELECT”. However, we can find grammars where only the first letter is capitalized, as in “Identifier”.

So, we just have to ensure that we properly capitalize the rule names in the JavaCC grammar. We’ve used the built-in String.capitalize method for lexer rules and the following Kotlin function for parser rules:

private fun String.uncapitalize(): String { return if (this.isNotEmpty() && this[0].isUpperCase()) { this[0].toLowerCase() + this.substring(1) } else { this } }

Then, we have to generate the code for the rules themselves. Here, we straightforwardly map each JavaCC concept to its corresponding ANTLR syntax.

For example, we have the concept of “choice” among several alternatives, which JavaCC represents with the class Choice extends Expansion (Expansion being the shared superclass of the elements that can go into a rule’s body, or expansion).

We convert a Choice into ANTLR grammar code simply by combining the choices with the pipe operator, “|”:

"(" + this.choices.joinToString(separator = " | ") { (it as Expansion).process(lexerDefinitions, namesToUncapitalize) } + ")"

Similarly, we translate ZeroOrMore nodes into application of the “*” operator:

"(${this.expansion.process(lexerDefinitions, namesToUncapitalize)})*"

We can see how, in both examples, we recursively apply the generation process – called expansion in this particular codebase.

Similarly, we handle all the other concepts such as OneOrMore (+), ZeroOrOne (?), etc.

Case Sensitivity A minor point to pay attention to is the case sensitivity of lexer rules.

JavaCC, like ANTLR3, is case sensitive by default but we can instruct it to ignore case when lexing. ANTLR4 doesn’t have that capability but it supports a well-known method for dealing with case-insensitive tokens.

We may leverage ANTLR’s import statement to confine case insensitive fragments into a file that we only import when needed. Then, it’s easy to automatically translate literal strings, for example, “foo” into F O O. It’s less straightforward, but still possible, to translate character classes such as [a-f] into their case-insensitive counterparts.

The Devil in the Details Upon further examination, we find that fully automated translation is a much harder task than it seems. That’s because of some semantic differences between JavaCC and ANTLR. One such example is the pattern for handling multi-line comments, such as, in Java:

/* this is a multi-line comment */

Here’s a simplified extract of the Java grammar used in JavaParser:

``` MORE : { : IN_MULTI_LINE_COMMENT }

SPECIAL_TOKEN : { : DEFAULT }

MORE : { } ```

We can see how, in JavaCC, we can use the “more” action to start building a token, change lexer mode, and continue lexing. Then, when the comment ends, we mark the whole token, accumulated so far, as “special”. In JavaCC, the parser ignores special” tokens. The concept of lexer modes is applicable to ANTLR as well; you may want to read The ANTLR Mega Tutorial for that and other advanced concepts.

In ANTLR, we cannot use that pattern (MORE + SPECIAL_TOKEN), because we can’t combine “more” with actions that skip the token or change the channel to make the token invisible to the parser. Therefore, we cannot translate the rules above as they are, one by one. We ought to recognize the semantics of the rules – what the author wanted to accomplish – and rewrite that in ANTLR style:

MULTILINE\_COMMENT\_START: '/*' -> pushMode(multiLineComment), channel(HIDDEN); mode multiLineComment; MULTILINE\_COMMENT\_END: '*/' -> channel(HIDDEN), popMode; MULTILINE\_COMMENT\_CONTENT: . -> channel(HIDDEN);

Notice how, both in ANTLR and in JavaCC, the end of the comment comes before the content, otherwise, the “catch-all” comment content rule would always match before the end-comment rule.

Then, we can rewrite the above rules in a simpler way, which is also closer to the JavaCC semantics. In fact, in ANTLR we don’t need to use lexer modes if we use a non-greedy operator:

MULTILINE\_COMMENT: ('/*' .*? '*/') -> channel(HIDDEN);

Here, the content of the comment consumes any character, but only up to the next star-slash sequence.

That said, algorithmically recognizing the above JavaCC pattern for comments is not simple, and indeed the proof-of-concept converter doesn’t try to do it. Instead, it uses a heuristic tailored to JavaParser’s “java.jj” grammar – if a rule contains “comment” in its name, then the generated lexer grammar will skip it. However, this doesn’t work for all JavaCC grammars, it’s just an accident of how the Java grammar is designed.

A more robust approach could be to bail out entirely and output a comment or warning urging the developer to translate those rules by hand.

In conclusion, fully automated translation is a hard problem, but we can go quite far with partially automated translation followed by later human intervention. As a reference, the java.jj grammar is ~2800 lines long, while the problematic comments part is just 30 lines, or slightly more than the 1% of the file. So, it makes sense to automatically translate 99% of the grammar!

The AST We’ve repeated this in various other articles: a parser, by itself, is useless. It’s only when we consume the output of the parser that we extract value from it. So, the integration of the parser with the application or library using it is a crucial point to discuss.

The output of a parser is a tree. We call it either “parse tree” or “abstract syntax tree” (AST) according to a fuzzy measure of how close the tree is to the structure of the grammar versus the abstract concepts that define the language.

A parser built with JavaCC may use the JJTree tool to build a tree according to some rules, or it may include actions that imperatively build the tree, written in Java by a developer. In either case, users of the legacy parser will have built their code against a certain API, which includes the classes that make up the nodes and leaves of the tree, and the methods to traverse them.

Instead, ANTLR (version 4) doesn’t give leeway to the developer to control how the tree is built. It builds a parse tree with a 1:1 correspondence to the rules of the grammar. Generally, we advise not to use the parse tree directly; instead, we suggest transforming it into an AST before further processing, for reasons that we won’t discuss here for brevity.

In the case of a conversion from JavaCC, we need to decide on a strategy for the AST:

1. Should we transform the parse tree produced by ANTLR into the tree that the existing code expects?
2. Or, should we write a new AST and break the API, forcing consumers to adapt to the new version?

There’s not a universally superior answer. In general, transforming the parse tree into an existing tree structure and API is not rocket science, and not markedly different from a transformation into a new AST. It may involve multiple passes and transformations, but we can find plenty of literature and examples to draw inspiration from.

However, maybe the conversion of the grammar is part of a greater modernization effort aimed at reducing technical debt. In that context, it may pay off to rewrite the AST with more modern practices and tools (such as Kolasu). Of course, such a decision depends on the project and the goals of the migration.

Note that we may still use some methods from Kolasu with the legacy tree, for traversal and other purposes.

Conclusions JavaCC and ANTLR use very different syntax but the underlying concepts map quite nicely one to one between the two, with some exceptions that are nontrivial to treat algorithmically. So, we can translate from JavaCC to ANTLR with a mostly automated process where some manual intervention by the human developer may be necessary.

You can find the proof-of-concept translator on GitHub and hack on it until it suits your needs. Please remember that Federico didn’t develop it for this article, it’s part of an earlier effort to migrate JavaParser to ANTLR. However, we’ve checked that it still builds at the time of writing on recent JDK versions.

The migration of parsers to ANTLR from older or less used parser generators (including previous ANTLR versions) is one of the services Strumenta provides to its customers, as part of our Legacy Modernization offerings.

Read more: If you want to understand how to use ANTLR you can read our article The ANTLR Mega Tutorial.

The post Converting from JavaCC to ANTLR appeared first on Strumenta.

View Details

Why this interview? Campbell Pryde is the CEO and Presiden of XBRL US, a global standard for reporting financial data for business and government, used pretty much in most countries around the world. In this interview, with Campbell Pryde, we try to find out more about XULE a rules language for XBRL.

Here there are some quotes I found particularly interesting:

[…] one of the things that we wanted to do was to basically write a rules language for XBRL, so that we could then write a lot of these rules a lot faster, and a lot more efficiently make debugging of those rules easier as well.

[…] the syntax was really designed in a way to make it as easy as possible to write, but at the same time, make it easy to debug and to understand, and to take up as little, to be as precise as we possibly could. And one of the underlying drivers for that was in theory, was to make it possible so accountants could write these rules and they could read them, or any other person who was like a data scientist. They could read it without necessarily having to go and learn a complex programming language to read the stuff.

[…] XBRL specification has evolved, so now you can get XBRL data in a JSON format, or you can get it in a CSV format, has something called inline XBRL, which is effectively XBRL data in a HTML format.

[…] we wanted to make it easier to pull multi-dimensional data and understand what those dimensions were.

[…] So if you want to, you can set it up so that you can point it to different taxonomies so that you can check it. So it’ll work with any taxonomy. So if you’re in Italy and you’re using IFRS, you can use it for that. It’s really not a US thing at all. So if you want to use it with any taxonomy, anywhere in the world that’s been published, then we’ll work with this tool as well.

[…] And it’s been very useful for improving the quality of the XBRL data, so particularly for the SEC data. So now that makes that data much more useful for other purposes. So people, as the quality gets better, more people start to use it. People use the data, they find issues in the data. They let us know, we’ll add a XULE rule, that’ll get fixed, more people start to use it, so it becomes the cycle, but we kind of needed to do that.

Federico Tomassetti:
So, thank you very much Campbell for taking the time for this interview. I’m very happy. And how are you today?

Campbell Pryde:
I’m good. Thank you for inviting me. I’m looking forward to talking about XULE.

Federico Tomassetti:
Good, good. And we will indeed talk about XULE, but before that maybe we can start explaining a little bit the context, and maybe you can talk a little bit about yourself and your organization.

Campbell Pryde:
Great. So just to give you some background. So XBRL is a global standard for reporting financial data predominantly, and it’s used pretty much in most countries around the world. Across Europe, there’s a big rollout of XBRL for public companies at the moment. It’s used heavily in Asia and in the United States, and also in Latin America, and also in the Middle East. It’s predominantly used by banking, regulators and securities regulators, but it’s also used by tax regulators as well. And the main purpose of it is to collect financial data. And this might be things that you’re familiar with, like a balance sheet, income statement, cash flow, that kind of information. Anything which is used to measure the performance of a business. So that’s what XBRL standard is used for. And throughout the world, there are a number of different jurisdictions who try to help people adopt XBRL, and then the US where we’re based, and that’s what we do.

Campbell Pryde:
So we predominantly help people like the SEC, which is the securities regulator here in the US, we’ve also help the Federal Energy Regulatory Commission. They’re the regulator who regulates the utilities industry or the electric gas and oil pipeline business. So they take financial data from their regulated entities and also a banking regulator called the FDIC, which is responsible for ensuring bank deposits in the US. And so they take a lot of financial matters as well. So, all of these entities are using XPRL to collect data. And once they collect it, then they make it available to people to use. So that’s kind of what we do is XBRL US, we help those regulators do it, but we also help people file to the regulators, and training and tools and trying to build a community for people to do that.

Campbell Pryde:
XBRL US itself, is a not-for-profit. We basically report into XBRL International, which is also a global organization, which is responsible for the experimental specifications. So, they basically will publish out the XBRL spec, which some of you may or may not be familiar with. You can go to their site, you can see all the specifications. And all those specifications do is really define a format about how a company should report their financial data to another entity who is collecting it. Usually a regulator, but it could also be anyone who’s in financial data, which lots of people do that. The insurance industry does it, or private equity, or whoever it is. So the standard can be used for many purposes, other than those ones I’ve just discussed. They’re just the main users globally.

Federico Tomassetti:
Yeah. And for example, Strumenta is a small Italian company, but after we talk, I looked into XBRL and I realized that for example, our financial data, that must be public. I mean, this is the case for all companies in Italy, as being also specified with this format. Yeah, so it’s quite widespread. Good, good. And so, I think you already covered what is the format, we is using it, and what is the purpose. So, I think you laid the perfect context to now explain what XULE is.

Campbell Pryde:
Great. So, one of the things that we’ve tried to do, one of our missions, one of our core goals is to try and make a lot of this expert data available to as many people as we can. And in the US predominantly that’s investors, but also we want to make it easier for regulators to use that data as well. And we undertook a number of initiatives. The first initiative we undertook was to basically start collecting all this data, and then make all that data available to folks out of a database. And we made an API and you can look at that. So you can bring the data back and you can look at it in things like Excel, or any other tool which can talk with an API and build an application, et cetera. One of the things that we found with all of this data being collected, the data, unlike traditional data formats, or traditional data that’s collected, financial data is very, very complicated.

Campbell Pryde:
And so, there’s a lot of complexity there. So to be able to collect this data and collect it accurately, can become problematic. So, one of the things that we wanted to do is to try and improve the quality of the data that was being reported to folks like the SEC. And the SEC is a little bit more complicated, because companies have the option to basically go and define their own metadata, which allows them to report things that may be unique to them. So if you can imagine you’ve got a model, you’ve got a data model, that the SEC publishes with the financial regulator, which is the FASBI here, the Financial Accounting Standards Regulator. They basically then define a model and they say people have to report on it, but they do give companies the option to extend that model, so they can then take the model, add some additional metadata to it, and then report to the regulator. And that’s what’s happened in the US. That’s also happening in Europe as well.

Campbell Pryde:
And this is typically done, not by banking regulators, but mainly by securities regulators. So, if you want to understand the data that’s being published, you need to be able to navigate these models. You need to be able to read them and you need to be able to apply logic to them. So, one of the things that was published by XBRL International was something called XBRL formula. And this is a way so you can define a formula relationships between data and also metadata, and use metadata to define those. There are a number of deficiencies in XBRL formula. Well, not deficiencies, just scope areas that it didn’t cover. The standard was also defined as an XML format, so it was a little bit difficult to express and to debug. So, one of the things that we wanted to do was to basically write a rules language for XBRL, so that we could then write a lot of these rules a lot faster, and a lot more efficiently make debugging of those rules easier as well.

Campbell Pryde:
So, we came out with something called XULE, which was really short for XBRL rules. So, that’s what it’s called XULE. And then at the syntax was really designed in a way to make it as easy as possible to write, but at the same time, make it easy to debug and to understand, and to take up as little, to be as precise as we possibly could. And one of the underlying drivers for that was in theory, was to make it possible so accountants could write these rules and they could read them, or any other person who was like a data scientist. They could read it without necessarily having to go and learn a complex programming language to read the stuff. Secondly, we wanted to put a lot of functions into the language so that it would make navigating the XBRL model very easily and intuitive. So, that was the genesis of doing this. That’s why we did it. It was really to get better quality. As we’ll talk later, it actually has a lot more uses than that as well.

Federico Tomassetti:
So, if I understand correctly, a lot of companies and institution provide their financial data in this format. And so, its very interesting to be able to access this data and to process it for very different doctors, I guess for example, for tax purposes, maybe one wants to do some calculation. But also, someone may want to process this data to assess the financial health of a company, or maybe even for internal users to understand how the company is going, maybe calculate indexes and guide decisions based on this data. And I understand that originally was created XBRL formula that is written in XML, so it’s a format that is great for machines, but you instead wanted also to provide a language that is more user-friendly, so that these many different doctors can use it to actually process the data, to query it, to calculate values and get information out of this data.

Campbell Pryde:
And one of the other issues I didn’t really mention is we’ve written it, it’s in the XML, but it expects to some extent the data that is provided to be in an XML format as well. Now, XBRL has kind of moved on and it’s tried to remove or separate more clearly the semantics and the syntax of the model. And so, XBRL specification has evolved, so now you can get XBRL data in a JSON format, or you can get it in a CSV format, has something called inline XBRL, which is effectively XBRL data in a HTML format. So the XBRL formula when it was first written, has a lot of syntactical artifacts in it, which refer to XML type constructs. Whereas when we did the XULE language, we tried to remove as much of that syntactical reference as possible, so that if XULE would work whether the data was submitted in JSON or CSV or HTML.

Campbell Pryde:
And so it works fine on any of those different, if the data is provided in that different syntax, it doesn’t really matter. There’s no real artifacts on the XULE language that directly references them. I would say that there are some, that sometimes you want that data for a specific purpose, but we try to avoid putting that into XULE at all so that we’re not in a position where you need to rely on the format being in a particular syntax.

Federico Tomassetti:
Yeah, because I understand that the people who have the competence to ask the right questions are typically not developers. They’re accountants, people that understand about finance or taxes. So yeah, probably they are not XML experts. And so, if we can provide things in a way that is more user-friendly, we make their life easier. And can you tell us something about the design of the language, its history, who has created it?

Campbell Pryde:
So, I used to work for Morgan Stanley, and one of the issues we’d had there was also similar. We’d tried to write languages to basically process this data. And one of the tricks of this is really processing multidimensional data. XBRL uses a dimensional model. And so, how do you reference data that’s in a multi-dimensional structure. And then, doing that in a way without being buried in hundreds of four loops. So that was one of the underlying design principles, is we wanted to make it easier to pull multi-dimensional data and understand what those dimensions were. And at the same time, XBRL, you can think of it is, this is kind of really before graphs really came about, but it really is a graph.

Campbell Pryde:
The XBRL model is really, it’s based on length, so it’s really linking nodes to two other nodes with arcs or networks, so you can think of that way. So you have two parts to it. You have this network which defines the model, and there’s multiple networks which can define the model. How things add up? How are things presented? What’s the relationship between these two concepts? Is this type of this, is this of this type that? There’s a lot of relationships that are defined in the model. Then at the same time, you have a lot of factual data, which is then reported based on that model. And so, what we tried to do is we said, “We really need a language which can allow us to access the model easily, and allow us to access the facts easily, and then combine them together, and then create different information.”

Campbell Pryde:
And the reason that’s important is because you may have the same data being reported at multiple places, and you might have data reported that one’s a component, one piece of data’s a component of another piece. But if you want to add something up, you don’t want to double count. So there’s all this complexity there, or you only want to add items which have equivalent decimal accuracy. So, to be able to understand which concepts relate to each other, and then what facts were reported to them, and how they’re dimensionalized, you need to be able to manage all that. So, XULE was initially designed. We sat down, I was in Guatemala at the time, and I think Philip (Philip Engel – Data Architect / Data Engineer at XBRL US, Inc.) at the time, he was in California. And then we said, “We really need to be able to address all these different things.”

Campbell Pryde:
And so we just sat down and went through everything that we needed to be able to do to address this. And we ended up with like a 100 page specification.

Federico Tomassetti:
Okay.

Campbell Pryde:
It was very precise. It just dealt with all of the issues. How do we navigate these networks? How do we pull the facts out? How do we handle dimensions? What functions do we need? What properties do we need? What’s the type of stuff that we’re going to need to be able to do. And we had a lot of experience with this in terms of looking at data that the SEC had reported. Looking at the stuff that we had done at Morgan Stanley. And so, we kind of knew what we needed and how we need to pull that data out. So a lot of it was really just designing the syntax of language, and how can we do this in the most efficient way possible? At the same time, without having a syntax that was ambiguous.

Campbell Pryde:
So that was kind of what we sat down and did. So we wrote this document, and then we said, “Let’s try and implement it.” So we build a parser to do that. And then we wrote a processor in Python, and then underneath it used software called RHEL, which is just like an open source XBRL processor. And then we could basically, we would then just basically did the grammar and then we could compile those rules and then we can run those rules. And then we went from there, and then we kind of gradually went through a whole bunch of use cases and refined those. And then we came up with a finalized syntax. And we had a processor that basically worked. And then went from there and refined it. So that was kind of the history of it. So, myself and Philip Engel really spent a lot of time on the design, and then trying to get this thing to work.

Federico Tomassetti:
Good. Good. And is there any project you are aware of that is using XULE, and that your team could be interested to share?

Campbell Pryde:
Yeah. So there’s a couple. One that we have implemented here in the US, is we produce a number of data quality rules, which we publish. And we publish those rules in XULE syntax as the raw syntax, but we also published a compiled version. And then folks can take those and then they can process those rules against a filing. So for example, let’s say I’m a Fortune 500 company, and I’m going to report my filing to the SEC. We try to encourage people to run these XULE rules before they file with the SEC. They’re freely available, so people can run them. And then what that does, is they’ll run them and they’ll see if they’ve got errors in their filing. And if they do, then it gives them a message. This is the issue, this is probably what we should do to correct that.

Campbell Pryde:
Then they can correct that, then run them again. And that there’s no errors and they’ll file it with SEC. Now, this is the initial design purpose of this that we wanted to achieve, because we wanted to improve the data quality of the filings with the SEC. It wasn’t necessarily consistency across filings. So these rules were designed to do that. Now, the SEC had rules as well, that were really more, think them as more like XBRL syntaxtical rules, XBRL valid, but these rules are more like really looking at things like your balance sheet balance, or have you paid out? Does your cashflow calculate the way it should be? Have you added something when you should have been subtracting it?

Campbell Pryde:
So it has a whole bunch of rules, or there’s inconsistencies between these types of elements. We would expect the first quarter, second quarter, third quarter and fourth quarter to add to the total of a year. So all of these things, these rules can be expressed in XULE, and then companies can run them before they file. And then that way they can improve the data quality. So pretty much, I would say the majority of the largest, 2000 filers are using these rules today, before they fall to the SEC. And then we’re working with software vendors to integrate the XULE into the software as well, so that they can all run these things.

Federico Tomassetti:
Yeah.

Campbell Pryde:
So that’s one. The other one is Federal Energy Regulatory Commission. They are also using XULE, and they’re using that for validation, but they’re also using it for rendering HTML documents. So, and this was never the intention of XULE, but it actually works really well for doing this, is so we create HTML templates. And then we take the XBRL data, which is coming in, in an XML format. And then we take XULE, and it basically pulls the data out of the XML. It takes data, it pulls out of the taxonomy that they’ve published, and that plonks it into a template, an HTML template. And then you can just open the HTML. So it’s basically acting like a style, like a FO type style sheet that some of you may be familiar with. But what’s nice about it, is if you make changes to the taxonomy or the model, then the rendered template is going to change.

Campbell Pryde:
If you change the document, then that data will change. And if you’ve got the same data that just reported or multiple schedules, you can change it once and it’ll update it every year. So it’s just using the ability to read the taxonomy and read the instance style data, and basically take that data, manipulate it and use it for something useful.

Federico Tomassetti:
No, I think it is very inspiring to think that so much data is available in this format, so when you create some useful tool, for example, to validate the quality of the data, or to present them in a readable format, you can operate at such a scale and this tool can be immediately useful for, well, probably millions of companies across the world. So that’s very inspiring, yeah.

Campbell Pryde:
Yeah. The other thing we actually did it for, we used it was actually it was just the easiest way to do it. We had a taxonomy and someone said, “Oh, we want to publish this as an open API using the swagger format, or open API format.” And we wrote a XULE rule that would take a taxonomy and output it as an open API, which is kind of interesting too. So it’s just really a way to take data, reorganize it and then output it based on all the input data that you’ve got.

Federico Tomassetti:
Good. Good. And we have talked about XULE, but one project also that you have created is the editor for XULE. And can you discuss about the goals of the project?

Campbell Pryde:
Right. So we initially started creating these rules, and one of the problems became, as the rules became more complicated and you wanted to do stuff, it became very difficult to one, write the rules. You had to know, oh, there’d be an error. You’ve got to compile it. You get a compile error, then you’d have to come back and you’d have to read through all this text and say, okay, where’s the problem here? I can’t find that. And trial and error, trial and error. And it might take, you might write a rule, and it might take you several days to write this thing. And most of the time, 90% of the time was finding, oh, I put a mistake here, or I did a mistake there. So there was no real tie, you had to cycle through a compile cycle. So there was no IDE for it.

Campbell Pryde:
So one of the things we said, this is really bad and this is just taking way too long. So, we contacted Strumenta, who we had worked with in the past, because we had looked to do something, some other project with them. And we knew that they would do a good job, and we knew that they knew what they were talking about. So that was good that we had that contact. And so then, we sat down with them to look, well, let’s start this. We just really need to start with an editor. And then let’s see what we can get that working, so that we can actually start to write these rules. We can know they’re valid before we actually compile them. And then we can save a lot of time in terms of going through this, trying to debug these rules to get them right.

Campbell Pryde:
And so that’s what we did. And so we sat down and we went through, and I can’t remember how long, it probably took like a month or two months, was it? You may remember better than I do, Federico. And we went through and we basically took the document that we’ve got, which is basically defines all the syntax. And then we went basically through and we created this editor, Strumenta created the editor, and then we would go back and test it and then come back, and made us realize the stakes in our document. We had a few of the fixes we found from going through that process, so that was actually very good as well. And what the editor did, is it basically would color highlight everything, color highlight the functions and FactSets and all these different aspects of the XULE language.

Campbell Pryde:
It also did checking against the taxonomy. So if you put a concept in, and it wasn’t on XBRL concept, it would highlight that for you. So then you could basically, typos were a big problem, or sometimes there’d be elements in one taxonomy that didn’t exist on a followup one, because it had been removed. So, checking for all of us became probably it increased our efficiency by at least 1,000%. It made a huge difference. This is one of the things that we did. I am so glad we did because it just saved us so much time.

Federico Tomassetti:
Good. And yeah, and I think when someone creates a language for someone who is not a developer, I mean, the bar is much higher. I mean, unless you build these tools to help them, its difficult because it’s not their profession to learn the language. So, and maybe they don’t use it every day. So, if they can have a tool helping them, you can yeah, help them.

Campbell Pryde:
Yeah. It makes a huge difference. And just, we have the auto complete on there, so that’s really useful. So it can audit the elements, you don’t have to go and look them up. It’s just like so many savings across the board. And we try to, when we write rules without using the editor, it’s almost, if you try and do that, I always go and copy and paste a rule that’s not in the editor, just even to check that the rules right, rather than go to the parse. It’s just easy to go through the editor. And it’s also because it’s color coded. It makes it easier to even think about the problems, and just the process of writing a rule just becomes easier, because your mind, you’re not consuming bandwidth with worrying about getting everything to work.

Federico Tomassetti:
Good. Good. So, at this point we could show a demo of the editor. Okay. [DEMO] Now, at this point we will set up the demo. And okay, now maybe we can discuss how can people get the editor, and how can people learn about it?

Campbell Pryde:
Okay. So, the editor itself is, we decided to write it in Visual Code, which is a Microsoft product. We’re not pro Microsoft or anything like that. But the reason we chose that software to do it, was just because it’s very high usage. That’s probably one of the most commonly used IDEs out there. It’s pretty quick. And a lot of the other editors… You don’t have to pay for it, it’s open source, it’s freely available. And then they have like a lot of libraries and it’s relatively easy to basically to be able to update extensions. So we could add it as an extension into the Visual Pro editor, so that was one of the reasons we chose it. We had used different editors, so we kind of switched to use it as well. And so I think that that was a reasonable choice.

Campbell Pryde:
We don’t have plans to put it into any other editors, just because it’s not really forcing anyone to buy anything to do it. So you can go out and get. You can download it, and then you can just look up XBRL, or you can look up the XULE and in the extension library, it’ll come up and then you can add it. And then you’ve got the XULE extension in VS Code. It’s very, very easy to install, and very, very easy to start using. So, there are a number of users are using it. The Federal Energy Regulatory Commission, they using it to write rules. And their environment, it’s very difficult to set up software, but setting up with XULE editor by far and away the easiest thing to set up, because they were using code, or your Microsoft shop, so I made it very, very easy to use.

Campbell Pryde:
So I haven’t seen anyone have problems implementing it. It seems to work through, to do that install, seems to work through most firewall setups, pretty seamlessly. So, in terms of using Visual Code, I think that was a good choice. I know Strumenta obviously they, I’m not sure how much experience you had before that, but they were very good in sorting things out and getting things to resolve. There was some complexities in using it, particularly checking what taxonomies that were used and what were the elements that were there. So they did a good job doing that.

Campbell Pryde:
So if you want to, you can set it up so that you can point it to different taxonomies so that you can check it. So it’ll work with any taxonomy. So if you’re in Italy and you’re using IFRS, you can use it for that. It’s really not a US thing at all. So if you want to use it with any taxonomy, anywhere in the world that’s been published, then we’ll work with this tool as well.

Federico Tomassetti:
Yeah. No, I think it’s interesting the fact that there are so many aspects to consider when choosing a platform, like for example, you talk about how difficult is it to install certain custom software on certain environments. So maybe using something like VS Code that is a widespread, make it easier and install the extension within VS Code its easy. And maybe also not a reason why in a similar project makes sense, in my opinion, to choose VS Code is that it doesn’t take too many resources. So it takes much less resources that alter ID that our target developers. And the problem is that typically developers has more powerful machines than other users. So if you don’t take these into account, your intended user could have problems actually, using the editor.

Campbell Pryde:
Yeah. And that’s how we found that the Federal Energy Regulatory Commission, we laid it out to a lot of people. They were just really, there were various business users, they were developers. It could be installed on very low end spec machines. It works totally fine. It’s not power hungry in any way, in terms of the memory or the processing power.

Federico Tomassetti:
Yeah. And is there something that you have learned from your users about the language or the editor or something that you decide to change after the first release of XULE or the editor?

Campbell Pryde:
Generally, we haven’t changed very much at all, and time to time things come up where we may need additional functions. One of the things about XULE, is you can define your own functions as well within there, but so if you need a function to do something, you can just define that within XULE. And we’ve written a couple of those, like if we want to check an ICE is valid, which is a security identify a code, or we want to check that kind of thing, we actually wrote functions within this syntax to do that. So, a lot of that kind of stuff, which is very domain specific, you can write those functions. Some, we have added some functions around regular expressions and looking up terms. So we’ve added a function for that in the underlying XULE processor.

Campbell Pryde:
So, and we added a function, I think it was a true function. We had case where we were adding white space in there. And so we actually updated the underlying code that was provided Strumenta just add that functions. It’s relatively easy to add a function into the IDE so it recognizes it and it comes up. So that hasn’t been a problem. So, obviously if we had to do a total redesign of the language, then obviously we would have to republish the IDE. But generally, most of the changes have been things like, we meet to add a new property, because we decided we needed this to be property like a trim on a string. Or we wanted to add regular expression matching, for example, on a string as well. That was just something we didn’t have.

Campbell Pryde:
And that just came up, because as more people wanted to look at the string data, that was just another functionality that we thought we should implement. Most of that functionality, it follows, if you’re familiar with Python, we’ve kind of modeled in a way based on the way that Python handles data. So it includes, its handling of sets and lists and dictionaries is literally identical to Python. So if you’re familiar with Python, a lot of the terms are similar, but we’ve tried to take the complexity out of it. So if someone’s good at doing things like Excel functions, this stuff’s not going to seem foreign to them at all.

Federico Tomassetti:
Good, good. So it seems that your initial design of the language proved to be right, and so you did just incremental changes, of course.

Campbell Pryde:
Very few changes, and it’s interesting is we can handle pretty much, it handles pretty much most situations. Well, pretty much all situations we’ve run into at this point. It hasn’t really been… and if it doesn’t, then maybe there’s a better tool that you should be using. But in general, we try to be as inclusive as we possibly could and get every possibility that we knew about. And so I think that generally has worked. And there’s been no… The only changes to the grammar has been to add a property here and a function there.

Federico Tomassetti:
Good. So it seems that the language is pretty stable. So I was wondering if you had any plan for the future, or a few new feature or upgrade, or do you think that the language and the editor works already quite well and so on?

Campbell Pryde:
One of the things that we are looking at updating is in the XBRL world they’ve released the new specifications for a CSV file. So you can express CSV in XBRL. And this means that some of the files you get are significantly larger. In fact, you could have files which are a constant stream of data, the way that the current, the XULE processor works is it pulls all the data in, and then it runs its validation on the data that is in a file, right? However, if you’ve got large continuous data files, let’s say you’ve got a continuous stream of data coming in, let’s say its coming from stock feeds or whatever it might be, but you want to validate those. And in some cases you can’t really wait until the entire file comes in, and the files are too large.

Campbell Pryde:
So one of the things we were looking at was adding functionality to XULE for that. So it would be doing validation on the stream as it comes in. That has some implications in the sense that you have to write rules a little bit differently. If you want to compare things to aggregate totals, you have to be running aggregators in the background, aggregates stuff as it comes in. So that kind of stuff, we’ve sat down and started to write, how would we write validation rules? And if you’ve got a record came in, obviously you can validate that record. But if that record has got references to other files, then there’s a sequencing type things that you need to address in the language as well. So those are some things we’re looking at updating. That’s one particular area, which is actually going to be quite a bit of work.

Campbell Pryde:
The other thing we’re also looking at is the ability just to bring in modules, I suppose, would be the right word for it. So let’s say someone’s gone and written a bunch of functions in XULE. When you compile those rules, you’ve got to take all those rules and you kind of put them all together into a folder or a folder structure, it’d be nice if you could just reference those, lets say they’re on a URI on the web, someone’s financial functions would be nice just to be able to import those when you compile the rules. So that’s the other thing we’re currently working on at the moment, its like an import function. So there was a couple of things that would be relatively easy to do, compared to the streaming stuff, which is a major deal. So we’re kind of holding off on that for now. But the modules is something that we’ll probably be implementing in the next two months.

Federico Tomassetti:
It’s interesting because I think that both streaming and module side, the kind of things that you may want to add to a language as it becomes more stable, more use, or is part of the engineering phase.

Campbell Pryde:
Right.

Federico Tomassetti:
It’s coherent with the fact that the language is proved to be well-designed, and so it’s getting more and more stable, so nice.

Campbell Pryde:
And one of the design principles is we don’t want to make it too complicated so that you’ve got to have all the setup, and we want people to be able to pick it up and they’ll run a real quickly and then be able to do stuff. So, there is trade-offs there, so we’re constantly very aware that we want to try and make it as easy as possible for people to use. And people can look at it, and we’ve had people comment that, not so much on the editor, but on the language itself, people have given, we’ve made the whole thing is available on GitHub, so you can go to GitHub. So we’ve had a number of comments from people who have been using it, particularly filing agents, a company who runs software. They have implemented it and been using it on their site.

Campbell Pryde:
And if they see issues or they see performance enhancements and they’re giving us stuff as well, so that’s so good feedback. Technically XULE it’s not an XBRL standard. So we’ve kind of launched this as an open source initiative, just to give people a tool that they can use to manipulate XBRL data and produce output file.

Federico Tomassetti:
Good, good. And I think it takes some discipline sometimes to keep the language simple and not just say yes, being too enthusiastic. Good, good. And is there any other tools that work with XBRL that you think our listeners may want to look into, or that is useful to them?

Campbell Pryde:
So we’ve worked also with Altova, some of you may be familiar with. And they’ve implemented XULE in their XBRL package as well, that’s part of their software solutions. So they’ve done it too, so we’ve got that. And then we’re working with a number of other software vendors to help them try and implement it. Most folks who have implemented have been software providers in the reporting industry, particularly in the US. And that’s because they want to run these rules locally in their software, before people follow with the SEC. So we’d published them out. So what a lot of them would do is they might have a gooey interface where people prepare their filing, and then if they make a mistake, the rules are running in the background and it’ll come up and say, “Hey, you’ve made an error here. There’s something wrong. This doesn’t make economic sense.” Or, “We would expect if your liabilities are X and your assets are Y, that this would be within this range, or that your reported amount, which shouldn’t be negative. How could this be negative?” So those rules are kind of incorporated into the software as well.

Campbell Pryde:
So, we were helping a lot of these software vendors provide that language into their software. And ideally the way that the software should work, it’s not always the case, but the way that the software should work is that they can then just upgrade those. We can publish out a new set of rules, and I can just update their software. Then they’ll have new rules in their software. So that’s kind of the way we’re trying to make it work. So it’s not like everyone has to go and re-code rules into their software. That would be, it’s very, very inefficient. So that’s kind of what we’re trying to get to.

Federico Tomassetti:
Yeah. So, through collaboration with the software vendors, a lot of people can benefit from your work on XULE. So that’s a very nice yeah, a really nice thing, I think.

Campbell Pryde:
Yeah. And it’s been very useful for improving the quality of the XBRL data, so particularly for the SEC data. So now that makes that data much more useful for other purposes. So people, as the quality gets better, more people start to use it. People use the data, they find issues in the data. They let us know, we’ll add a XULE rule, that’ll get fixed, more people start to use it, so it becomes the cycle, but we kind of needed to do that. And having this language available, it has made the data quality much, much better, and Strumenta has been really helpful in terms of getting that editor. That’s probably meant that we’ve been able to get 10 times more rules out than we would have been otherwise been able to do. And it also means that the tool can be used by others. We couldn’t give this to people without having a nice IDE to use.

Federico Tomassetti:
Good, good. Well, thank you all for saying nice things about working with us, I appreciate it. And now we are reaching the end of our interview. So, I wanted to ask you if there is anything that you wanted to add, anything I forgot to ask you?

Campbell Pryde:
I think, the only thing I would add si, I think we’re just scratching the surface in some of the stuff we can do. We currently have projects that we’re working on to take this data and then to normalize it, that basically means take those data report in XBRL format, and then repackage it. The tool, the ability to do this as quite powerful, particularly when you’ve got lots of networks and things like you may have heard about this, there’s big talk about graphs and navigating graphs and doing data for that, graph APIs and all this kind of stuff. But XULE was actually, from what I’ve seen, is one of the best tools to do it. We spend a lot of time working out what’s the best way to navigate networks, so we put a lot of functionality into that.

Campbell Pryde:
So if you’ve used tools in the graph databases, they have some of these similar SQL type queries, how you can navigate a graph. We’re spent a lot of time, it’s not like SQL because I will try to make it as close to SQL as we could, so that people who understand that concept of data, can make the mental leap a little bit easier. But then for the graph navigation, we’ve tried to make it a similar way that you’ve navigate a graph node.

Campbell Pryde:
I start at this node, I go down this node. One of the things that we have in XULE, which is really useful, is you can check the properties of attributes on those arcs. And then that can then control which arcs you go down. It also means you can collect all the children of an arc, or the descendants, the ancestors, the siblings, if there’s an implied order on them. So it gives you a lot of functionality to be able to manipulate, or to navigate through those graphs and pull data out and then do stuff with it.

Federico Tomassetti:
Yeah. It seems a problem that is not trivial because it’s so abstract that provide a language, that make it easy. I think it’s challenging. Yeah.

Campbell Pryde:
Yeah. And this is an area which is really going to explode probably in the next five years. Particularly with things like AI is using these networks, and a lot more data is being described in this way. Particularly, and I know people talk about Web 2.0 which came, and how can you navigate this stuff? But I think with things like AI is bringing a lot of the stuff to the fore, because they kind of work the same way. And then building rich models, XBRL is kind of that, make them talk about things like RDF and using it, there’s no real data available. But XBRL actually has a lot of data available now, which you can use and XULE is just a tool which allows you to navigate all of that stuff.

Federico Tomassetti:
Yeah, it’s also data that is a very high quality. Well, also thank you to your quality checks, but I think this makes a difference with a lot of data that is available out there in other formats that is not really well polished or refined, so. Good, good. Well, I would like to thank you a lot for your time and sharing your experience with us. I think it’s been very interesting also, because the problems you’re working on at such a scale where they are really worthwhile relevant problems. So it was really a pleasure. Thank you very much for your time.

Campbell Pryde:
And thank you Federico. Thank you for helping us out so much. I really appreciate all the stuff that you guys have done. And if anyone has any other questions based on this interview, you can go to our website, and we’re more than happy to talk to you or anyone about this.

Federico Tomassetti:
Good, good. And we will add all the links in the article. Yeah. Thank you. Have a nice day.

Campbell Pryde:
You too. Thanks. Thank you.

Federico Tomassetti:
Bye.

The post Interview with Campbell Pryde on XULE a rules language for XBRL. appeared first on Strumenta.

View Details

Langium is a language engineering tool designed to help create DSLs and low code platforms: you can quickly create DSLs with Langium. Langium is lightweight, based on Visual Studio Code and allows you to create a language and an editor in one step.

In this article we are going to take a look at this new tool and create an example language with Langium. This article will be a quick review and tutorial of the tool from the perspective of people that already have experience with parsing. It is not a good introduction to parsing from scratch.

You can find the companion repository for this article on GitHub.

How Langium and Xtext are Similar Langium is yet another interesting creation of TypeFox, a consulting and research company that created Eclipse Theia, Gitpod and Xtext. This new tool is similar to the last one mentioned: they are both tools designed to help creating DSLs. They are also both built upon opensource libraries and tools: Xtext is built upon Eclipse and ANTLR, while Langium is built upon Visual Code and Chevrotain.

An interesting side note is that Chevrotain itself is not a parser generator, it is described as a parsing building toolkit. We reviewed it in our article about parsing in JavaScript. This means that Langium developers created the generator step themselves. Basically, it is all just TypeScript code. Therefore you could potentially alter the process to suit your needs.

Xtext and Langium also have a similar strategy: everything depends upon the grammar declaration of the language you are creating. For example, you can use the grammar to define some validation rules, like allowing a property dogs_name to have only values of dogs that have been already defined in the current file.

How Langium is Different The differences between the two are due to the new development environment and new objectives.

Xtext is a tool built to design both traditional programming languages and DSLs. Instead Langium helps you create DSLs or low-code languages. In other words, Xtext can aid you in creating ecosystems of languages of different complexity, while, at the moment, Langium aids you in creating quicker individual, simple languages. Xtext is part of the EMF galaxy of tools, so it is compatible with other tools that use this technology. This is a powerful and widespread technology that powers other language engineering tools, but it is quite complex to work with.

Visual Studio Code has become the new standard for development environment. In fact, it is also the basis for Theia. There are certainly a myriad of other good options, however it is good enough for most use cases.

So it is the safe choice for development tools. Since Visual Studio code uses TypeScript, the whole Langium project also uses TypeScript. This is to avoid having a codebase in multiple languages or the need to integrate different tools or runtimes. This is also why Langium does not use a parser generator, but it just creates code.

Is Langium the New Xtext? An interesting result of these differences is that you can more easily integrate different languages created with Xtext or other technologies. That is because Xtext follows a classical structure of language execution (i.e., parsing creates an AST, which is based on the Eclipse Modelling Framework, etc.). So, if you are a language engineer, you know where to put your hands on to integrate different languages or tools that use EMF.

On the other hand, with Langium this is harder to do, because it all relies on TypeScript. For instance, the AST is based upon TypeScript interfaces. So, everything is just TypeScript code, that is specific to Langium.

This is not a flaw per se, because this makes it easier and quicker to create a DSL with Langium. And this serves better the objective of Langium. So, Langium does not replace Xtext, but it serves a different purpose.

Or to put it in another way, the competition of Xtext is a series of custom designed tools that cover parts or the whole process of creating language tools that are either more efficient or better integrated with the rest of the project. For instance, instead of using Xtext in your workflow you could build just a parser used in a parsing service or opt to create a custom transpiler from scratch.

What Langium Can Replace The alternatives to Langium are custom environments/applications (e.g., a custom Desktop app) used by non-developers and developers alike. Some people might scoff at the idea, because Langium is based on VS Code which is an IDE, that is to say a tool for developers.

However, VS Code is not complicated to use. In fact, it does not look more complex than a standard text editor. So, it can be used by non-developers. On the other hand Xtext looks very complicated to the untrained eye. To be fair, Xtext allows you to create custom distributions of the IDE with just the elements of the UI that you need, but that requires some work.

Now that you understand the idea behind Langium, let’s see it at work.

Getting Started With Langium The first pleasant surprise is how easy it is to start working with Langium. You just need to install the corresponding yeoman generator:

npm install -g yo generator-langium

And then you can launch it:

yo langium

This will start the generator that will create the skeleton of your language just by answering a few questions. This process will also create a langium-quickstart.md file with the information about the structure of a langium project.

Unfortunately that is also the extent of the documentation that is available for Langium. This project is still in the early stages of development, so this is what happens at the beginning. At this point Langium is just a few months old.

The generator creates a readme file that explains the structure of a Langium project. The generated project also comes with a few example files, like one for a language definition and validation. There are also language examples in the github repository of the project. Looking at the arithmetic example, you can see that Langium does not support easily handling expressions, like ANTLR. You would have to create a cascade of ever-more specific expressions rather than one rule expression.

The issue of limited documentation is somewhat mitigated by the fact that Langium is built on opensource and readily available components. So, you can get started by reading other documentation. For instance, the npm package documentation says:

The grammar declaration language of Langium is very similar to Xtext. Please follow the Xtext documentation to learn how to use this language.

So, if you need a reference for the grammar language, you can look for it by searching for the Xtext grammar language. Notice, though, that while the rules are largely the same, there is not a perfect correspondence. For instance, the peculiar until token of Xtext is not supported in Langium.

The Langium Workflow Everybody that has already created a language server for Visual Studio Code will also recognize the same basic components. For instance, the language-configuration.json file is the standard VS Code file that contains definitions for syntax highlighting. It contains the definitions used to enable syntax highlighting for elements like comments or brackets. To know more about such elements, you could follow our tutorials on creating language servers for Visual Studio Code, such as Integrating Code Completion in Visual Studio Code – With the Language Server Protocol.

Basically, to take advantage of Langium at this stage you should already have some experience with language engineering. Otherwise you must be ready to dig some information from other sources or resign to trial and error. The alternative is to keep reading this article.

Another thing that you may want to do is installing the Langium VS Code extension available for VS Code. This extension adds language support for langium files themselves, such as syntax highlighting, autocompletion, etc.

Technically, with Langium you are just developing a VS Code extension, so the workflow should be familiar to any VS Code extension developer.

You first run this command to observe and automatically compile your TypeScript code:

npm run watch

You use this command specifically to run Langium:

npm run langium:generate

This will make Langium do its magic and generate the code from your grammar definition and code.

Then you use F5 to launch the extension.

The Structure of a Langium Project The langium-quickstart.md file explains the structure of a Langium project.

You can safely ignore the files package.json, extension.ts and main.ts for most of your Langium projects, since they essentially contain the code to integrate Langium into VS Code.

You might want to take a look at the file language-configuration.json. This is the language configuration that is used to enable syntax highlighting in VS Code for elements such as blocks of code or comments.

The file <language-name>-module.ts is used to set up the Langium project. You can add modules or language services here that will perform some operations on a language file. For instance, a module to serialize or validate a file.

By default, the Langium generator creates a validator module that checks that a file is valid according to the rules of your language. For example, your language might require variable names to start with a capital letter. This default module is in the file <language-name>-validator.ts.The validation is done with standard TypeScript code.

The main file you will work with is the grammar file: <language-name>.langium. This contains the definition of your language. For the most part this defines the parser. However, you can also use it to define the type of the corresponding node in the AST. For instance, the generated example file contains this rule:

terminal INT returns number: /[0-9]+/;

This makes sure that the type of an INT element is a number, rather than the default string.

Creating Our Language: Lexer In our example, we generated a Langium project with the name LangiumGame. If we open the file langium-game.langium we can see the rules for the default example language. This is a language for creating greetings to people. This is not very useful, so we are doing something different for our example. We are going to create a language that defines simple games, which is clearly much more useful and productive. Okay, it is equally useless, but at least it is uncommon.

Our Langium file starts with the name and the tokens (terminals) that will not appear in the AST. These tokens are indicated using the command hidden.

grammar LangiumGame hidden(WS, COMMENT)

We choose to have just one type of comment. We ignore that and whitespace.

terminal COMMENT: /§[^\n\r]*[\n\r]*/;

Our comment starts with the character § (section sign) and ends with a newline. That character might not be on your keyboard, so now you know how I feel when I see the ~ (tilde) character.

In Langium, lexer rules definitions (i.e., terminals or tokens) are delimited by the slash character. Aside from that, the definition of rules is intuitive and depends on the typical regular expression format.

The terminal COMMENT, and all the terminals, are put at the end of the grammar file, after all the parser rules. So, in the same file you have the grammar name and list of hidden tokens at the beginning, then the parser rules and finally the lexer rules.

terminal NEWLINE: /[\n\r]+/; terminal ID: /[\_a-zA-Z][\w\_]*/; terminal WS: /\s+/; terminal INT returns number: /[0-9]+/; terminal TEXT: /"[^"]*"|'[^']*'/;

Creating Our Language: Parser The parser rules are very easy to understand and to define. That is because, just like for terminals, they follow the typical rules of the EBNF format. However, there are a few things worth mentioning.

``` Game: 'Game' name=ID description=TEXT NEWLINE (rules+=Riddle)+ (suggestions+=Suggestion)* ;

Riddle:
'Riddle' name=ID question=TEXT 'Answer' answer=TEXT NEWLINE ;

Suggestion: 'Suggestion' TEXT 'for' ID 'at' 'time' minutes=INT ':' seconds=INT NEWLINE ;
```

The rule game captures a game definition with a name, a description and a series of rules and suggestions. The first peculiarity is that you need to give a label to everything that you want to easily access later in the AST.

For example, the ID of the game will be accessible with the property name in the AST. The terminal NEWLINE or the string ‘Game’ will not be available in the AST. You could get the whole text matched by the rule, using the property $cstNode?.text, but there does not seem to be a simple way to access terminals directly.

This happens because these labels are used in the TypeScript interface that corresponds to a node for that rule. For instance, this is the interface for the rule Game.

export interface Game extends AstNode { description: string name: string rules: Array<Riddle> suggestions: Array<Suggestion> }

The second important thing is that the first rule that you define is the main rule. This rule must capture the whole content of the file. In our example, the Langium-generated parser will try to parse the file with the rule Game.

Our little game is essentially a series of questions with one correct answer. The format also supports delivering suggestions after a specific time to a player that is stuck.

The format works well with trivia games, but would also support more complex logical games, given that it allows for suggestions. It would not work that well for escape rooms, and the like, given that it does not support defining a setting or discovering objects.

A Small Bug In theory, this simple grammar would work fine. However, it does not. The issue is a bug triggered when your grammar does not uses cross-references. This bug has already been solved in the development version of Langium, but this has not been published yet.

In fact, if you generated the grammar at this point you would see that the generated/ast.ts file would contain this line:

export type LangiumGameAstReference = ;

This is not valid TypeScript, which generates an error at compile time. You could fix this bug by manually changing this line in:

export type LangiumGameAstReference = never;

Or by adding a dummy rule with a cross-reference like the following one.

Description: 'Description' desc=[Riddle];

Now, you might ask, what is a cross-reference? This is a neat feature of Langium (and Xtext) grammars. Graphically, it is indicated by using two enclosing square brackets. In the previous example it is used in desc=[Riddle].

This is a constraint which allows for the property only already defined values of the specified type. To better understand what this means, let’s change the rule Suggestion in our grammar:

Suggestion: 'Suggestion' text=TEXT 'for' riddle=[Riddle] 'at' 'time' minutes=INT ':' seconds=INT NEWLINE ;

Now, the property riddle in a Suggestion can only reference a previously defined Riddle rule.

If you now try to reference a non-existent Riddle, you get an error.

For cross-reference to work, the referenced rule should have a property name. If you recall, this is how our rule Riddle looked like.

Riddle: 'Riddle' name=ID question=TEXT 'Answer' answer=TEXT NEWLINE ;

This is a feature that technically belongs to the validation phase, rather than the parsing phase. However defining it directly in the grammar makes it easy to use and to understand. This is what we meant when we said that Langium is heavily dependent on the grammar declaration.

Cross-references are a cool example of how well it integrates with Visual Studio code. If you define a cross-reference you automatically get autocompletion for the specific item. In our example, you will get suggested Riddle names when writing a Suggestion.

Validating Our Games Speaking of the validation phase, Langium comes with a default validation module. In our example the validation is contained in a file called langium-game-validator.ts.

First, you need to associate the validation methods to the specific type.

/** * Registry for validation checks. */ export class LangiumGameValidationRegistry extends ValidationRegistry { constructor(services: LangiumGameServices) { super(services); const validator = services.validation.LangiumGameValidator; const checks: LangiumGameChecks = { Game: validator.checkDescriptionIsLongEnough }; this.register(checks, validator); } }

In this case, we make sure that all Game objects are checked by the method checkDescriptionIsLongEnough.

Then we define the aforementioned method.

``` /* * Implementation of custom validations. / export class LangiumGameValidator {

checkDescriptionIsLongEnough(game: Game, accept: ValidationAcceptor): void {
    if (game.description.length < 50) {                        
        accept('warning', 'The description of the game should be longer.', { node: game, property: 'description' }); 
    }
}

} ```

We want all game descriptions to be at least 50 characters long. Since this is just a suggestion, rather than a requirement, you get only a warning in case of failure. You can also see that we can explicitly indicate to which property the check refers to.

Summary We have just seen a simple introduction to Langium. It is a new language engineering tool for the VS Code world we live in. The fact that it is built on top of VS Code makes it a great choice for designing and delivering simple DSLs or formats.

Despite having still a few rough edges and bugs, it is already a tool to keep an eye on. It is amazing how useful it is after reaching just version 0.1.

It is particularly useful if you are already familiar with Xtext or VS Code extension development, but it can be a good choice even for people that just have some experience in parsing.

If you have an experience in none of these things, it is probably too early to use it. On the long term it could be a great choice for beginners, since it does not need to include a parser generator. It just relies on TypeScript code. This simplifies the workflow and distribution of the parser.

We are not there yet though, given that it lacks independent documentation. It also does not have a beginner-friendly feature like support for easily handling expressions with one unified rule like ANTLR. This is not a big problem for formats or declarative DSLs, but it makes it a bit more complicated starting out for people unfamiliar with parsing patterns.

As usual, you can find the companion repository for this article on GitHub.

The post Quickly create DSLs with Langium appeared first on Strumenta.