The value of validation – interview with Dorien Elleboog

Engineers use software because computers can perform repetitive analyses and design calculations thousands of times faster than they could manually with pen and paper.
In theory, every design of the same project will lead to the same result. But if you ask 10 engineers to design the same project using software, there will be variation in the results.
We interview Dorien Elleboog, support engineer at BuildSoft, about the value of software and calculation validation.

Starting at the beginning: what is validation?

In its broadest sense, “validation” means demonstrating through verification that a device, system, mathematical model or instrument is capable of delivering the intended results with a high degree of certainty.
If we project that concept onto structural analysis software, it boils down to checking, on the one hand, whether the data has been entered correctly into the software and, on the other hand, whether the software uses valid methods for its analyses and design.

Why is validation so important?

Never loose out of sight that calculation software is a tool. You should see software as a calculator that runs certain inputs (= the geometry, the loads) through certain algorithms (= such as the mesh and analysis settings) and thereby generates a certain output (= results). No check is made on the consistency or logic of those inputs. If you enter garbage, you will get garbage as a result.
At no point is Diamonds, or any other calculation software, capable of interpreting or judging that input or output.

Who performs this validation?

In my opinion, checking whether the input of the data leads to the desired (expected) result is personal and project-specific.

• Let me explain the word “personal” in more detail. Suppose I enter certain input into software, check the results and conclude that they are meaningful. But what is that statement worth if I don’t share what input I used and how I entered it? Because without that information, no one can learn how to enter that input correctly.
• ‘Project-specific’ because every project has its own issues. The validation of one project is not necessarily valid for another.

So I hope that not only the users of our software validate their own calculations, but also those of their colleagues. Because those discussions, the sharing of insights and knowledge, are invaluable.
Furthermore, you should not forget the external control mechanisms, such as Kiwa and Seco. Kiwa is a company specialising in the certification of all kinds of processes, including calculation software. Seco is more likely to be called in to check existing calculations for important projects.

Do you validate your software yourselves?

Absolutely! For each release, new and existing features (such as calculation methods for analysis and design) are tested.
I have long searched for a way to publish these tests in a simple manner, and a selection of these validation tests is now available on our support website.
We have tried to cover a wide range of calculations: loads, concrete, steel, fire, static and modal analysis. This list is a work in progress. Whenever I come across a calculation example that could add value to the list, I add it.
The examples were sometimes taken literally from the literature, sometimes inspired by it, and sometimes they are pure manual calculations.
I hope that these examples will give our users more insight into how our software works.

Manual calculations: are they a first step towards validation?

It is good practice to have an idea of the order of magnitude of the results to be calculated. A manual calculation is certainly appropriate here.
The aim of a manual calculation is not to arrive at exactly the same result as the software, but rather an “acceptable” result.
As a general rule, you can assume that if the results from a manual calculation deviate by less than 10% from the software result [1], both the software and your manual calculation are correct. If the results deviate by more than 20%, you have made a mistake somewhere. Assuming that the manual calculation is correct, the error is most likely in:

• an incorrect input
  E.g.: incorrect boundary conditions, incorrect concrete cover, etc.
• not understanding how (default) settings work
  E.g.: settings for the mesh
• not understanding how the software works
  E.g.: using unnecessary eccentricities
• not understanding the limitations of the software
  E.g.: modelling volume elements in software that is not suitable for this purpose.

Below are some specific examples in Diamonds to clarify this:

1. When the mesh is generated, the mesh algorithm in Diamonds detects whether two surfaces are adjacent to each other or not. If two surfaces are adjacent, their common edge is coloured pink. If a surface is not adjacent to another surface, the relevant edge is coloured green. A green edge is therefore equivalent to an edge where no forces are transferred.
   Suppose I create a Diamonds model in which the walls of two floors are not perfectly aligned. The mesh algorithm sees that the walls are not neatly aligned and colours the edge between the two walls green… when it should actually be pink because I want those two walls to be connected.
   At no point will Diamonds say, ‘Hey Dorien, watch out, because I think these two walls are not nicely aligned!’ Because for Diamonds, these are not two walls above each other, but two surfaces that are somewhere in a three-dimensional space.

2. Diamonds calculates the deflection of a slab, but cannot check whether that deflection is acceptable or not. This is due to the span length: if the slab has no preferred bearing direction and an irregular shape, as in the example below, there are many possibilities for the span: the yellow, orange, blue, green, etc. lines. As a structural engineer, I can make a meaningful choice here, but Diamonds cannot. This is because the program only sees lines that are part of a floor but cannot interpret the shape these lines are forming.

Manual calculations, are they the holy grail?

No. I think that every calculation method or tool has its limitations. This also applies to manual calculations. Calculation software can take into account effects that are neglected in manual calculations… Even though these effects cannot be neglected in practice! So caution is required.

Are there other ways to perform validation?

Judging whether a result is meaningful or not requires some experience. You learn that intuition by being critical, simplifying things and analysing them.
Manual calculations will certainly help, but the added value of training, an internal knowledge base, internal procedures and support from more experienced colleagues should not be underestimated. As an engineer, you have to be open to learning throughout your entire career.

“An engineer is forever a student”

At BuildSoft, we provide training on how to use the software at least three times a year (go to the calendar). This ranges from clicking the right buttons to thoroughly researching Eurocode to find out why it is the way it is. We also have an extensive knowledge base and online training courses via e-learning.

Do you have any tips for our readers?

I think the most underestimated feature of Diamonds is the mesh configuration. See the example of a few questions above.
All too often, I get the impression that users think: ‘Diamonds calculates, so the results must be meaningful’. During the design process, however, some oddities come to light and the question ‘Where does that come from?’ arises. I fear for the inattentive user who doesn’t notice the oddities in the results… While 98% of the problems with a model are actually already visible in the mesh. So I can’t repeat it enough: ‘check the mesh’. Practice makes perfect.

Literature:
[1] Validating the results of structural engineering software, Cliff Swinger (Structure magazine article)
[2] Quality assurance for structural engineering firms, Cliff Swinger