An Experiment in Model Checking

Many of you know my love for the Model Checker. I wasn’t there for its birth, but I spent a lot of time ushering it through its childhood and into its adult life (adult for software). With all the changes coming from MCPs and the new Model Analytics available on Forma, I do wonder about its future.

A father will always worry…

Simple Check Test and AI Making Crap Up

As more of an experiment that had been in my brains for some time, I recently dumped all the sample checkset XMLs I could find in a chat with C. C seemed to know right away what I had provided and was willing and able to build a check for me.

The XML structure should be easy, but as many of you know, the Model Checker can be particular about how checks are made.

I wanted the first test to be straightforward, so I asked “Tell me about any fire rated walls under 8 feet.” At first, C just displayed the XML, so I asked for the file and it served it up.

Made some sample walls in Revit, loaded up the new checkset, and it worked! Mostly.

This was from way back, when we changed the schema of the checks to have an ID for each individual check, filter, etc., among other additions. The samples I provided were pre-update, so I get why C missed that.

However, when I asked about the warning Model Checker gives, C assured me “That’s a known quirk with Model Checker,” and then went to tell me the problem was with the Windows vs. Unix line endings.

First of all, don’t tell me about quirks with Model Checker.

Secondly, and this is nothing you haven’t seen discussed elsewhere, the confidence that C has that they are right when they are wrong continues to astound me. There are guardrails you can put up, but I feel like there should be a Dunning-Kruger dial in the application settings.

I continued to educate C, uploaded all the current examples I could find (“That’s a goldmine!”) and C listed all the new things it learned.

But, with something so niche as the Model Checker, it still only knows what I tell it.

Let’s Try Something Trickier

For my next test, I asked it “I would like an error on any door or window that has a Fire Rating that doesn’t match the Fire Rating of the host wall. Can you do that?”

I knew this was doable, albeit convoluted, but C had no examples of it. At least this time, it was up front and told me the limitations without making something up. Still made some very bold claims about “What teams typically do instead…”

So, another bit of education. I linked to a blog post I wrote years ago when that feature was introduced. C was excited that it could be accomplished, but had zero knowledge of the backend XML for it. Rather smartly, it asked me to make a sample. It didn’t need a big sample, just something that had filter examples so it could see the schema potential: Category="HostParameter" as one word, with Property and Value pointing to the parameters I was comparing.

This One Confuses Humans As Well

Last round was the dreaded OR joiner. The check that C gave me looked at Doors and Windows and their Host’s fire ratings. But it was 2 separate checks. I wanted a single check.

C knew that it needed the OR to combine them, but it did not understand that the OR essentially creates a whole new check… both sides of the OR need to be completely functioning checks with its own filters etc.

What I found interesting is that C made the same error that many humans do when first using OR to bridge checks. I understand the why they assume what they assume, but that’s not how Model Checker works.

Luckily, I have a lot of experience telling folks about this, and a quick chat got C straightened out. The end result is a working checkset that flags any door or window instance whose Fire Rating doesn’t match the Fire Rating of its host wall.

But Why?

I don’t know if I will keep feeding more checks into C to make them better, but the groundwork is there and I think the potential is there to help speed up checkset creation.

The work I am doing with C keeps reinforcing the idea that it has the potential to be a great partner, but I cannot see it being a replacer.

Like I stated before, I don’t know the future of the Model Checker, so this truly was just an itch I had to scratch.