How to generate a BPMN flowchart from a whiteboard photo, using AI

Users
1

As some of you know, I’m using bpmn-js to build a flowchart generating tool, BA Copilot.

I’ve been getting really valuable feedback from you on this forum, so I hope you don’t mind me posting more stuff as I build it! :pray:

I’ve added the functionality to generate editable flowcharts from whiteboard photos:

Here’s a demo video:

It also works with photos of handdrawn flowcharts.

What do you think?

2

Hi,

Great to see another usage of bpmn.io, this time with AI. Did you build any custom extensions in the process which could be contributed back to the community?
What were the main challenges in your project? How did you overcome them?

3

Thanks.

No I didn’t build any custom extensions.

The main challenge was/is in the bpmn modeller interactivity.
It sometimes gets ‘stuck’ i.e. the selected line or element won’t move.
I haven’t overcome it fully yet!
I suspect it’s an implementation issue in my code, rather than an issue with bpmn-js itself.
I say that because I can’t replicate the bug in the Camunda web modeler.

1 Like
4

Are there any errors in the console that might help fix that issue?

5

To provide a little feedback: The guys at our company were intrigued by this and gave it a couple of goes. Firstly a (very basic) hand drawn process flow on paper, photographed and entered for use. It recreated it but missed one gateway and one arrow - but it did label the swimlanes more appropriately, which was cool.

Secondly, we tried a png export of a draw.io diagram to see if it can recreate it. Unfortunately, it was either too complex or we didn’t prompt it well (we had asked to recreate the process flow “in the attached.” with the image attached), because the results were like a completely different process flow. With only two tests allowed on the free account, we weren’t able to try a new prompt, and it failed.

6

Thanks for this feedback.

Yes, sometimes the generation is too ‘fuzzy’ i.e. far from the exact description the user made.

This works well for users who are less prescriptive e.g. their prompt is ‘here’s a meeting transcript of our purchase-to-pay process. Create a flowchart of it’.

It’s not good enough for your use case though (picture to flowchart), so I’m working on improving it.

I’ll message you when it’s complete :+1:.

7

I’ve updated the bpmn-js package and it seems to be working now!

If it reappears, then I’ll check the console, thanks!