We all know that when two tasks are in the same pool, they are connected by sequence flow, and when one task is moved to another pool, the connection between them automatically becomes message flow, and vice versa. I want to implement similar function for custom elements, what should I do?
As you’ve figured out already all of this is made possible through rules: https://github.com/bpmn-io/bpmn-js/blob/master/lib/features/rules/BpmnRules.js#L400
Create you own rules and return the type of connection based on their source and target parents.
Edit: As mentioned in How to automatically connect two custom elements due to a bug overriding rules might not work in some cases. In the case of moving elements it’s probably going to work.
Furthermore you can have a look inside the ReplaceConnectionBehavior on how the connection got replaced.