Creating a new element using AutoPlace should create a connection as well. First of all you can simplify the rule to:
if (isCustom(source) || isCustom(target)) {
return { type: 'custom:connection' };
}
Please try to debug AppendShapeHandler and check if it’s trying to connect the elements and why it doesn’t work.