NIFI-13402: Add ability for Python source and transform processors to output multiple FlowFiles

[mattyb149]

Summary

NIFI-13402 This PR adds 2 new interfaces, MultipleFlowFileSource and FlowFileTransformMultipleOutput, for source and transform processors to be able to output a collection of FlowFiles rather than a single FlowFile.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Although I appreciate the interest in pursuing these changes, I currently remain opposed to further extension of the Python Processor API to support this type of use case with multiple FlowFiles.

Python Processors have inherent scalability challenges due to the decoupled nature of the Python process itself, and the socket communication between Python processes and the JVM.

Extending the Python API introduces new maintenance concerns on top of the foundational issues.

I think it would be helpful to discuss this in Jira before proceeding with code changes, so I'm going to vote against this addition for now.

[exceptionfactory]

On a procedural note, as this has a direct relationship to public interfaces designed for extension, it should go through the NiFi Improvement Proposal process. That would also provide the opportunity for additional discussion before moving back to implementation.