feat(connectors): add MySQL source polling connector#3568
feat(connectors): add MySQL source polling connector#3568tusharagrahari wants to merge 4 commits into
Conversation
|
Thanks for the PR. It is labeled Slash commands (own line, regular comment) move it around the queue:
See CONTRIBUTING.md for details. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3568 +/- ##
=============================================
- Coverage 74.02% 44.91% -29.11%
Complexity 937 937
=============================================
Files 1247 1245 -2
Lines 127567 112197 -15370
Branches 103435 88110 -15325
=============================================
- Hits 94427 50391 -44036
- Misses 30103 59073 +28970
+ Partials 3037 2733 -304
🚀 New features to boost your workflow:
|
atharvalade
left a comment
There was a problem hiding this comment.
Hey @tusharagrahari, thanks for the contribution and for taking the time to put this together.
A couple of process things before we dig into review:
-
Issue approval: I can see you created #3445 yourself, which is great. However, per our CONTRIBUTING.md, new functionality needs maintainer approval on the issue before coding begins (a
good-first-issuelabel or an explicit comment). Connectors are also listed as a high-risk area that needs a design discussion in the issue first. We typically ask first-time contributors to start with something labeledgood-first-issueto get familiar with the repo's workflow and CI, then move on to bigger features like this. -
Failing CI: Could you take a look and get those green?
Not saying this won't get merged, the implementation looks good to start with. Just want to make sure we follow the process so there are no surprises down the line.
| if !processed_ids.is_empty() { | ||
| self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids) | ||
| .await?; | ||
| } |
There was a problem hiding this comment.
mark_or_delete_processed_rows runs inside poll_tables() before the messages reach the runtime's producer.send(). If the process crashes or publish fails after this point, those rows are gone from MySQL but never delivered to Iggy. It would be better to return the pending IDs in ProducedMessages and let the runtime execute the delete after confirmed send.
There was a problem hiding this comment.
Valid concern — I hit this too. The delete-before-send window exists because the SDK has no post-send ack primitive, not a conscious design choice. I modeled it on postgres_source as the reference, but that's context not a justification. Proper fix needs a post-send hook on the Source trait — SDK-level change, out of scope for this PR.
|
Hi @atharvalade, thanks for the detailed feedback! On the process side — I did reach out on the Discord community before starting, where I got clearance to proceed with the connector work. I understand that might not substitute for a formal maintainer approval comment on the issue itself, and I'll make sure to get that explicitly documented going forward before picking up anything in high-risk areas. On the CI failures — on it, will get those green shortly. Also working through the inline review comments now. Will push the fixes once the CI is sorted. |
1c4d3d9 to
dd4ea5c
Compare
|
/ready |
Which issue does this PR address?
Relates to #3445
Rationale
MySQL is a widely used relational database with no existing Iggy source connector. This adds incremental table polling so users can stream MySQL rows into Iggy topics without CDC infrastructure.
What changed?
Before this PR, there was no way to source data from MySQL into Iggy.
This adds a mysql_source connector plugin supporting incremental table polling via a configurable tracking column. Rows are streamed as JSON, raw bytes, or text depending on the configured payload format. Post-processing options (delete after read, mark as processed) and custom SQL queries with parameter substitution are also supported. CDC (binlog-based) is out of scope and planned as a follow-up.
Local Execution
AI Usage