Connectors (BI, Kafka, Spark)

  1. Get schema validation "feedback" in Kafka Mongo Sink Connector

    Objective :

    We want to be able to validate that data matches some requirements. We would like to to perform this data validation by adding a JSON schema in Mongo (such as it is described here : https://docs.mongodb.com/manual/core/schema-validation/).

    Problem is that current implementation of the current Mongo DB Kafka Sink connector does not implement the required elements to benefit from features brought by this KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors

    So if we define such a validation on Mongo, if a message has a value that does not match the definition, it would not go in the dead letter queue, and the…

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Kafka Connector  ·  Flag idea as inappropriate…  ·  Admin →
  2. Kafka source connector once only semantics

    Added as a suppport case here : https://support.mongodb.com/case/00634630

    When using the connector as a Source, i.e we capture change streams from the Source Mongo DB and stream that to a Kafka endpoint.

    Imagine these are updates on financial transactions in mongodb and they are NOT tolerant to
    1) missed data and
    2) duplicated data
    in that order.

    So, we need to make sure that the Change Streams that we are observing(matching) on, are delivered once and exactly once to the Kafka pipeline. (Blog on the same : https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/). If exactly-once semantics are enabled, it makes commits transactional by default.

    2 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Kafka Connector  ·  Flag idea as inappropriate…  ·  Admin →
  3. Dropbox

    Lead with the drop that will very allow. Permit to be inside and all the time useful !!
    stay tuned.

    1 vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Kafka Connector  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Connectors (BI, Kafka, Spark)

Categories

Feedback and Knowledge Base