Project Feline is a sophisticated data generation tool designed to populate Kafka topics with realistic, simulated data. Utilizing the robust capabilities of the Java Faker library, this service efficiently creates a diverse range of fake data tailored to various testing and development needs.

• Realistic Data Simulation: Leverages Java Faker to produce a wide array of realistic data types, enhancing testing accuracy and reliability.
• Kafka Integration: Seamlessly integrates with Kafka topics, ensuring smooth data flow and efficient testing of Kafka-based applications.
• Customizable Data Sets: Offers flexibility in data generation, allowing users to tailor data according to specific schema requirements.

• Application Testing: Perfect for developers needing to test applications with realistic data scenarios.
• Performance Analysis: Assists in evaluating the performance of Kafka implementations under varying data loads.
• Data Pipeline Validation: Useful for validating data pipelines and ensuring the integrity of data processing and transformation.

Note: To successfully run this project, Java 17 or later must be installed on your system. Ensure that you have the appropriate version of Java before attempting to run the project.

  KAFKA_BOOTSTRAP_SERVERS: for kafka bootstrap servers - default = localhost:9092
  KAFKA_SCHEMA_REGISTRY_URL: for schema registry url - default = http://localhost:8081
  CONFIG_FILE_PATH: for data generator config file path - default = ./config/config.json

Additional Kafka Connection Properties

Kafka connection properties should be prefixed with KAFKA_. The part after this prefix is transformed to match Kafka's connection property names. For example:

• KAFKA_SASL_MECHANISM environment variable corresponds to the Kafka property sasl.mechanism
• KAFKA_SECURITY_PROTOCOL environment variable corresponds to the Kafka property security.protocol

  ./run.sh

[
  {
    "target": "kafka",
    "topic": "customers",
    "key": {
      "_gen": "with",
      "with": "#{Internet.uuid}"
    },
    "value": {
      "profile": {
        "name": {
          "_gen": "with",
          "with": "#{Name.full_name}"
        },
        "job_title": {
          "_gen": "with",
          "with": "#{Job.title}"
        },
        "blood_group": {
          "_gen": "with",
          "with": "#{Name.blood_group}"
        },
        "creditCardNumber": {
          "_gen": "with",
          "with": "#{Finance.credit_card}"
        }
      },
      "address": {
        "country": {
          "_gen": "with",
          "with": "#{Address.country}"
        }
      },
      "contact": {
        "email": {
          "_gen": "with",
          "with": "#{Internet.email_address}"
        },
        "phone": {
          "_gen": "with",
          "with": "#{PhoneNumber.phone_number}"
        }
      }
    },
    "config": {
      "throttle": {
        "ms": 5000
      }
    }
  },
  {
    "target": "kafka",
    "topic": "products",
    "key": {
      "_gen": "with",
      "with": "#{Internet.uuid}"
    },
    "value": {
      "name": {
        "_gen": "with",
        "with": "#{Commerce.product_name}"
      },
      "color": {
        "_gen": "with",
        "with": "#{Commerce.color}"
      },
      "price": {
        "_gen": "with",
        "with": "#{Number.number_between(40, 1000)}"
      }
    },
    "config": {
      "throttle": {
        "ms": 2000
      }
    }
  },
  {
    "target": "kafka",
    "topic": "purchases",
    "key": {
      "_gen": "with",
      "with": "#{Internet.uuid}"
    },
    "value": {
      "customerId": {
        "_gen": "matching",
        "matching": "customers.key"
      },
      "productId": {
        "_gen": "matching",
        "matching": "products.key"
      },
      "timestamp": {
        "_gen": "with",
        "with": "#{current_timestamp}"
      },
      "quantity": {
        "_gen": "with",
        "with": "#{Number.number_between(1, 100)}"
      }
    },
    "config": {
      "throttle": {
        "ms": 12000
      }
    }
  }
]

Example enum values for Dune.quote(Quote) and Dune.saying(Saying):

• Quote: Can be one of GUILD_NAIVGATOR, EMPEROR, PAUL, THUFIR, JESSICA, IRULAN, MOHIAM, GURNEY, LETO, STILGAR, LIET_KYNES, PARDOT_KYNES, BARON_HARKONNEN, PITER, ALIA, MAPES, DUNCAN, YUEH
• Saying: Can be one of BENE_GESSERIT, FREMEN, MENTAT, MUADDIB, ORANGE_CATHOLIC_BIBLE

Note: Current support is limited to enum, int, long, double, boolean and String input parameters. Please ensure you use the correct type for each command as per your data generation requirements.

1. enum values are case sensitive
2. example config for int value: #{Number.number_between(1, 100)}
3. example config for String value: #{Internet.safe_email_address('abc')}
4. example config for double value: #{Commerce.price(5.5, 6.5)}
5. example config for long value: #{Number.random_double(10, 10000L, 100000L)}
6. example config for boolean value: #{Lorem.characters(true)} | #{Lorem.characters(false)}

Important Node on Function Support: The current implementation does not support direct invocation of functions like Faker.botify(String)