schema

BigQuery:  Efficient Data Warehouse Schema Design

Reading Time: 3 minutes Conventional data warehouses support data models based on star schema and snowflake schema. In these models, there are a number of fact tables and dimension tables. In order to minimize redundancy it is recommends to split data into multiple tables in . This is a normalization process. Normalization is the technique of eliminating the redundant data. It minimize the insertion, deletion, and update anomalies. It saves the disk Continue Reading

All you need to know about Avro schema

Reading Time: 4 minutes In this post, we are going to dive into the basics of the Avro Schema. We will create a sample avro schema and serialize it to a sample output file and also read the file as an example according to the avro schema. Intro to Avro Apache Avro is a data serialization system developed by Doug Cutting, the father of Hadoop that helps with data Continue Reading

Delta Lake: Schema Enforcement & Evolution

Reading Time: 4 minutes Nowadays data is constantly evolving and changing. As well as the business problems and requirements are evolving, the shape or the structure of the data is also changing. When that happens, we want to be in control of how the data or schema changes. But how we can achieve this? Delta Lake has good ways to control how schema changes. With Delta Lake, users have Continue Reading

Error Registering Avro Schema | Multiple Schemas In One Topic

Reading Time: 4 minutes org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {“type”:”record”,”name”:”schema1″,”namespace”:”test”,”fields”:[{“name”:”Name”,”type”:”string”},{“name”:”Age”,”type”:”int”},{“name”:”Location”,”type”:”string”}]} Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema; error code: 409 at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:170) at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:188) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:245) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:237) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:232) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:59) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:91) at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:72) at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:158) at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:57) at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala) You might have come across a similar exception while working with AVRO schemas. Kafka throws this exception due to a compatibility issue Continue Reading