Apache Kafka - Quick Start on Windows

In this post I will be demonstrating how to setup and use Apache Kafka on windows environment. Before that, I will briefly describe about Kafka and then take you in practical world. You can also refer following video to setup apache kafka in windows environment.

About Apache Kafka

Kafka is a distributed publish-subscribe messaging solution. It is fast, scalable, and durable as compared to traditional messaging systems. If you think about traditional publish-subscribe messaging system, producers produce/write messages to topic and on another side consumers consume/read messages from this topic. Kafka is designed in a manner where topics can be partitioned and replicated across multiple servers.  

If you would like to get more detail please refer this link. 

I referred this blog (source: http://blog.cloudera.com/blog/2014/09/apache-kafka-for-beginners/). It provides good and easy explanation about Kafka and its concepts.  Below two paragraphs are taken from same blog.

"Messages are simply byte arrays and the developers can use them to store any object in any format – with String, JSON, and Avro the most common. It is possible to attach a key to each message, in which case the producer guarantees that all messages with the same key will arrive to the same partition. When consuming from a topic, it is possible to configure a consumer group with multiple consumers. Each consumer in a consumer group will read messages from a unique subset of partitions in each topic they subscribe to, so each message is delivered to one consumer in the group, and all messages with the same key arrive at the same consumer."

"What makes Kafka unique is that Kafka treats each topic partition as a log (an ordered set of messages). Each message in a partition is assigned a unique offset. Kafka does not attempt to track which messages were read by each consumer and only retain unread messages; rather, Kafka retains all messages for a set amount of time, and consumers are responsible to track their location in each log. Consequently, Kafka can support a large number of consumers and retain large amounts of data with very little overhead."

Now you must have a question i.e. "How to setup Apache in windows environment?". Don't worry, I am stepping ahead to describe same. You just need to follow simple steps and you are done.

Download and Change Required Properties

1.   Download Kafka from here (kafka_2.9.1-0.8.2.2) and unzip at your desired location. 

2.   Go to <kafka_dir>\config\server.properties file and change log file location property 'log.dirs' as per your environment.

log.dirs=<kafka_dir>\kafka-logs

3.   Go to <kafka_dir>\config\zookeeper.properties file and change data directory location property 'dataDir' as per your environment

dataDir=<kafka_dir>\zookeeper-data

Start Zookeeper and Kafka Servers

Kafka internally uses Zookeeper. If you would like to get more detail on Zookeeper you can refer this link.

First you need to start Zookeeper server. To start it, execute below command:

<kafka_dir>\bin\windows\zookeeper-server-start.bat ..\..\config\zookeeper.properties

Now open another command prompt and start Kafka server:

<kafka_dir>\bin\windows\kafka-server-start.bat ..\..\config\server.properties

Create Topic

Now you need to create topic to publish and subscribe messages. To create topic you just need to execute below command. As per below command you will be creating topic 'mytopic' with single partition.

<kafka_dir>\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic

After executing above command you must see below message on command prompt:

Created topic "mytopic".

Produce and Consume Messages

Open a command prompt and execute below command. This command prompt will be treated as 

producer.

<kafka_dir>\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic

Now open another command prompt and execute below command. This command prompt will be treated as consumer.

<kafka_dir>\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic mytopic

If you will type any message in the producer command prompt and press enter, it will be consumed by consumer and you must be able to see the same message in consumer command prompt.

 If you reached at this stage, that means you are able to setup Kafka successfully in your windows environment. If you need more detail with respect to above post you can post comment on it. Thanks for reading this.

Extracting Personal Information from Driving License

Most of the web and mobile applications may have the requirement to collect personal information of end user that includes first/middle/last name, date of birth, address, gender etc. To collect personal information, we generally create a web page containing personal information fields. This web page is presented to the end user and user has to fill personal information and submit it. Isn’t it time consuming process for end user? What if personal information is pre-populated by some mechanism? If we pre-populate data from some authentic sources, this will not only save end user’s typing time but also reduces typing errors. This way we can improve user experience of application.

There could be multiple sources to fetch personal information. Driving license is one of the best sources that contain personal information say first/middle/last name, date of birth, address, gender etc. To read the information, we have to scan or get pic of driving license, read bar code and then decode the information. But notice that, only couple of countries provide driving license with 2D bar code. United State of America provides driving licenses with 2D bar codes. In this post, I will walk you through how you can read driving license 2D bar code out of scanned image, decode the personal information and use it in your application.

How to Extract Driving License Information

I was searching for different tools on net that provides Java API to extract personal information. I reached to Inlite, this is one of the best options I was searching for. Inlite provides the solution to extract personal information from US/Canada driver licenses along with other bar code related features. 

Below is the sample driving license with 2D bar code [this is taken from Inlite site from here].

Follow below steps to extract driving license information in your local environment:

 1. Download image processing SDK ‘ClearImage SDK’ from Inlite. This is free for developers and supports any number of developers.

2. You will get executable file like ‘ClearImageSDK.8_3_4524.exe’. Now install clear image SDK in your local environment by executing this file. Clear image installation directory will look like below:

3. Now in your eclipse IDE create sample java project and set following library in your project’s build path.

<clearImageInstallDir>\workspace.ClearImageJava\libs\ClearImageJ.jar

4. After following this link, I have used same code snippet to read the driving license 2D bar code in following class. Notice that, in following class you need to replace two strings:

<clearImageInstallDir> :  Replace this with actual clear image installation directory.

<downloadedImagePath> : Download scanned image of sample driving license from here and replace this with image file path.

package com.xx.dl.reader; import ClearImageJNI.*; public class DLBarCodeReader {               public static void main(String[] args) throws CiException {                                  String s =   CiServer.loadClearImage("<clearImageInstallDir>\\                            workspace.ClearImageJava\\libs\\win64                            \\ClearImage.dll");              if (s != ""){                     System.out.println("ERROR Loading: " + s);              }else{                     System.out.println("Loaded DLL file successfully");              }                                      String infoXML = readBarcode2D("<downloadedImagePath>", 1);        }               static String readBarcode2D(String fileName, int page) throws CiException       {              CiServer objCi=new CiServer();              ICiServer Ci=objCi.getICiServer();              try {                     // Create and configure barcode reader                     ICiPdf417 reader = Ci.CreatePdf417();                                    // Open Image file                     reader.getImage().Open(fileName, page);                                  // Read barcodes                     reader.Find(0);                                      // Process results                     for (int i = 1; i <= reader.getBarcodes().getCount(); i++) {                            ICiBarcode Bc =                               reader.getBarcodes().getItem(i);                                                      //This will print XML containing information of DL                            retun Bc.getText();                     }              } catch (Exception ex) {                     ex.printStackTrace();                                } finally {              // Collect garbage periodically to release JNI-managed objects                     System.gc();              }              return null;        } }

5.   Run above program. You will get following xml containing personal information:

<?xml version="1.0" encoding="UTF-8"?> <AAMVA>   <user>     <last e="DCS">Lastnamexyxyxyxyxyxyxyxyxxyxyxyxyxyxyxyx</last>     <first e="DAC">Firstxyxyxyxyxyxyxyxyxxyxyxyxyxyxyxyxxyx</first>     <middle e="DAD">Xyxyxyxyxyxyxyxyxxyxyxyxyxyxyxyxxyxyxyxy</middle>     <suffix e="DCU">SUFIX</suffix>     <dob e="DBB">1977-10-31</dob>     <eyes e="DAY">BRO</eyes>     <hair e="DAZ">BLK XY1XY1XY</hair>     <sex e="DBC">M</sex>     <height e="DAU">5'8"</height>     <weight e="DAW">150 lbs</weight>     <street e="DAG">1234 Any Street Xy1Xy1Xy1Xy1Xy1Xy1X</street>     <city e="DAI">City Xy1Xy1Xy1Xy1Xy1</city>     <state e="DAJ">CA</state>     <postal e="DAK">00000-0000</postal>     <country e="DCG">USA</country>     <id e="DAQ">D1234562 XYXYXYXYXYXYXYXYX</id>     <issued e="DBD">2009-10-31</issued>     <expires e="DBA">2014-10-31</expires>   </user>   <head>     <filetype name="File Type">ANSI</filetype>     <format name="Data Format">21</format>     <issuer name="Issuer Identification Number">636014</issuer>     <state name="Issuer Name">California</state>     <st name="Issuer Name Abbreviated">CA</st>   </head>   <subfile designator="DL">     <element id="DAQ" name="Customer ID Number">D1234562 XYXYXYXYXYXYXYXYX</element>     <element id="DCS" name="Customer Family Name">LASTNAMEXYXYXYXYXYXYXYXYXXYXYXYXYXYXYXYX</element>     <element id="DDE" name="Family name truncation">U</element>     <element id="DAC" name="Driver First Name">FIRSTXYXYXYXYXYXYXYXYXXYXYXYXYXYXYXYXXYX</element>     <element id="DDF" name="First name truncation">U</element>     <element id="DAD" name="Driver Middle Name or Initial">XYXYXYXYXYXYXYXYXXYXYXYXYXYXYXYXXYXYXYXY</element>     <element id="DDG" name="Middle name truncation">U</element>     <element id="DCA" name="Jurisdiction-specific vehicle class">A XYXY</element>     <element id="DCB" name="Jurisdiction-specific restriction codes">NONEY1XY1XY1</element>     <element id="DCD" name="Jurisdiction-specific endorsement codes">NONEX</element>     <element id="DBD" name="Document Issue Date">10312009</element>     <element id="DBB" name="Date of Birth">10311977</element>     <element id="DBA" name="Document Expiration Date">10312014</element>     <element id="DBC" name="Physical Description - Sex">1</element>     <element id="DAU" name="Physical Description - Height">068 IN</element>     <element id="DAY" name="Physical Description - Eye Color">BRO</element>     <element id="DAG" name="Address - Street 1">1234 ANY STREET XY1XY1XY1XY1XY1XY1X</element>     <element id="DAI" name="Address - City">CITY XY1XY1XY1XY1XY1</element>     <element id="DAJ" name="Address - Jurisdiction Code">CA</element>     <element id="DAK" name="Address - Postal Code">000000000</element>     <element id="DCF" name="Document Discriminator">00/00/0000NNNAN/ANFD/YY X</element>     <element id="DCG" name="Country Identification">USA</element>     <element id="DCU" name="Name Suffix">SUFIX</element>     <element id="DAW" name="Physical Description - Weight">150</element>     <element id="DAZ" name="Hair color">BLK XY1XY1XY</element>     <element id="DCK" name="Inventory control number">XY1XY1XY1XY1XY1XY1XY1XY1X</element>     <element id="DDA" name="Compliance Type">F</element>     <element id="DDB" name="Card Revision Date">MMDDCCYY</element>     <element id="DDD" name="Limited Duration Document Indicator">1</element>   </subfile>   <subfile designator="ZC">     <element id="A" name="Optional field A">Y</element>     <element id="B" name="Optional field B">CORR LENS</element>     <element id="C" name="Optional field C">BRN</element>     <element id="D" name="Optional field D">XYX</element>     <element id="E" name="Optional field E">XYXYXYXYXYXYXY</element>     <element id="F" name="Optional field F">XY1XY1XY1XY1XY1XY1XYXYXYXYXYXYXY</element>   </subfile> </AAMVA>

6.   Using JAXB, parse above xml data into your own java pojo.

a.      Download ‘jaxb-ri-2.2.11.zip’ from this link and then extract it.

b.      After extracting this zip file, add external jar files (jaxb-ri-2.2.11\jaxb-ri\lib\*.jar) into your eclipse project build path.

c.      Generate XSD from above xml by following this link.

d.      Follow section 2.4 of this link  to generate POJO classes from generated XSD. 

e.      Create below class to hold the personal information.

package com.xx.dl.reader; public class DLUserInfo {        private String firstName;        private String middleName;        private String lastName;          private String dob;        private String street;        private String city;        private String state;        private String postal;        public String getFirstName() {              return firstName;        }        public void setFirstName(String firstName) {              this.firstName = firstName;        }        public String getMiddleName() {              return middleName;        }        public void setMiddleName(String middleName) {              this.middleName = middleName;        }        public String getLastName() {              return lastName;        }        public void setLastName(String lastName) {              this.lastName = lastName;        }        public String getDob() {              return dob;        }        public void setDob(String dob) {              this.dob = dob;        }        public String getStreet() {              return street;        }        public void setStreet(String street) {              this.street = street;        }        public String getCity() {              return city;        }        public void setCity(String city) {              this.city = city;        }        public String getState() {              return state;        }        public void setState(String state) {              this.state = state;        }        public String getPostal() {              return postal;        }        public void setPostal(String postal) {              this.postal = postal;        } }

f.       Map xml to your own pojo:

package com.xx.dl.reader; import java.io.StringReader; import javax.xml.bind.JAXBContext; import javax.xml.bind.JAXBException; import javax.xml.bind.Unmarshaller; import com.xx.dlxmlreader.jaxb.pojo.AAMVA; import com.xx.dlxmlreader.jaxb.pojo.AAMVA.User; public class DLBarCodeXMLDecoder {                // Input argument is the same XML that is mentioned in above steps        public DLUserInfo decodeXml(String dlBarCodeXML) throws                                            JAXBException{                          JAXBContext jaxbContext =                             JAXBContext.newInstance(AAMVA.class);                Unmarshaller jaxbUnmarshaller =                              jaxbContext.createUnmarshaller();                StringReader reader = new StringReader(dlBarCodeXML);               // AAMVA.java should be genereated in step 6.d               AAMVA dlBarCodeXml = (AAMVA)                        jaxbUnmarshaller.unmarshal(reader);                               User userInfo = dlBarCodeXml.getUser();                              DLUserInfo dlUserInfo = new DLUserInfo();               dlUserInfo.setFirstName(userInfo.getFirst().getValue());               dlUserInfo.setMiddleName(userInfo.getMiddle().getValue());               dlUserInfo.setLastName( userInfo.getLast().getValue());               dlUserInfo.setDob(userInfo.getDob().getValue().toString());               dlUserInfo.setStreet( userInfo.getStreet().getValue());               dlUserInfo.setCity(userInfo.getCity().getValue());               dlUserInfo.setState(userInfo.getState().getValue());               dlUserInfo.setPostal(userInfo.getPostal().getValue());                              return dlUserInfo;        } }

  

g.    Call DLBarCodeXMLDecoder.decodeXml in  ‘DLBarCodeReader’ class to map XML information into required POJO.

String infoXML = readBarcode2D("<downloadedImagePath>", 1);    DLUserInfo dlUserInfo = DLBarCodeXMLDecoder.decodeXml(infoXML); System.out.println(dlUserInfo);

I hope this post helped you in extracting information from driving license.