Deployment modes and job submission in Apache Spark

There are various ways of submitting an application in spark. In Addition to client and cluster modes of execution there is also a local mode of submitting a

What is Spark RDD and Why Spark needs it?

RDD (Resilient Distributed Dataset) is the fundamental data structure of Apache Spark. This post will cover What is RDD, Why Spark needs it and how to create an

Files with special character or Encoding in Spark

Encoding is used to translate the numeric values into a readable character it provides the information that your computer needs to display the text on the screen. Spark read.csv()

Capture bad records while loading csv in data frame

Loading a csv file and capturing all the bad records is a very common task in ETL projects. The bad records are analyzed to take corrective or preventive