Question: I want to display the contents which is loaded from CSV file as a DataFrame. But with the show function columns data are getting truncated. How can i see the entire content of the column. Below is my sample code
val df = spark.read.format("com.databricks.spark.csv").option("header", "true").load("HadoopExam.csv")
df.createOrReplaceTempView("HETable")
results = sql("select * from HETable");
results.show()
Answer: In this case you have to use proper option for the show() function.
Where show(arg1, arg2);
- arg1 : Number of records/rows you want to see from the DataFrame
- arg2 : Do you want to truncate the columns or not.
Use below statement to view the full column and 20 number of rows in the output.
results.show(20, false)
There is another option without putting the number of records limit as below.
results.show(false)
If you are using Databricks Platform then they have their own method as below. Which would show in properly html formatted table.
display(results)
If you are not sure how many rows in your DataFrame then you can use the below method. In first argument you would be finding the number of Rows and then using second argument you are specifying not to truncate the result.
df.show(df.count.toInt, false)