Cloud - DeepTech
Welcome to my Channel "DeepTech" - My Name is Deep. In this channel I will talk about multiple technologies like Cloud Computing, Data Engineering, System Design, Machine Learning etc.
You can post all your questions in the "Comments" section, I will try my best to answer those.
2. Output Partitions 1 - repartition, coalesce
1. Spark Input Partitions (spark.sql.files.maxPartitionBytes)
23 What should be the value of Shuffle Partition No (spark.sql.shuffle.partitions)
24 Assignment Project 2 - Shuffle & Spark UI
21 Shuffle Process in join with Spark UI explanation
25 Summary - Transformations & Actions Part 1
22 DO NOT use show for analysis in Spark
20 Shuffle Process in groupBy with Spark UI details
19 Spark Shuffle Process & External Shuffle Service (ESS)
18 HandsOn Transformation - union(), unionAll(), intersect(), withColumn()
17 HandsOn Transformation - subtract()
16 HandsOn Transformation - orderBy(), selectExpr()
15 HandsOn Function - col()
14 HandsOn Transformation - join()
10 File Performance Comparison - CSV vs Parquet
12 Transformations - agg, groupBy, orderBy, selectExpr, subtract, withColumn, union
8 Spark UI - Jobs, Environment
7 Spark Action - HandsOn
11 Assignment Project 1 - Transformation & Action
13 HandsOn Transformation - agg, alias, groupby
9 load - Transformation or Action?
5 Transformations - describe, sort, limit, drop, dropDuplicates
3 Step by step Transformation
6 Spark Actions
4 Chain Transformation
2 Intro to Transformation HandsOn
1 Introduction to Transformation
Define DataFrame Schema StructType, StructField
SparkSession Details
Partition, Split, Task, Executor Relation