PySpark Cheat Sheet

Sort

>>> rdd2.sortBy(lambda x: x[1]).collect()  #Sort RDD by given function 
 [('d' ,1), ('b' ,1), ('a',2)]
>>> rdd2.sortByKey().collect()   #Sort (key, value) RDD by key
 [('a' ,2), ('b' ,1), ('d' ,1)]

Mathematical Operations

>>> rdd.subtract(rdd2).collect()  #Return each rdd value not contained in rdd2
 [('b' ,2), ('a' ,7)]
#Return each (key,value) pair of rdd2 with no matching key in rdd
>>> rdd2.subtractByKey(rdd).collect() 
 [( 'b', 1)]
>>> rdd.cartesian(rdd2).collect()  #Return the Cartesian product of rdd and rdd2

Repartitioning

null

Reference

• PySpark Cheat Sheet - Cheat Sheet Maker