kill all chrome processes powershell

Powershell command:

> kill -name chrome

Enjoy!

Remove seconds pyspark sql

 To remove seconds in pyspark sql in notebook:

SELECT 

        date_trunc('day', current_timestamp) as DateTimeWithZeroHoursMinutes  

        date_trunc('minute', current_timestamp) as DateTimeWithZeroSeconds,

        date_trunc('second', current_timestamp) as DateTimeWithZeroMilliseconds,

        date_format(current_timestamp, "yyyy-MM-dd HH:mm") as DateWithNoSeconds

Access file/folder path in databricks

In databricks notebook, following are the way 2 ways to access the folder (example folder name: mnt):

1. If you are reading from DBFS with pyspark, then you start your filename like this ‘/mnt/….’
2. If you are reading form DBFS with regular python, you start your filename like this ‘/dbfs/mnt/…’

HTH

Setup Single Node Job Cluster from Datafactory

 To Setup Cluster from datafactory job - 

1. First we need to make the Workers to 0 - Yes set it to 0. See image below:
   
   
2. Then under Addition Cluster Settings, look for Cluster Spark conf and set following name/values.
   
   "spark.databricks.cluster.profile": "singleNode"
   "spark.master": "local[*, 4]" 
   
   
   
   
   
3. Last we need to set Tag under Cluster custom tags:
   
   
   "ResourceClass": "SingleNode" 
   
   
   

   
   
   

That's it!!!

HTH

org.apache.spark.sql.AnalysisException: cannot resolve 'explode(`mycol`)' due to data type mismatch: input to function explode should be array or map type, not struct line 1 pos 0;

Hello,

If you exploding and got above error then it means you dont need to explode it as it is not type array or map.

You can simply access values as:

df_expanded = df.withColumn("prop1", expr("mycol.prop1"))

HTH

Is HBL and Bank AL Habib same???

 Hello -

The answer is NO.

These are 2 different banks - 

HBL Iban starts as: PK HABB

Bank AL Habib Iban start as: PK BAHL

HTH