Remove seconds pyspark sql

 To remove seconds in pyspark sql in notebook:

SELECT 

        date_trunc('day', current_timestamp) as DateTimeWithZeroHoursMinutes  

        date_trunc('minute', current_timestamp) as DateTimeWithZeroSeconds,

        date_trunc('second', current_timestamp) as DateTimeWithZeroMilliseconds,

        date_format(current_timestamp, "yyyy-MM-dd HH:mm") as DateWithNoSeconds

Access file/folder path in databricks

In databricks notebook, following are the way 2 ways to access the folder (example folder name: mnt):

1. If you are reading from DBFS with pyspark, then you start your filename like this ‘/mnt/….’
2. If you are reading form DBFS with regular python, you start your filename like this ‘/dbfs/mnt/…’

HTH

Setup Single Node Job Cluster from Datafactory

 To Setup Cluster from datafactory job - 

1. First we need to make the Workers to 0 - Yes set it to 0. See image below:
   
   
2. Then under Addition Cluster Settings, look for Cluster Spark conf and set following name/values.
   
   "spark.databricks.cluster.profile": "singleNode"
   "spark.master": "local[*, 4]" 
   
   
   
   
   
3. Last we need to set Tag under Cluster custom tags:
   
   
   "ResourceClass": "SingleNode" 
   
   
   

   
   
   

That's it!!!

HTH

org.apache.spark.sql.AnalysisException: cannot resolve 'explode(`mycol`)' due to data type mismatch: input to function explode should be array or map type, not struct line 1 pos 0;

Hello,

If you exploding and got above error then it means you dont need to explode it as it is not type array or map.

You can simply access values as:

df_expanded = df.withColumn("prop1", expr("mycol.prop1"))

HTH

Is HBL and Bank AL Habib same???

 Hello -

The answer is NO.

These are 2 different banks - 

HBL Iban starts as: PK HABB

Bank AL Habib Iban start as: PK BAHL

HTH

Power BI Dynamic Row Level Security (RLS)

If you were like me trying to implement RLS using fields other than email, then you have come to right place.

RLS will automatically replace the value in filter for you.

Simply use the UserPrincipalName() or UserName() function on the column you want to identify users. It can be any string that can identify user. 

Example: Email Id, Guid, Name etc.

Power BI automatically replace the value in the function based on the value that will be supplied to apply the security in the report.

HTH.

Get Row Count in Powerbi Report (paginated report)

 Hi, How to get total number of rows in your report?

1. Make sure you select a field which is NOT Nullable.
2. Then insert a Textbox in the report
3. Write Expression to display data in the expressions for Textbox.
4. Expression:  =Count(Fields!EmployeeName.Value, "DataSet1")

HTH