Manab's Notes

Data factory in Microsoft Fabric provides cloud-scale data movement and data transformation services that allow you to solve the most complex data factory and ETL scenarios. It's intended to make your data factory experience easy to use, powerful, and truly enterprise-grade.

Create a new Lakehouse- SalesLakehouse

Create a new Data Pipeline- SalesPipeline

Click on Copy Data activity

Click on HTTP under Generic Protocol

Create a new connection using https://raw.githubusercontent.com/MicrosoftLearning/dp-data/main/sales.csv

Click on Next in the below page

Click Next again in the below page

Click on Lakehouse

Select Sales Lakehouse

Select Files and type folder name and file name

Change Row Delimiter

Click Save and Run

Check the Lakehouse

Now we will use a notebook for the data transformation before storing the data to the lakehouse table(delta parquet format)

Click on open note book-> New Notebook

Write below spark code and run

table_name= "sales"

from pyspark.sql.functions import *

#read the sales data

df= spark.read.format("csv").option("header","true").load("Files/SalesFolder/Sales.csv")

#Add OrderMonth and OrderYear columns from OrderDate

df= df.withColumn("OrderYear", year(col("OrderDate"))).withColumn("OrderMonth",month(col("OrderDate")))

#Derive FirstName and LastName Columns

df= df.withColumn("FirstName", split(col("CustomerName"), " ").getItem(0)).withColumn("LastName", split(col("CustomerName"), " ").getItem(1))

#Filter and reorder columns

df= df["SalesOrderNumber", "SalesOrderLineNumber","OrderDate", "OrderYear", "OrderMonth", "FirstName","LastName", "EmailAddress","Item","Quantity", "UnitPrice","TaxAmount"]

#Load the data into a table as a delta parquet format

df.write.format("delta").mode("append").saveAsTable(table_name)

Check the table created after the refresh

Open the pipeline, bring a new Delete activity

Add notebook activity to load the data from Files to Table

Save and Run

Google Sites

Report abuse