![TechLake](/img/default-banner.jpg)
- 453
- 4 428 570
TechLake
India
Приєднався 1 сер 2013
Hi Friends,
I am working in IT From past 15+ Years on SQL, PLSQL ,Data Warehousing, Oracle Data Integrator , OBIEE , LINUX , Python , Azure Cloud, Azure Databricks, Azure Data Factory, Pyspark, Spark, Delta Lake, Hadoop, Hive, Big Data , PHP Technologies and Version Controlling Tools like SVN, GIT and Virtualization tools Docker and VMWare Tools.
I will share my Knowledge and Experience through this Channel. If you like my videos Please Subscribe My Channel for latest updates and Videos.
My Instagram profile link :
**************************
trraveendratechie
Please find my github repository for sample code and data files.
github.com/raveendratal/PysparkRaveendra
you can find python code dbc file in below location:
github.com/raveendratal/PysparkRaveendra/blob/master/Python_Basics_tutorial.dbc
Thank you....
Learn, Like , Share & Subscribe....
I am working in IT From past 15+ Years on SQL, PLSQL ,Data Warehousing, Oracle Data Integrator , OBIEE , LINUX , Python , Azure Cloud, Azure Databricks, Azure Data Factory, Pyspark, Spark, Delta Lake, Hadoop, Hive, Big Data , PHP Technologies and Version Controlling Tools like SVN, GIT and Virtualization tools Docker and VMWare Tools.
I will share my Knowledge and Experience through this Channel. If you like my videos Please Subscribe My Channel for latest updates and Videos.
My Instagram profile link :
**************************
trraveendratechie
Please find my github repository for sample code and data files.
github.com/raveendratal/PysparkRaveendra
you can find python code dbc file in below location:
github.com/raveendratal/PysparkRaveendra/blob/master/Python_Basics_tutorial.dbc
Thank you....
Learn, Like , Share & Subscribe....
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data #delta #databricks
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data #delta #databricksDelta
Lake Column Mapping: Everything You Need to Know,
How to Use Delta Lake Column Mapping to Evolve Your Schema,
Delta Lake Column Mapping: A Deep Dive,
Delta Lake Column Mapping: The Ultimate Guide,
Delta Lake Column Mapping: Benefits and Best Practices,
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data,
Delta Lake Column Mapping: A Tutorial,
Delta Lake Column Mapping: What It Is and Why You Should Use It,
Delta Lake Column Mapping: A Game-Changer for Schema Evolution,
Delta Lake Column Mapping: How to Use It to Improve Your Data Lake,
Delta Lake Column Mapping: A Step-by-Step Guide,
Delta Lake Column Mapping: The Future of Schema Evolution,
Delta Lake Column Mapping: Everything You Need to Know to Get Started,
Delta Lake Column Mapping: A Powerful Tool for Data Engineers,
Delta Lake Column Mapping: How to Use It to Improve Your Data Pipeline,
Lake Column Mapping: Everything You Need to Know,
How to Use Delta Lake Column Mapping to Evolve Your Schema,
Delta Lake Column Mapping: A Deep Dive,
Delta Lake Column Mapping: The Ultimate Guide,
Delta Lake Column Mapping: Benefits and Best Practices,
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data,
Delta Lake Column Mapping: A Tutorial,
Delta Lake Column Mapping: What It Is and Why You Should Use It,
Delta Lake Column Mapping: A Game-Changer for Schema Evolution,
Delta Lake Column Mapping: How to Use It to Improve Your Data Lake,
Delta Lake Column Mapping: A Step-by-Step Guide,
Delta Lake Column Mapping: The Future of Schema Evolution,
Delta Lake Column Mapping: Everything You Need to Know to Get Started,
Delta Lake Column Mapping: A Powerful Tool for Data Engineers,
Delta Lake Column Mapping: How to Use It to Improve Your Data Pipeline,
Переглядів: 4 262
Відео
what is small files problem in spark , How to Fix It in delta lake #optimize #delta #smallfilesissue
Переглядів 4 тис.10 місяців тому
Small file problem in Hadoop and Spark - How delta lake helps, Reading Millions of Small JSON Files from S3 Bucket in , How to handle small file problem in spark structured streaming?, Improving performance for Spark with a large number of small , Reading many small files Spark, what is small files problem in spark, spark small files problem s3, small file problem in hadoop, how to solve small ...
Liquid Clustering in Databricks,What It is and How to Use, #liquidclustering #clusterby #databricks
Переглядів 8 тис.10 місяців тому
Liquid Clustering in Databricks,What It is and How to Use, #liquidclustering #clusterby #databricksLiquid Clustering in Databricks,The Ultimate Guide, Databricks Liquid Clustering,What It Is and How to Use It, Improve Your Databricks Performance with Liquid Clustering, How to Use Liquid Clustering to Optimize Your Databricks Workloads, Databricks Liquid Clustering,A Step-by-Step Tutorial, Datab...
Azure Data Factory part 11 : how to migrate onprem files to adls gen2 #selfhosted #azure #adf
Переглядів 3,9 тис.Рік тому
what is linked self hosted integrated runtime #selfhosted #azure #adf Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple file...
Azure Data Factory part 9 : what is linked self hosted integrated runtime #selfhosted #azure #adf
Переглядів 1,1 тис.Рік тому
Azure Data Factory part 9 : what is linked self hosted integrated runtime #selfhosted #azure #adf Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4...
Azure Data Factory part 8 : how to split one file into multiple files using git-scm #split #adf
Переглядів 1,8 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory part 7 : What is Binary File Dataset and when we will use this? #adf #binaryfile
Переглядів 1,7 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory Part 6 : How to migrate Onpremises Oracle data to azure datalake gen2 #ADF #azure
Переглядів 6 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory part 5 : self hosted integration runtime configuration #ADF #selfhosted #azure
Переглядів 2,8 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 #ADF #azure #copyactivity
Переглядів 2 тис.Рік тому
Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 #ADF #azure #copyactivity
Azure Data Factory Part 3 : Adls gen1 to gen2 migration #adf #datafactory #adlsgen1togen2 #adlsgen2
Переглядів 2 тис.Рік тому
Azure Data Factory Part 3 : Adls gen1 to gen2 migration #adf #datafactory #adlsgen1togen2 #adlsgen2
Data warehouse Vs Datalake Vs Lakehouse #datawarehouse #datalake #lakehouse
Переглядів 11 тис.Рік тому
Data warehouse Vs Datalake Vs Lakehouse #datawarehouse #datalake #lakehouse
Azure Data Factory part 10 : How do i resolve this error self-hosted IR for file linked service
Переглядів 5 тис.Рік тому
Azure Data Factory part 10 : How do i resolve this error self-hosted IR for file linked service
Microsoft Fabric Lakehouse #fabric #azurefabric #microsoftfabric #azure #lakehouse
Переглядів 2,9 тис.Рік тому
Microsoft Fabric Lakehouse #fabric #azurefabric #microsoftfabric #azure #lakehouse
Snowflake sql tutorial part 2 : why sql and where are using sql #snowflake #snowsql
Переглядів 758Рік тому
Snowflake sql tutorial part 2 : why sql and where are using sql #snowflake #snowsql
Azure data factory Tutorial 2 : adf components #adf #datafactory #azuredatafactory #adftutorial
Переглядів 1,7 тис.Рік тому
Azure data factory Tutorial 2 : adf components #adf #datafactory #azuredatafactory #adftutorial
Snowflake SQL Tutorial part 1 : Introduction to snowflake sql
Переглядів 2,4 тис.Рік тому
Snowflake SQL Tutorial part 1 : Introduction to snowflake sql
Introduction to azure data factory tutorial part 1 #AzureDataFactory #datafactory #azurecloud #ADF
Переглядів 4,5 тис.Рік тому
Introduction to azure data factory tutorial part 1 #AzureDataFactory #datafactory #azurecloud #ADF
how to create databricks free trail , how to create databricks community edition #databricks #azure
Переглядів 12 тис.Рік тому
how to create databricks free trail , how to create databricks community edition #databricks #azure
how to create Snowflake free trail? #snowflake #snowflakefreetrail #snowsql #snowflakedwh
Переглядів 318Рік тому
how to create Snowflake free trail? #snowflake #snowflakefreetrail #snowsql #snowflakedwh
Azure Datalake Gen2 SFTP Pricing #azuredatalake #adlsgen2 #azurepricing #azurestorage #cloudstorage
Переглядів 337Рік тому
Azure Datalake Gen2 SFTP Pricing #azuredatalake #adlsgen2 #azurepricing #azurestorage #cloudstorage
delta lake tutorial 8 : What is Delta Lake Optimize #deltalake #optimize #databricks #delta #sql
Переглядів 3,7 тис.Рік тому
delta lake tutorial 8 : What is Delta Lake Optimize #deltalake #optimize #databricks #delta #sql
delta lake tutorial 7: What is Delta Lake Restore #deltalake #restore #databricks #delta #sql
Переглядів 2,5 тис.Рік тому
delta lake tutorial 7: What is Delta Lake Restore #deltalake #restore #databricks #delta #sql
delta lake tutorial 6 : What is Delta Lake Vacuum #deltalake #vacuum #databricks #delta #sql
Переглядів 3,2 тис.Рік тому
delta lake tutorial 6 : What is Delta Lake Vacuum #deltalake #vacuum #databricks #delta #sql
delta lake tutorial 5 : Delta Lake Timetravel and Audit Log #deltalake #timetravel #auditlog #delta
Переглядів 3,4 тис.Рік тому
delta lake tutorial 5 : Delta Lake Timetravel and Audit Log #deltalake #timetravel #auditlog #delta
Pyspark Scenarios 23 : How do I select a column name with spaces in PySpark? #pyspark #databricks
Переглядів 20 тис.Рік тому
Pyspark Scenarios 23 : How do I select a column name with spaces in PySpark? #pyspark #databricks
Pyspark Scenarios 22 : How To create data files based on the number of rows in PySpark #pyspark
Переглядів 6 тис.Рік тому
Pyspark Scenarios 22 : How To create data files based on the number of rows in PySpark #pyspark
Databricks Unity Catalog part3 How to create storage credentials and external location #UnityCatalog
Переглядів 6 тис.Рік тому
Databricks Unity Catalog part3 How to create storage credentials and external location #UnityCatalog
Databricks Unity Catalog part2 How to Configure databricks unity catalog #UnityCatalog #databricks
Переглядів 9 тис.Рік тому
Databricks Unity Catalog part2 How to Configure databricks unity catalog #UnityCatalog #databricks
Databricks Unity Catalog part1 what is databricks unity catalog? #unitycatalog #databricks #catalog
Переглядів 12 тис.Рік тому
Databricks Unity Catalog part1 what is databricks unity catalog? #unitycatalog #databricks #catalog
This tutorial is a hidden gem💎. Can't Thank Enough for your efforts..
Hi Sir, This is Great… Thanks for sharing your knowledge… I have a Question -- Batch count in for each activity - 20 But Self hosted integration runtime - concurrent jobs is set to - 10 How the for each activity’s parallel processing is related to concurrent jobs - Now, for each activity will process 10 files parallely or 20 files - ? What is the maximum number that can be specified in for each activity parallel processing -- if the concurrent jobs set to 10 ? Thank You, Sir....
Good explanation about spark core. Thanks 👍🏻
This is what we called real time Teaching. Hoping to be blessed later as well. Also please make a video on transfering data form on premises to another in details steps. Like makeing linked connection n all.
Hi, I am getting error : java.lang.NoSuchMethodError: scala.collection.immutable.Seq.map(Lscala/Function1;)Ljava/lang/Object;
Sir what is your email you are my guru and amazing explanation sir love you🤗
Short and meaningful.....
can we create external location on an existing container ?
Y sir that much file , it's like confusing
please also show the conatiner content from unity cataloag
I want to take paid training can you give your number
amazing
Well explained
hello Sir can i get the power point presentation which you are sharing for spark
Sir, can you provide the roadmap of azure data engineering, not getting what to do when
Can we learn this Fabric directly or any prerequisite for this to learn??
Thankyou Sir for this Data engineering dedicated python playlist. I was searching this for long time. One question... Is this playlist enough for all Data Engineering tasks from basic to advance? Waiting for ur response..
Python basics is enough
Thanks for making this video sir, very clear explanation. Thank you sir.
Please anyone answer. So I learnt SQL for data engineering, and I want to watch this entire playlist to learn python. My question is after this playlist can I start making data engineering projects and applying for Data engineering internship or do I watch more videos. Please answer I feel kind of lost.
Please anyone answer. So I learnt SQL for data engineering, and I want to watch this entire playlist to for python. My question is after this playlist can I start making data engineering projects and applying for Data engineering internship or do I watch more videos. Please answer I feel kind of lost.
Please anyone answer. So I learnt SQL for data engineering, and I want to watch this entire playlist to for python. My question is, after this playlist can I start making data engineering projects and applying for Data engineering internship or do I watch more videos. Please answer I feel kind of lost.
Still we could not find the proper reason why the records went into corrupt when the column are very huge
Thank you Ravi to explain things in more simple way! Can we connect to transform myself(web developer 13 years exp. ) into Data Engineering
Perfect !! thanks a lot for yours efforts for informative video .. Cheers !!
when i'm trying to use this getting error
i got , query returns no records
Please how to fix error odi 12308
how to fix error ODI 12308 please
This is also called a managed table right?
Great, thank you for the video.
your literally awesome bro.....and this content is really meaning for us.....keep it up
if iwant to take only one file from multiple subfolders. like format would be like this /mnt/test/2024/01/01/.csv files, same like we have different date wise subfolders. what is the approach pls explain me
Can we do the same for XML and JSON files?
plz upload other content on your GitHub lots file is not available......
Nice explanation👌
I'm trying to write to show my gratitude for this incredible entire tutorials ever can help anybody from down to earth but I don't think saying "Thanks" is enough.Really appreciated your efforts and time ! Keep up the good work !
Unable to Perform in databricks community edition
yes throwing Erroe: java.lang.NoSuchMethodError: scala.collection.immutable.Seq.map(Lscala/Function1;)Ljava/lang/Object;
error java.lang.NoSuchMethodError: scala.collection.immutable.Seq.map(Lscala/Function1;)Ljava/lang/Object;
Hi, I am trying to do the partition on one of the existing table for the year 2023 on Monthly basis so can you please help me with Partition Function and Partition Scheme queries please. Here I am using uploaddate column as a partition key.
India not for beginners 😂😂
thanks brother its really helped for beginner's
Can you please sort your videos in this playlist in correct sequence?
Hi, thanks for the video. In my case my nested struct type column does not contain any nested array type column instead it contains nested struct type. Is there any other way I could use?
Please add a text file on these Q&A...couldn't understand clearly..Voice is too low...
bro, thanks for your inputs. can you please help me how to handle this? empid,fname|lname@sal#deptid 1,mohan|kumar@5000#100 2,karna|varadan@3489#101 3,kavitha|gandan@6000#102 Expected output empid,fname,lname,sal,deptid 1,mohan,kumar,5000,100 2,karan,varadan,3489,101 3,kavitha,gandan,6000,102
what would be a better option to generate a surrogate key? Edit: I discovered monotonically_increasing_id, please y'all comment if you encountered is better or to propose another one.
If we have 4 cores then 8 task can run in parallel right ?
Thanks for uploading video on ODI procedure
I ❤❤❤❤