TechLake
TechLake
  • 453
  • 4 428 570
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data #delta #databricks
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data #delta #databricksDelta
Lake Column Mapping: Everything You Need to Know,
How to Use Delta Lake Column Mapping to Evolve Your Schema,
Delta Lake Column Mapping: A Deep Dive,
Delta Lake Column Mapping: The Ultimate Guide,
Delta Lake Column Mapping: Benefits and Best Practices,
Delta Lake Column Mapping: How to Rename and Drop Columns Without Rewriting Data,
Delta Lake Column Mapping: A Tutorial,
Delta Lake Column Mapping: What It Is and Why You Should Use It,
Delta Lake Column Mapping: A Game-Changer for Schema Evolution,
Delta Lake Column Mapping: How to Use It to Improve Your Data Lake,
Delta Lake Column Mapping: A Step-by-Step Guide,
Delta Lake Column Mapping: The Future of Schema Evolution,
Delta Lake Column Mapping: Everything You Need to Know to Get Started,
Delta Lake Column Mapping: A Powerful Tool for Data Engineers,
Delta Lake Column Mapping: How to Use It to Improve Your Data Pipeline,
Переглядів: 4 262

Відео

what is small files problem in spark , How to Fix It in delta lake #optimize #delta #smallfilesissue
Переглядів 4 тис.10 місяців тому
Small file problem in Hadoop and Spark - How delta lake helps, Reading Millions of Small JSON Files from S3 Bucket in , How to handle small file problem in spark structured streaming?, Improving performance for Spark with a large number of small , Reading many small files Spark, what is small files problem in spark, spark small files problem s3, small file problem in hadoop, how to solve small ...
Liquid Clustering in Databricks,What It is and How to Use, #liquidclustering #clusterby #databricks
Переглядів 8 тис.10 місяців тому
Liquid Clustering in Databricks,What It is and How to Use, #liquidclustering #clusterby #databricksLiquid Clustering in Databricks,The Ultimate Guide, Databricks Liquid Clustering,What It Is and How to Use It, Improve Your Databricks Performance with Liquid Clustering, How to Use Liquid Clustering to Optimize Your Databricks Workloads, Databricks Liquid Clustering,A Step-by-Step Tutorial, Datab...
Azure Data Factory part 11 : how to migrate onprem files to adls gen2 #selfhosted #azure #adf
Переглядів 3,9 тис.Рік тому
what is linked self hosted integrated runtime #selfhosted #azure #adf Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple file...
Azure Data Factory part 9 : what is linked self hosted integrated runtime #selfhosted #azure #adf
Переглядів 1,1 тис.Рік тому
Azure Data Factory part 9 : what is linked self hosted integrated runtime #selfhosted #azure #adf Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4...
Azure Data Factory part 8 : how to split one file into multiple files using git-scm #split #adf
Переглядів 1,8 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory part 7 : What is Binary File Dataset and when we will use this? #adf #binaryfile
Переглядів 1,7 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory Part 6 : How to migrate Onpremises Oracle data to azure datalake gen2 #ADF #azure
Переглядів 6 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory part 5 : self hosted integration runtime configuration #ADF #selfhosted #azure
Переглядів 2,8 тис.Рік тому
Introduction to azure data factory tutorial part 1 ua-cam.com/video/oPNWFPsmOK4/v-deo.html Azure data factory Tutorial 2 : adf components ua-cam.com/video/p5cF3QXGtqU/v-deo.html Azure Data Factory Part 3 : Adls gen1 to gen2 migration ua-cam.com/video/XE5jPzds474/v-deo.html Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 ua-cam.com/video/YS_0tr-hwjQ/v-deo.html Azure Dat...
Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 #ADF #azure #copyactivity
Переглядів 2 тис.Рік тому
Azure Data Factory Part 4 : How to Copy Multiple files from Gen 1 to Gen2 #ADF #azure #copyactivity
Azure Data Factory Part 3 : Adls gen1 to gen2 migration #adf #datafactory #adlsgen1togen2 #adlsgen2
Переглядів 2 тис.Рік тому
Azure Data Factory Part 3 : Adls gen1 to gen2 migration #adf #datafactory #adlsgen1togen2 #adlsgen2
Data warehouse Vs Datalake Vs Lakehouse #datawarehouse #datalake #lakehouse
Переглядів 11 тис.Рік тому
Data warehouse Vs Datalake Vs Lakehouse #datawarehouse #datalake #lakehouse
Azure Data Factory part 10 : How do i resolve this error self-hosted IR for file linked service
Переглядів 5 тис.Рік тому
Azure Data Factory part 10 : How do i resolve this error self-hosted IR for file linked service
Microsoft Fabric Lakehouse #fabric #azurefabric #microsoftfabric #azure #lakehouse
Переглядів 2,9 тис.Рік тому
Microsoft Fabric Lakehouse #fabric #azurefabric #microsoftfabric #azure #lakehouse
Snowflake sql tutorial part 2 : why sql and where are using sql #snowflake #snowsql
Переглядів 758Рік тому
Snowflake sql tutorial part 2 : why sql and where are using sql #snowflake #snowsql
Azure data factory Tutorial 2 : adf components #adf #datafactory #azuredatafactory #adftutorial
Переглядів 1,7 тис.Рік тому
Azure data factory Tutorial 2 : adf components #adf #datafactory #azuredatafactory #adftutorial
Snowflake SQL Tutorial part 1 : Introduction to snowflake sql
Переглядів 2,4 тис.Рік тому
Snowflake SQL Tutorial part 1 : Introduction to snowflake sql
Introduction to azure data factory tutorial part 1 #AzureDataFactory #datafactory #azurecloud #ADF
Переглядів 4,5 тис.Рік тому
Introduction to azure data factory tutorial part 1 #AzureDataFactory #datafactory #azurecloud #ADF
how to create databricks free trail , how to create databricks community edition #databricks #azure
Переглядів 12 тис.Рік тому
how to create databricks free trail , how to create databricks community edition #databricks #azure
how to create Snowflake free trail? #snowflake #snowflakefreetrail #snowsql #snowflakedwh
Переглядів 318Рік тому
how to create Snowflake free trail? #snowflake #snowflakefreetrail #snowsql #snowflakedwh
Azure Datalake Gen2 SFTP Pricing #azuredatalake #adlsgen2 #azurepricing #azurestorage #cloudstorage
Переглядів 337Рік тому
Azure Datalake Gen2 SFTP Pricing #azuredatalake #adlsgen2 #azurepricing #azurestorage #cloudstorage
delta lake tutorial 8 : What is Delta Lake Optimize #deltalake #optimize #databricks #delta #sql
Переглядів 3,7 тис.Рік тому
delta lake tutorial 8 : What is Delta Lake Optimize #deltalake #optimize #databricks #delta #sql
delta lake tutorial 7: What is Delta Lake Restore #deltalake #restore #databricks #delta #sql
Переглядів 2,5 тис.Рік тому
delta lake tutorial 7: What is Delta Lake Restore #deltalake #restore #databricks #delta #sql
delta lake tutorial 6 : What is Delta Lake Vacuum #deltalake #vacuum #databricks #delta #sql
Переглядів 3,2 тис.Рік тому
delta lake tutorial 6 : What is Delta Lake Vacuum #deltalake #vacuum #databricks #delta #sql
delta lake tutorial 5 : Delta Lake Timetravel and Audit Log #deltalake #timetravel #auditlog #delta
Переглядів 3,4 тис.Рік тому
delta lake tutorial 5 : Delta Lake Timetravel and Audit Log #deltalake #timetravel #auditlog #delta
Pyspark Scenarios 23 : How do I select a column name with spaces in PySpark? #pyspark #databricks
Переглядів 20 тис.Рік тому
Pyspark Scenarios 23 : How do I select a column name with spaces in PySpark? #pyspark #databricks
Pyspark Scenarios 22 : How To create data files based on the number of rows in PySpark #pyspark
Переглядів 6 тис.Рік тому
Pyspark Scenarios 22 : How To create data files based on the number of rows in PySpark #pyspark
Databricks Unity Catalog part3 How to create storage credentials and external location #UnityCatalog
Переглядів 6 тис.Рік тому
Databricks Unity Catalog part3 How to create storage credentials and external location #UnityCatalog
Databricks Unity Catalog part2 How to Configure databricks unity catalog #UnityCatalog #databricks
Переглядів 9 тис.Рік тому
Databricks Unity Catalog part2 How to Configure databricks unity catalog #UnityCatalog #databricks
Databricks Unity Catalog part1 what is databricks unity catalog? #unitycatalog #databricks #catalog
Переглядів 12 тис.Рік тому
Databricks Unity Catalog part1 what is databricks unity catalog? #unitycatalog #databricks #catalog

КОМЕНТАРІ

  • @subhajitdutta7921
    @subhajitdutta7921 День тому

    This tutorial is a hidden gem💎. Can't Thank Enough for your efforts..

  • @gurumoorthy5321
    @gurumoorthy5321 3 дні тому

    Hi Sir, This is Great… Thanks for sharing your knowledge… I have a Question -- Batch count in for each activity - 20 But Self hosted integration runtime - concurrent jobs is set to - 10 How the for each activity’s parallel processing is related to concurrent jobs - Now, for each activity will process 10 files parallely or 20 files - ? What is the maximum number that can be specified in for each activity parallel processing -- if the concurrent jobs set to 10 ? Thank You, Sir....

  • @CRT26
    @CRT26 7 днів тому

    Good explanation about spark core. Thanks 👍🏻

  • @123featbydeepak
    @123featbydeepak 8 днів тому

    This is what we called real time Teaching. Hoping to be blessed later as well. Also please make a video on transfering data form on premises to another in details steps. Like makeing linked connection n all.

  • @kavitasahu5094
    @kavitasahu5094 18 днів тому

    Hi, I am getting error : java.lang.NoSuchMethodError: scala.collection.immutable.Seq.map(Lscala/Function1;)Ljava/lang/Object;

  • @shyamsundarmolli
    @shyamsundarmolli 19 днів тому

    Sir what is your email you are my guru and amazing explanation sir love you🤗

  • @rushikeshshisode5947
    @rushikeshshisode5947 25 днів тому

    Short and meaningful.....

  • @Amin-tx4ku
    @Amin-tx4ku 27 днів тому

    can we create external location on an existing container ?

  • @bharathmamidisetti3896
    @bharathmamidisetti3896 28 днів тому

    Y sir that much file , it's like confusing

  • @zubairmushtaq7912
    @zubairmushtaq7912 28 днів тому

    please also show the conatiner content from unity cataloag

  • @sridharnallagatla7442
    @sridharnallagatla7442 29 днів тому

    I want to take paid training can you give your number

  • @rohitsharma-bo5yl
    @rohitsharma-bo5yl Місяць тому

    amazing

  • @lnvsprasad
    @lnvsprasad Місяць тому

    Well explained

  • @shyamsundarmolli
    @shyamsundarmolli Місяць тому

    hello Sir can i get the power point presentation which you are sharing for spark

  • @sanjanaringane4159
    @sanjanaringane4159 Місяць тому

    Sir, can you provide the roadmap of azure data engineering, not getting what to do when

  • @koolsrikant92
    @koolsrikant92 Місяць тому

    Can we learn this Fabric directly or any prerequisite for this to learn??

  • @entertainerbhai1111
    @entertainerbhai1111 Місяць тому

    Thankyou Sir for this Data engineering dedicated python playlist. I was searching this for long time. One question... Is this playlist enough for all Data Engineering tasks from basic to advance? Waiting for ur response..

  • @mustapharaimilawal8053
    @mustapharaimilawal8053 Місяць тому

    Thanks for making this video sir, very clear explanation. Thank you sir.

  • @BeeEU-pj4hi
    @BeeEU-pj4hi 2 місяці тому

    Please anyone answer. So I learnt SQL for data engineering, and I want to watch this entire playlist to learn python. My question is after this playlist can I start making data engineering projects and applying for Data engineering internship or do I watch more videos. Please answer I feel kind of lost.

  • @BeeEU-pj4hi
    @BeeEU-pj4hi 2 місяці тому

    Please anyone answer. So I learnt SQL for data engineering, and I want to watch this entire playlist to for python. My question is after this playlist can I start making data engineering projects and applying for Data engineering internship or do I watch more videos. Please answer I feel kind of lost.

  • @BeeEU-pj4hi
    @BeeEU-pj4hi 2 місяці тому

    Please anyone answer. So I learnt SQL for data engineering, and I want to watch this entire playlist to for python. My question is, after this playlist can I start making data engineering projects and applying for Data engineering internship or do I watch more videos. Please answer I feel kind of lost.

  • @mohitupadhayay1439
    @mohitupadhayay1439 2 місяці тому

    Still we could not find the proper reason why the records went into corrupt when the column are very huge

  • @StuckTechIT
    @StuckTechIT 2 місяці тому

    Thank you Ravi to explain things in more simple way! Can we connect to transform myself(web developer 13 years exp. ) into Data Engineering

  • @Tech.S7
    @Tech.S7 2 місяці тому

    Perfect !! thanks a lot for yours efforts for informative video .. Cheers !!

  • @vijaykumar-vx8dp
    @vijaykumar-vx8dp 2 місяці тому

    when i'm trying to use this getting error

  • @user-gs8xh9ww9i
    @user-gs8xh9ww9i 2 місяці тому

    i got , query returns no records

  • @redouanbelghiti8226
    @redouanbelghiti8226 2 місяці тому

    Please how to fix error odi 12308

  • @redouanbelghiti8226
    @redouanbelghiti8226 2 місяці тому

    how to fix error ODI 12308 please

  • @nitropan
    @nitropan 2 місяці тому

    This is also called a managed table right?

  • @user-ue6gu6uk9p
    @user-ue6gu6uk9p 2 місяці тому

    Great, thank you for the video.

  • @chetanmavale663
    @chetanmavale663 2 місяці тому

    your literally awesome bro.....and this content is really meaning for us.....keep it up

  • @Bgmifortimepass
    @Bgmifortimepass 3 місяці тому

    if iwant to take only one file from multiple subfolders. like format would be like this /mnt/test/2024/01/01/.csv files, same like we have different date wise subfolders. what is the approach pls explain me

  • @mohitupadhayay1439
    @mohitupadhayay1439 3 місяці тому

    Can we do the same for XML and JSON files?

  • @akashkadam4814
    @akashkadam4814 3 місяці тому

    plz upload other content on your GitHub lots file is not available......

  • @ajithselvaraju8123
    @ajithselvaraju8123 3 місяці тому

    Nice explanation👌

  • @Tasfiomi
    @Tasfiomi 3 місяці тому

    I'm trying to write to show my gratitude for this incredible entire tutorials ever can help anybody from down to earth but I don't think saying "Thanks" is enough.Really appreciated your efforts and time ! Keep up the good work !

  • @DataEngineerPratik
    @DataEngineerPratik 3 місяці тому

    Unable to Perform in databricks community edition

    • @madhavareddynarala128
      @madhavareddynarala128 2 місяці тому

      yes throwing Erroe: java.lang.NoSuchMethodError: scala.collection.immutable.Seq.map(Lscala/Function1;)Ljava/lang/Object;

  • @DataEngineerPratik
    @DataEngineerPratik 3 місяці тому

    error java.lang.NoSuchMethodError: scala.collection.immutable.Seq.map(Lscala/Function1;)Ljava/lang/Object;

  • @devakumar2911
    @devakumar2911 3 місяці тому

    Hi, I am trying to do the partition on one of the existing table for the year 2023 on Monthly basis so can you please help me with Partition Function and Partition Scheme queries please. Here I am using uploaddate column as a partition key.

  • @LITEMUSICP84
    @LITEMUSICP84 3 місяці тому

    India not for beginners 😂😂

  • @user-sk8vi1xy7q
    @user-sk8vi1xy7q 3 місяці тому

    thanks brother its really helped for beginner's

  • @Skd833
    @Skd833 3 місяці тому

    Can you please sort your videos in this playlist in correct sequence?

  • @amanpal5958
    @amanpal5958 3 місяці тому

    Hi, thanks for the video. In my case my nested struct type column does not contain any nested array type column instead it contains nested struct type. Is there any other way I could use?

  • @chandrashekarlingam9548
    @chandrashekarlingam9548 3 місяці тому

    Please add a text file on these Q&A...couldn't understand clearly..Voice is too low...

  • @Ameem-rw4ir
    @Ameem-rw4ir 3 місяці тому

    bro, thanks for your inputs. can you please help me how to handle this? empid,fname|lname@sal#deptid 1,mohan|kumar@5000#100 2,karna|varadan@3489#101 3,kavitha|gandan@6000#102 Expected output empid,fname,lname,sal,deptid 1,mohan,kumar,5000,100 2,karan,varadan,3489,101 3,kavitha,gandan,6000,102

  • @davidcardenas4266
    @davidcardenas4266 3 місяці тому

    what would be a better option to generate a surrogate key? Edit: I discovered monotonically_increasing_id, please y'all comment if you encountered is better or to propose another one.

  • @MANGESH_AHIRE
    @MANGESH_AHIRE 3 місяці тому

    If we have 4 cores then 8 task can run in parallel right ?

  • @sontijagadeesh3990
    @sontijagadeesh3990 3 місяці тому

    Thanks for uploading video on ODI procedure

  • @reanwithkimleng
    @reanwithkimleng 3 місяці тому

    I ❤❤❤❤