Git LFS - How to handle large Git repos?

Video: https://youtu.be/YQzNfb4IwEY

Git became really successful in the Open Source world and, more recently, in the corporate world as well. However, Git is not suited for large files by design which is a major obstacle for many adopters. GitLFS is an extension to Git which addresses this problem. In this talk I will explain how GitLFS works and how Autodesk uses it at scale for 4000+ engineers and 1000+ repositories. You will also learn how to use it properly and how to avoid the most common pitfalls. If you have worked with Git before then this talk will prepare you to handle large files without trouble in Git!

Transcript

1. GIT LFS
   - How to handle large Git repos? -
   Lars Schneider
   Technical Lead 

   for Git at Autodesk
   Git and Git LFS
   contributor
   @kit3bus
   

   View Slide

2. What is a
   large Git repo?
   

   View Slide

3. Large number of files
   https://github.com/Microsoft/GVFS
   

   View Slide

4. Large number of commits
   

   View Slide

5. Large files
   

   View Slide

6. Why are large files a problem?
   = 100 MB video file
   

   View Slide

7. Clone:
   Large files in Git
   master
   301MB
   + 100 MB
   + 100 MB
   1 MB
   100 MB
   1 MB
   100 MB
   1 MB
   100 MB
   

   View Slide

8. 301 MB 

   x 

   Number of Engineers
   Clone:
   Large files in Git
   1 10
   1 10
   1 10
   ...
   Git Server
   

   View Slide

9. Large files in Git LFS
   Large File Storage

   https://git-lfs.github.com
   

   View Slide

10. LFS Server
    Clone:
    Large files in Git LFS
    1MB
    master
    1 MB
    100 MB
    1 MB
    1 MB
    100 MB
    0 MB
    0 MB
    0 MB
    100 MB
    

    View Slide

11. LFS Server
    Large files in Git LFS
    master
    1 MB
    100 MB
    1 MB
    1 MB
    100 MB
    0 MB
    0 MB
    0 MB
    100 MB
    Clone: 1 MB
    + Checkout: 100 MB
    

    View Slide

12. LFS Server
    Large files in Git LFS
    master
    1 MB
    100 MB
    1 MB
    1 MB
    100 MB
    0 MB
    0 MB
    0 MB
    100 MB
    Clone: 1 MB
    + Checkout: 100 MB
    60% saving!
    

    View Slide

13. How to use Git LFS?
    

    View Slide

14. How to use Git LFS?
    tracking only based on file name,
    file size is not relevant!
    |- .git
    |
    |- code.cpp
    - readme.md
    - video.mp4
    $ git lfs track "*.mp4"
    

    View Slide

15. How to use Git LFS?
    |- .git
    |- .gitattributes
    |- code.cpp
    - readme.md
    - video.mp4
    $ git lfs track "*.mp4"
    $ git add .
    

    View Slide

16. How to use Git LFS?
    |- .git
    |- .gitattributes
    |- code.cpp
    - readme.md
    - video.mp4
    $ git lfs track "*.mp4"
    $ git add .
    $ git commit
    

    View Slide

17. How to use Git LFS?
    |- .git
    |- .gitattributes
    |- code.cpp
    - readme.md
    - video.mp4
    $ git lfs track "*.mp4"
    $ git add .
    $ git commit
    $ git push
    

    View Slide

18. How to use Git LFS?
    |- .git
    |- .gitattributes
    |- code.cpp
    - readme.md
    - video.mp4
    $ git lfs track "*.mp4"
    $ git add .
    $ git commit
    $ git push
    ! That's it!
    

    View Slide

19. Autodesk uses Git LFS
    

    View Slide

20. Autodesk
    uses 

    Git LFS
    • Best known for AutoCAD

    2D and 3D computer-aided design

    • 35 years in business

    • 4000 engineers, hundreds of products,
    terabytes of code and asset data
    

    View Slide

21. • Integration Test Data

    (3D Models, ...)

    • Auxiliary Data

    (Documentation, Images, Videos, ...)

    • Build Artifacts

    (not recommended)
    Autodesk
    uses 

    Git LFS
    

    View Slide

22. What have we learned ?
    22
    

    View Slide

23. Developer
    (INCLUDES DESIGNER, TESTER, ...)
    

    View Slide

24. What is too large for Git?
    

    View Slide

25. What files should go to LFS?
    Mon Tue Wed Files that 

    do not compress well
    and
    change frequently
    Don't worry about files
    smaller than 500kb
    

    View Slide

26. git lfs track "*.png"
    How to track Git LFS files?
    Use Git 2.11+ and Git LFS 1.5+
    for 80x speed improvement! 

    http://autode.sk/git-filter
    

    View Slide

27. git lfs track "*.lfs.*"
    e.g. /images/mountain.lfs.png
    How to track Git LFS files?
    

    View Slide

28. git lfs track "/big/*"
    e.g. /big/mountain.png
    How to track Git LFS files?
    

    View Slide

29. git lfs track "/mountain.png"
    How to track Git LFS files?
    

    View Slide

30. How to track Git LFS files?
    git lfs track "*.png"
    Case sensitive:
    Win/macOS: demo.PNG
    Linux: demo.PNG
    

    View Slide

31. How to track Git LFS files?
    git lfs track "*.png"
    Case sensitive:
    git lfs track "*.[pP][nN][gG]"
    Case in-sensitive:
    Use "git lfs ls-files"
    to check what
    files are tracked!
    Win/macOS: demo.PNG
    Linux: demo.PNG
    

    View Slide

32. Git LFS Gotchas
    

    View Slide

33. Gotchas
    Speed up Clones
    git lfs clone
    I'm working on making 

    this wrapper obsolete!
    

    View Slide

34. Gotchas
    Setup your Git credential helper 

    (or use SSH)!
    

    View Slide

35. Gotchas
    No line ending conversions 

    on Git LFS files!
    FirstLine
    SecondLine
    FirstLine▯Second
    Line

    Windows
    macOS
    

    View Slide

36. Gotchas
    $ git
    Your branch is up-to-date with 'origin/master'.
    Encountered 2 file(s) that should have been pointers, but weren't:
    path/to/some/file.PNG
    other/image.PNG
    

    View Slide

37. Administrator
    38
    MANAGING GIT REPOS
    

    View Slide

38. Setup Git LFS 

    on dev machines!
    Enterprise Config for Git

    https://autode.sk/git
    

    View Slide

39. Distribute recent
    Git / Git LFS versions
    

    View Slide

40. Configure a file size limit on
    your Git server!
    

    View Slide

41. Takeaways
    • Git LFS works at scale
    • Use smart tracking patterns
    • Speed up clones with `git lfs clone`
    • Reject large files on the Git server
    Questions? @kit3bus
    

    View Slide