class: title # Why it took 4 years to get a lock files spec ## Brett Cannon ## [EuroPython 2025](https://ep2025.europython.eu/) (Prague) ### https://opensource.snarky.ca/Talks/2025/EuroPython/Slides  --- class: title # Why it took 4 years to get a lock files spec ## Brett Cannon ## [EuroPython 2025](https://ep2025.europython.eu/) (Prague) ### https://opensource.snarky.ca/Talks/2025/EuroPython/Slides  --- # Thanks - EuroPython ✉️ - Microsoft ✈️🏨🥗📅 - [GitHub Sponsors](https://github.com/sponsors/brettcannon) ✈️ - [Astral](https://astral.sh) - [traal](https://github.com/harkabeeparolus), [Timothée Mazzucotelli](https://github.com/pawamoy), [Christian Heinze](https://github.com/christian-hnz), [Mike Fielder](https://github.com/miketheman) - My family ✈️📅 - Russell Keith-Magee & [BeeWare](https://beeware.org/) 💾 --- class: title # Getting something installed --- class: title # Choosing _how_ to install --- # Source tree ``` example/ ├── pyproject.toml └── src └── example.py ``` .footnotes[ I do not condone using a `src` layout, it just helps in two slides. ] --- # `pyproject.toml` ```toml [project] name = "example" version = "42" [build-system] requires = ["pdm-backend"] build-backend = "pdm.backend" ``` --- # Source distribution ## AKA "sdist" ``` example-42.tar.gz ├── PKG-INFO ├── pyproject.toml └── src └── example.py ``` --- # `PKG-INFO` ``` Metadata-Version: 2.1 Name: example Version: 42 ``` --- # Wheel ``` example-42-py3-none-any.whl ├── example-42.dist-info │   ├── entry_points.txt │   ├── METADATA │   ├── RECORD │   └── WHEEL └── example.py ``` ??? Structure of a wheel file name --- class: title # Choosing _what_ to install --- class: title class: title # Putting it all together ``` example[bonus] > 38; python_version < "3.14" ``` --- class: title # Project ``` example ``` --- class: title # Version requirements ``` example > 1.2.3 ``` --- class: title # Extras ``` example[bonus] ``` --- class: title # Environment markers ``` example; python_version < "3.14" ``` .footnotes[ This is **not** the same as `requires-python`! ] --- class: title # This is why resolving is hard! - Do the environment markers apply? - What about version requirements? - Is there a wheel? What about an sdist? Source tree? - Do you prioritize newer versions or wheels? - Indirect dependencies add their own requirements - New dependencies → might need to redo choices - This is an NP-complete problem! ??? NP-complete due to backtracking --- class:title # What goes into a lock file? --- # Direct dependencies in `pyproject.toml` - `project.dependencies` - `[project.optional-dependencies]` (aka extras) - `[dependency-groups]` .footnotes[ Where you write down your **direct** dependencies ] --- # Where do you write down **ALL** of your dependencies? - `requirements.txt` - `poetry.lock` - `pdm.lock` - `uv.lock` .footnotes[ Don't forget your dependencies have dependencies! ] --- class: title # Couldn't we have just **ONE** lock file format? ## 🤔 --- class: title # Mostly! ## Some lock files use stuff that isn't standardized (yet?) 😅 ## Apparently it takes 4 years of work to get this far👴 ### https://packaging.python.org/en/latest/specifications/pylock-toml --- # Design goals - Written by software, readable by people - Secure by default - Fast to install from - No resolving - _Lockers_ and _installers_ can be different tools - Installers don't have to be written in Python - Support _single-use_ and _multi-use_ environment scenarios --- class: title # `pylock.toml` ## `pylock.*.toml` --- class: title # File-level details ```toml lock-version = "1.0" environments = ["..."] requires-python = "..." extras = ["..."] dependency-groups = ["..."] default-groups = ["..."] created-by = "..." ``` --- class: title ```toml [[packages]] name = "..." version = "..." marker = "..." requires-python = "..." ``` .footnotes[ `extras` and `dependency_groups` added to environment markers ] --- class: title ```toml [packages.vcs] type = "..." url = "..." path = "..." requested-revision = "..." commit-id = "..." subdirectory = "..." ``` .footnotes[ Based on `direct_url.json` spec ] ??? Stéphane Bidoul, Chris Jerdonek; `direct_url.json` You can't search for a source tree, so it's always installed _directly_ --- class: title ```toml [[packages.directories]] path = "..." editable = ... subdirectory = "..." ``` --- class: title ```toml [packages.archive] url = "..." path = "..." size = ... upload-time = ... hashes = {...} subdirectory = "..." ``` --- class:title ```toml [packages] index = "..." ``` .footnotes[ Sdists and wheels can be found on an index server (unlike source trees) ] --- class: title ```toml [packages.sdist] name = "..." upload-time = ... url = "..." path = "..." size = ... hashes = {...} ``` --- class: title ```toml [[packages.wheels]] name = "..." upload-time = ... url = "..." path = "..." size = ... hashes = {...} ``` --- class: title ```toml [[packages.attestation-identities]] kind = "..." ... ``` ??? PEP 740 by William Woodruff, Facundo Tuesca, and Dustin Ingram --- class: title ```toml [[packages.dependencies]] ``` --- class: title ```toml [packages.tool] ``` --- class: title ```toml [tool] ``` --- # What is supported by the spec? - Artifacts - Source trees - Sdists - Wheels - Multiple environments in a single lock file - Dependency groupings - Extras - Dependency groups --- # Installation pseudocode ```Python for pkg in packages: if not markers_apply(pkg): continue elif not python_supported(pkg): raise UnsupportedPythonVersionError elif version_conflict(pkg, to_install): raise AmbiguityError else: to_install.add(pkg) for pkg in to_install: install(pkg) ``` .footnotes[ All of this can be done concurrently! ] --- class: title # Why the heck did _that_ take 4 years?!? --- class: title # It all started on an old website ...  --- class: title # 2019 (106 posts) ## `requirements.txt` v2 --- class: title # 2020 (43 posts) ## `requirements.txt` v2 continues .footnotes[ I was busy helping with PEP 621 this year. ] --- class: title # 2021 (359 posts) ## [PEP 665](https://peps.python.org/pep-0665/) written ??? Jan: research started w/ Tzu-Ping and Pradyun Jul: posted PEP --- # PEP 665 - Wanted security and reproducibility - Only wheels; no sdists or source trees - Assumption is you would build wheels out-of-band and lock to those --- class: title # 2022 (106 posts) ## PEP 665 rejected ??? Jan: PEP rejected Community didn't like sdist being left out. People also wanted locking of build back-ends to help with reproducibility. --- class: title # 2023 (54 posts) ## Striking out on my own ??? Decided to do it MY way, and along the way implement what pip does as libraries. `packaging.metadata` Start writing a resolver using `resolvelib` --- class: title # 2024 (974 posts) ## [PEP 751](https://peps.python.org/pep-0751/) ??? Feb: uv is announced Apr: become a parent Jul: PEP posted --- # PEP 751, take 1 - Based on `pdm.lock` (which is based on `poetry.lock`) - Set of projects to install - Lock for multiple environments - Lock for wheels **and** sdists --- # PEP 751, take 2 - A graph like `uv.lock` - Otherwise the feature set of v1 ??? uv was after more flexibility, so moving to a graph gave them that as it mirrored that flexibility. --- class: title Hynek said ...  -- ... which led to Charlie Marsh (of Astral) saying ...  --- class: title # 😫 -- ## `#NeverFlyWithHynek` -- ## `#NeverLetHynekCommentOnYourPEP` .footnotes[ Also, `#NeverGoForDinnerWithHynekWhenYouHaveAKeynoteTheNextMorning`. ] --- # PEP 751, take 3 (2025 edition) - Back to a set of projects - Only works with a **single** environment - No extras or dependency groups - I'm willing go strip out what's necessary to get the PEP accepted - Paul Moore said he would rather that anyway - I became very strict about adding features and getting responses to questions - Andrea was tired of hearing about lock files - I set a personal deadline of Mar 31 --- class: title # Agreement! --  --- # Can we have our 🍰? - Support extras and dependency groups - Multi-use environment support .footnotes[ Or is [the 🍰 a lie](https://en.wikipedia.org/wiki/The_cake_is_a_lie)? ] --- class: title # 2025 (150 posts) ## PEP 751 accepted! ??? Mar 31: PEP accepted as-is! Apr: Spec posted --- class: title # 4 years and 1.8K posts later ... - Primary lock file - [PDM](https://pdm-project.org/) (opt-in) - Installation - PDM, [uv](https://docs.astral.sh/uv/) - Creation - PDM, uv, [pip](https://pip.pypa.io/) .footnotes[ It's currently less than 4 months since the PEP was accepted! ] --- class: title # Q&A ## https://snarky.ca