Commit graph

112 commits

Author SHA1 Message Date
Robert Hensing d4f6b1d38b
Merge pull request #9497 from edolstra/move-access-control
Move restricted/pure-eval access control out of the evaluator and into the accessor
2023-12-08 22:21:50 +01:00
Peter Kolloch fc6f29053a Renamed HashFormat::Base32 to HashFormat::Nix32
...and also adjusted parsing accordingly.

Also added CLI completion for HashFormats.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Peter Kolloch 5334c9c792 HashType: Rename to HashAlgorithm
To be consistent with CLI, nix API
and many other references.

As part of this, we also converted it to a scoped enum.

https://github.com/NixOS/nix/issues/8876
2023-12-06 23:43:42 +01:00
Eelco Dolstra 8cafc754d8 Move access control from FSInputAccessor to FilteringInputAccessor 2023-11-30 21:54:53 +01:00
Théophane Hufschmitt 6a94755b12 Allow user input in git commit
We occasionnally commit to git repositories (like with `nix flake update --commit-lock-file`).
This shells out to `git commit`, which might wait for user input (for a signing key passphrase for instance).
Disable the progress bar while this is running to make sure that the
user can enter it.
2023-11-24 11:45:37 +01:00
Eelco Dolstra 38844943d0
Merge pull request #9413 from edolstra/fetchToStore-cache
Persistently cache `InputAccessor::fetchToStore()`
2023-11-22 12:19:25 +01:00
Robert Hensing 9cd69e1c39
Merge pull request #9411 from edolstra/path-display
Improve SourceAccessor path display
2023-11-21 18:12:01 +01:00
Eelco Dolstra 99d5204baa Persistently cache InputAccessor::fetchToStore()
This avoids repeated copying of the same source tree between Nix
invocations. It requires the accessor to have a "fingerprint" (e.g. a
Git revision) that uniquely determines its contents.
2023-11-20 20:04:37 +01:00
Eelco Dolstra a0162d5732 Improve SourceAccessor path display
Backported from lazy-trees. This allows SourceAccessors to show the
origin of the accessor. E.g. we now get

  copying '«git+https://github.com/blender/blender.git?ref=refs/heads/main&rev=4edc1389337dd3679ff66969c332d2aff52e1992»/' to the store

instead of

  copying '/' to the store
2023-11-20 18:54:36 +01:00
Eelco Dolstra e4066c0444 Fetch specific Git revisions
This is more efficient, and necessary when using shallow=1 with a rev.
2023-11-20 17:25:40 +01:00
Eelco Dolstra 7ab91e7238 Implement shallow fetching 2023-11-15 15:00:17 +01:00
Eelco Dolstra 5dd4ae8687 Remove unused cacheType field 2023-11-15 14:08:34 +01:00
Eelco Dolstra 2890999911 Show Git fetch progress 2023-11-15 13:57:20 +01:00
Eelco Dolstra 2964a9f562 Fix relative submodule handling
Tested on

  nix flake prefetch 'git+https://github.com/blender/blender.git?rev=4ed8a360e956daf2591add4d3c9ec0719e2628fe&submodules=1'
2023-11-14 16:00:21 +01:00
Eelco Dolstra 6ec6b8aa36 Improve git submodule error reporting 2023-11-14 15:52:18 +01:00
Eelco Dolstra 7f576f5dfe Rename UnionInputAccessor to MountedInputAccessor 2023-11-14 14:02:57 +01:00
Eelco Dolstra 38b07d6347
src/libfetchers/git.cc: Apply suggestion
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-11-14 13:38:03 +01:00
Eelco Dolstra 98a120b8b8 Merge remote-tracking branch 'origin/master' into libgit2 2023-11-09 16:48:41 +01:00
John Ericson ac89bb064a Split up util.{hh,cc}
All OS and IO operations should be moved out, leaving only some misc
portable pure functions.

This is useful to avoid copious CPP when doing things like Windows and
Emscripten ports.

Newly exposed functions to break cycles:

 - `restoreSignals`
 - `updateWindowSize`
2023-11-05 12:20:02 -05:00
BootRhetoric 098f0615c9
fetchGit and flake: add publicKeys list input
This adds publicKeys as an optional fetcher input attribute to flakes
and builtins.fetchGit to provide a nix interface for the json-encoded
`publicKeys` attribute of the git fetcher.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-11-03 20:23:31 +01:00
BootRhetoric 6df32889a5
Add git commit verification input attributes
This implements the git input attributes `verifyCommit`, `keytype`,
`publicKey` and `publicKeys` as experimental feature
`verified-fetches`. `publicKeys` should be a json string.
This representation was chosen because all attributes must be of type bool,
int or string so they can be included in flake uris (see definition of
fetchers::Attr).
2023-11-03 20:15:12 +01:00
Eelco Dolstra 39ea46abb1 Merge remote-tracking branch 'origin/master' into libgit2 2023-11-02 16:16:37 +01:00
John Ericson 0707db2b1c
Merge pull request #9255 from NixOS/libfetcher-docs-json
libfetcher doc automation
2023-11-01 09:38:02 -04:00
Eelco Dolstra 0c5eac9c45 Git fetcher: Handle submodules for workdirs 2023-10-31 15:59:25 +01:00
Eelco Dolstra 669b074f51 Cleanup 2023-10-31 14:52:21 +01:00
Eelco Dolstra d88106df24 Git fetcher: Improve submodule handling
Instead of making a complete copy of the repo, fetching the
submodules, and writing the result to the store (which is all
superexpensive), we now fetch the submodules recursively using the Git
fetcher, and return a union accessor that "mounts" the accessors for
the submodules on top of the root accessor.
2023-10-31 14:52:21 +01:00
Eelco Dolstra ee36a44bf2 GitInputScheme: Use libgit2
This replaces most calls to the "git" binary with libgit2.
2023-10-31 14:52:21 +01:00
John Ericson 8381eeda6f Systematize fetcher input attribute validation
We now have `schemeName` and `allowedAttrs` functions for this purpose.
We look up the schema with the former; we restrict the set of input
attributes with the latter.
2023-10-30 10:17:24 -04:00
Eelco Dolstra 95d657c8b3 Input: Replace markFileChanged() by putFile()
Committing a lock file using markFileChanged() required the input to
be writable by the caller in the local filesystem (using the path
returned by getSourcePath()). putFile() abstracts over this.
2023-10-25 18:30:29 +02:00
Eelco Dolstra af302267e5 Input::hasAllInfo(): Remove 2023-10-20 17:19:36 +02:00
Yueh-Shun Li e026f3e1ae treewide: Reference HashFormat members with scope
Base* -> HashFormat::Base*
2023-10-19 00:56:41 +08:00
Eelco Dolstra 856fe13533 fetchTree cleanup
Two changes:

* The (probably unintentional) hack to handle paths as tarballs has
  been removed. This is almost certainly not what users expect and is
  inconsistent with flakeref handling everywhere else.

* The hack to support scp-style Git URLs has been moved to the Git
  fetcher, so it's now supported not just by fetchTree but by flake
  inputs.
2023-10-13 14:34:23 +02:00
Eelco Dolstra d9e7758f47 Don't require .tar/.zip extension for tarball flakerefs
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
2023-08-01 16:07:20 +02:00
Michael Hoang a7b49086c7 Add dirtyRev and dirtyShortRev to fetchGit
Fixes #4682
2023-06-24 14:17:25 +10:00
Konstantin Vukolov 31ffd0c1fe Remove unused 2023-05-18 13:26:23 +03:00
Konstantin Vukolov 4c4ae887b8 Add option isInteractive 2023-05-18 13:18:34 +03:00
Konstantin Vukolov 25434df0d9 Ask for git credentials in fetcher 2023-05-17 02:00:32 +03:00
figsoda 31d588db8b respect allRefs=1 when using nix flake prefetch 2023-03-15 11:29:01 -04:00
Eelco Dolstra 862e56c23d
Improve comment
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-09 16:42:45 +01:00
Eelco Dolstra 15313bfdb7
Fix activity message
Co-authored-by: Josef Kemetmüller <josef.kemetmueller@gmail.com>
2023-02-09 16:42:14 +01:00
Eelco Dolstra 7a6daf61e8 Fix activity message 2023-02-07 22:22:50 +01:00
Eelco Dolstra a8fe0dc16c Speed up fetching submodules
Previously we would completely refetch the submodules from the
network, even though the repo might already have them. Now we copy the
.git/modules directory from the repo as an optimisation. This speeds
up evaluating

  builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }

(where /path/to/blender already has the needed submodules) from 121s
to 57s.

This is still pretty inefficient and a hack, but a better solution is
best done on the lazy-trees branch.

This change also help in the case where the repo already has the
submodules but the origin is unfetchable for whatever reason
(e.g. there have been cases where Nix in a GitHub action doesn't have
the right authentication set up).
2023-02-07 16:01:36 +01:00
Eelco Dolstra 2edd5cf618 Fix the origin URL used for fetching submodules
We cannot use 'actualUrl', because for file:// repos that's not the
original URL that the repo was fetched from. This is a problem since
submodules may be relative to the original URL.

Fixes e.g.

  nix eval --impure --json --expr 'builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }'

where /path/to/blender is a clone of
https://github.com/blender/blender.git (which has several relative
submodules like '../blender-addons.git').
2023-02-07 16:01:36 +01:00
Eelco Dolstra 81e75e4bf6 Add some progress indication when fetching submodules 2023-02-07 16:01:36 +01:00
Eelco Dolstra 703d863a48 Trivial changes from the lazy-trees branch 2022-12-07 14:06:34 +01:00
Andrew Brooks a259084c50 Fix #7146
When fetching a non-local git repo by ref (and no rev), don't consider unrelated
cached revs for the same repository.
2022-10-14 18:04:47 -05:00
Eelco Dolstra c80a74b7d5 Don't pass --force to 'git add'
Fixes #5810.
2022-08-17 16:59:02 +02:00
Guillaume Girol e8109cf405 fetchGit: document shallow argument 2022-06-26 12:00:00 +00:00
Naïm Favier da8f8668ca
libfetchers/git: add missing --git-dir flags 2022-06-10 12:57:13 +02:00
Théophane Hufschmitt 027fd45230 Fix a segfault in the git fetcher
The git fetcher code used to dereference the (potentially empty) `ref`
input attribute. This was magically working, probably because the
compiler somehow outsmarted us, but is now blowing up with newer nixpkgs
versions.

Fix that by not trying to access this field while we don't know for sure
that it has been defined.

Fix #6554
2022-05-27 16:15:28 +02:00