#22878 pants lint :: and pants lint /some/path behave differently with ruff, but not isort for sorting python deps Issue created by benceferdinandy-signifyd Describe the bug We're trying to switch from using

black

and

isort

to

ruff

. I'm aware of this bug, but I don't think it's this one, because it does not trigger using

isort

only with

ruff

. For context, our monorepo has multiple resolves and several packages in each resolve, which might affect this. My starting state is

isort

configured for sorting python import. Running

pants lint ::

is green, and running

pants lint /path/to/a/specific/file.py

is also green. After removing

isort

from linting and adding

ruff

the behaviour changes: Running

pants lint ::

is showing a bunch of import sorting errors, including in

/path/to/a/specific/file.py

, but running

pants lint /path/to/a/specific/file.py

does NOT so an error. Running ruff directly (

uv run ruff check /path/to/a/specific/file.py

(version 0.11.5)) also shows the error. This is especially problematic, because

pants fix ::

behaves as the individual file linting (so does

pants fix /path/to/a/specific/file.py

), which means it is not possible to achieve a state where

pants lint ::

and

pants fix ::

agree, if I manually format files by running ruff directly

pants fix

will change it back. Note that running ruff directly on the entire repo, or just a single file will always fix the import order to what

pants lint ::

is expecting. Pants version We're using 2.28, but I can reproduce with 2.31.0.dev1 as well. OS MacOS (I have not tested on linux) Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant. pantsbuild/pants

#22886 Unable to run pants lock when using both python-repos.indexes and pants_requirements Issue created by bobthemighty Describe the bug We use a custom index for installing Torch across architectures, with the following stanza in pants.toml

[python-repos]
indexes = [
    "<https://pypi.org/simple/>",
    "pytorch_cpu=<https://download.pytorch.org/whl/cpu>"
]

We also use pants_requirements to set up the deps for our internal plugins. We have recently updated to take advantage of the PR we raised for multi-arch lockfiles, but we are now unable to run

pants lock

. The root cause, per the error message, seems to be that in this commit the pants_requirements target added an index for pants, which has no alias, and that conflicts with the "pypi.org/simple" index that we have added in pants.toml. It's possible that I'm just holding it wrong, but this smells like a bug. Pants version 2.30 OS Both Additional info

❯ pants generate-lockfiles --resolve=inference
10:49:16.36 [WARN] /home/bob/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0a0/lib/python3.11/site-packages/pants/init/options_initializer.py:14: UserWarning: pkg_resources is deprecated as an API. See <https://setuptools.pypa.io/en/latest/pkg_resources.html>. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

/home/bob/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0a0/lib/python3.11/site-packages/pants/init/options_initializer.py:14: UserWarning: pkg_resources is deprecated as an API. See <https://setuptools.pypa.io/en/latest/pkg_resources.html>. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
10:49:18.10 [INFO] Initializing scheduler...
10:49:18.20 [INFO] Scheduler initialized.
10:49:19.83 [INFO] Completed: Scheduling: Find interpreter for constraints: CPython<3.12,>=3.11
10:49:20.68 [INFO] Completed: Generate lockfile for inference
10:49:20.68 [INFO] Completed: Scheduling: Generate lockfile for inference
10:49:20.68 [ERROR] 1 Exception encountered:

Engine traceback:
  in root
    ..
  in pants.core.goals.generate_lockfiles.generate_lockfiles_goal
    `generate-lockfiles` goal

Traceback (most recent call last):
  File "/home/bob/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0a0/lib/python3.11/site-packages/pants/core/goals/generate_lockfiles.py", line 569, in generate_lockfiles_goal
    results = await concurrently(
              ^^^^^^^^^^^^^^^^^^^
... <SNIP>
pants.engine.process.ProcessExecutionFailure: Process 'Generate lockfile for inference' failed with exit code 1.
stdout:

stderr:
The following names are being re-used across indexes and find links repos:
Package repository names must be unique.

Repro is here: https://github.com/carbon-re/pants-resolve-issue pantsbuild/pants

#22654 Arguments for pex_binary targets to build pex scie's Issue created by gdfast Is your feature request related to a problem? Please describe. The PEX tool offers options to build PEX files that include full python distributions (or lazily retrieves a python distribution). These are compelling options for building PEX files that are maximally portable. pants is commonly used to build PEX files, and I think we should include a way to easily build scie pex files. Describe the solution you'd like A new argument for

pex_binary

and

pex_binaries

targets: pex_binary( ... scie="eager", # or "lazy", default: None ) There possibly needs to be a few other optional args to make this work. I think the python interpreter determined by pants should be the version of python used. Describe alternatives you've considered One alternative is better documentation about how to do this now using extra_build_args, but this isn't very user friendly Additional context Discussion in slack: https://pantsbuild.slack.com/archives/C046T6T9U/p1741182156086479 Relates to #22044, but unlike that one is more about making this user-friendly pantsbuild/pants

#22897 add support for binary build with pyinstaller Issue created by stan-dot Is your feature request related to a problem? Please describe. We'd like to ship

.exe

to non-technical users who use Windows Describe the solution you'd like Another target similar to

pex_binary

is available that if computed, results in an

exe

in the

dist

folder IF ran on Windows (Pyinstaller is not a cross compiler) https://pyinstaller.org/en/stable/ . Describe alternatives you've considered Alternatives: • Currently

.pex

could be shipped, but the users would need to turn on Windows Dev Mode - that is not secure enough • Another way of shipping python apps - Nuitka - would require this issue done #14752 which sounds more complicated. • if we get a simple Python script as a target, perhaps with locked metadata this would be also workable • also https://github.com/ofek/pyapp might be of use here OR • embedded python (windows only ) https://www.python.org/downloads/release/python-3123/ Additional context pantsbuild/pants

#18698 add `DurationOption` type for duration-type config options Issue created by tdyas There are several Pants options which take an integer value (of varying time units) in order to configure duration-related configuration like timeout duration. Some of the options use seconds, others use milliseconds. In some cases, the time unit is encoded into the option name (

-millis

suffix). To reduce user confusion, we should add a

DurationOption

type for these options which could take human-readable time syntax, for example

2mins30secs

or

1234ms

and just convert to a single type internally. pantsbuild/pants

#18502 `help backends` should indicate which backends are used by `update-build-files` Issue created by cognifloyd Is your feature request related to a problem? Please describe.

./pants help backends

shows that

pants.backend.build_files.fix.deprecations

and

pants.backend.build_files.fmt.black

are NOT enabled by default. That surprised me because the

update-build-files

goal definitely uses black. Describe the solution you'd like In the

help backends

output:

Enabled backends are marked with `*`. To enable a backend add it to `[GLOBAL].backend_packages`.

Let's add another character, maybe

-

, to show that a backend is conditionally enabled. The

update-build-files

goal will use a formatter backend without regard to the

[GLOBAL].backend_packages

setting. By default, it uses

pants.backend.build_files.fmt.black

and

pants.backend.build_files.fix.deprecations

. The formatter can be switched from

black

to

yapf

or

buildifier

based on the `[update-build-files].formatter` setting. Using

fix.deprecations

backend is controlled by the `[update-build-files].fix_safe_deprecations` setting (either

True

or

False

). So, these goal-specific backends should get a

-

indicator to show that it is enabled, but only for that one special-purpose goal based on different settings. Describe alternatives you've considered Just remove the backends from the output? That would be confusing too, because the backends exist. Also, you can enable the

pants.backend.build_files.fix.deprecations

backend so that it ALSO runs during the

fix

goal, not just during

update-build-files

. Additional context current output

% ./pants help backends

Backends
--------

List with all known backends for Pants.

Enabled backends are marked with `*`. To enable a backend add it to `[GLOBAL].backend_packages`. 

...
[ ] pants.backend.build_files.fix.deprecations                                              [pants]
[ ] pants.backend.build_files.fmt.black                                                     [pants]
[ ] pants.backend.build_files.fmt.buildifier                                                [pants]
[ ] pants.backend.build_files.fmt.yapf                                                      [pants]
...
[*] pants.backend.python                                                                    [pants]
     Support for Python.
     
     See <https://www.pantsbuild.org/docs/python-backend>.
[ ] pants.backend.python.lint.autoflake                                                     [pants]
     Autoformatter for removing unused Python imports.
     
     See <https://github.com/myint/autoflake> for details.
[*] pants.backend.python.lint.bandit                                                        [pants]
     Security linter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://bandit.readthedocs.io/en/latest/>.
[*] pants.backend.python.lint.black                                                         [pants]
     Autoformatter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://black.readthedocs.io/en/stable/>.
...

proposed output

% ./pants help backends

Backends
--------

List with all known backends for Pants.

Enabled backends are marked with `*`. To enable a backend add it to `[GLOBAL].backend_packages`. 
The `update-build-files` goal uses backends marked with `-` based on `[update-build-files]` settings.

...
[-] pants.backend.build_files.fix.deprecations                                              [pants]
[-] pants.backend.build_files.fmt.black                                                     [pants]
[ ] pants.backend.build_files.fmt.buildifier                                                [pants]
[ ] pants.backend.build_files.fmt.yapf                                                      [pants]
...
[*] pants.backend.python                                                                    [pants]
     Support for Python.
     
     See <https://www.pantsbuild.org/docs/python-backend>.
[ ] pants.backend.python.lint.autoflake                                                     [pants]
     Autoformatter for removing unused Python imports.
     
     See <https://github.com/myint/autoflake> for details.
[*] pants.backend.python.lint.bandit                                                        [pants]
     Security linter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://bandit.readthedocs.io/en/latest/>.
[*] pants.backend.python.lint.black                                                         [pants]
     Autoformatter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://black.readthedocs.io/en/stable/>.
...

pantsbuild/pants

#18613 VS Code Support Issue created by chrisjrn see https://github.com/robotpajamas/suspenders pantsbuild/pants

#22909 Coursier transitive dependencies bug results in incorrect lockfile Issue created by enragedginger Given three dependencies A, B, and C where A depends on B which depends on C and A only depends on C transitively through B (no explicit dependency), the Coursier locking functionality sometimes will not include C in the dependencies of A. For some combinations of A->B->C such as spring-context->spring-core->spring-jcl, the bug does not exist. But for timbre->encore->truss, the bug exists. This seems to happen in situations where you request a version of B from Coursier that isn't the version of B that your version of A wants by default. This one is fine: cs fetch --json-output-file timbre_single.json com.taoenssotimbre6.3.1 -r clojars timbre has both truss and encore in its dependencies This one is fine as timbre 6.3.1 wants encore 3.68.0 by default: cs fetch --json-output-file timbre_double.json com.taoenssotimbre6.3.1 com.taoenssoencore3.68.0 -r clojars Requesting a different version of encore than what timbre wants by default produces the error: cs fetch --json-output-file timbre_bug.json com.taoenssotimbre6.3.1 com.taoenssoencore3.132.0 -r clojars In the first two example, timbre will correctly show both encore and truss in its dependencies. In the third example, timbre will have encore in its dependencies, but truss won't be there. However, if you look at the coord for encore, truss will be present there as a dependency. The issue persists with Coursier 2.1.24. However, the issue is resolved on 2.1.25-M19 (and perhaps an earlier milestone release). So once they cut a release for 2.1.25, we should update. In my personal repo, I've just set the default Coursier version to 2.1.25-M19, and that's fine me for now. pantsbuild/pants

#18291 Add support for RPM Linux packages Issue created by chrisjrn pantsbuild/pants

#22913 go/goals/test 'types.UnionType' object is not iterable in debug log Issue created by sureshjoshi Found this in the debug logs, never seen it before - doesn't seem to error or visibly warn.

pants/src/python/pants/backend/go/goals/test.py:333

Rule visitor failed to inspect assignment expression for ['_exit_code', '_stderr'] - tuple[int, str] | None:

'types.UnionType' object is not iterable

pantsbuild/pants

#22916 Installing pants behind a firewall New discussion created by Niccolum Hi! I want to use pants internally at my company. I've reread the documentation and the source code, but I have several questions related to the company blocking access to all external resources, including GitHub. It's easier to forgive me in advance if it's customary to ask one question per thread, but mine are very related. 1. Can sci-pants only be downloaded via brew and GitHub releases? Is there a way to install it locally by downloading the binary? On linux, for example. Inside the сurl variant, downloads from GitHub releases still occur. 2. I've encountered a problem with sci-pants pulling its own Python versions for its own operation. Can I set a common path to them, or is the only way to specify them in PANTS_BOOTSTRAP_URLS? 3. Can I replace the Python used by uv/pyenv for pants' internal operation? I've seen some news that this might be problematic, but having local copies of all Pythons in an internal repository also seems like a strange idea. Thanks in advance for your answers. I've been looking through the document for several days and still haven't been able to find the answers. Thank you. pantsbuild/pants

#22789 Performance issue with very large set of ad-hoc tool calls Issue created by svenier Transferring from slack thread https://pantsbuild.slack.com/archives/C0D7TNJHL/p1761162382770779 Attempting to build a large graph of 163,375 targets results in pants memory growth and "unbounded" execution time. The source repository is at https://github.com/NVIDIA/Dependency-Patterns/tree/main/scaling and the command to run it is

pants run :go

. I was seeing this performance issue with 2.28.0 running on Linux accessing the files via NFS. Benjy was able to reproduce it running on macOS with local storage (not sure which pants version he was using). Some internal profiling shows lots of time spent waiting on the GIL. We have not done any stack inspections to see if, for example, transitive dependency traversal is doing redundant work for nodes deep in the graph. Our next experiment is going to be to try it against a BuildBarn remote execution farm to see if offloading it makes much difference. As a point of comparison Bazel is able to do the build in 9-10 minutes. pantsbuild/pants

#22918 Add a path to feed bytes to the stdin of a Process Issue created by svenier In our build flow we have some commands that read their input from stdin. With the current Process object, we need to setup and run a shell command instead of directly running our desired command to redirect input from a file. If stdin were handled symmetrically to how stdout/stderr work, we could just directly invoke our desired command without the overhead (populating the sandbox with everything needed to run the shell) or risk (users stuffing shell metacharacters) of setting up a shell. See https://pantsbuild.slack.com/archives/C01CQHVDMMW/p1763760163111359 for a long conversation about this. [edit to fix typos] pantsbuild/pants

#22280 call-by-name syntax does not allow forward references to a rule defined later in file Issue created by tdyas The engine's call-by-name syntax requires that any rule referenced by name must be fully defined before it can be referenced. This causes errors when one rule refers to another rule defined later in the same file. See #22269 (review) for example for this issue causing an error in migrating the Helm backends to call-by-name syntax. The error itself is generated from Rust at pants/src/rust/engine/src/nodes/task.rs Lines 63 to 69 in</pantsbuild/pants/commit/0be5c8020247b8716e2d93a8f140841557af56ca|0be5c80> | let entry = edges.entry_for(&dependency_key).ok_or_else(\|| { | | ----------------------------------------------------------------------- | | // NB: The Python constructor for

Call()

will have already errored if | | //

type(input) != input_type

. | | throw(format!( | | "{call} was not detected in your @rule body at rule compile time." | | )) | | })?; | pantsbuild/pants

#22920 Config MyPy for Django and pure python project in monorepo structure New discussion created by mrvafa I'm facing a configuration problem with Pants + mypy in a monorepo.

pure_python/
django_project/
utils/
pyproject.toml
pants.toml

I’m trying to keep a monorepo layout. The problem arises when executing:

pants check ::

I have set mypy to use django-stubs for type checking inside of django_project, but the Django-specific mypy configuration is being applied to all Python targets including pure_python. This causes errors such as:

django_project.settings not found

because pure_python obviously has no Django settings module. I want Django type-checking with django-stubs only for django_project/. However MyPy seems to apply plugin config globally. This makes non-Django packages fail pants check. Is there a recommended way to scope mypy configuration to only certain targets / source roots so that Django settings aren't required for unrelated packages? Thanks! pantsbuild/pants

Comment on #22916 Installing pants behind a firewall Discussion answered by Niccolum Thanks for the quick replies and help. I, in turn, figured out how to work with all this at the same time, so I'll write about each point: 1. At first, I didn't realize that the scie-pants release is a binary and can be run immediately after downloading it. I thought I needed to install it somehow. But no – I just downloaded it and it worked. 2. I specified cpython-3.9.23+20250612 and cpython-3.11.13+20250612 in PANTS_BOOTSTRAP_URLS. Different ones for different OS and architectures (8 links in total), and it started working just fine. 3. I didn't find a way to do this, but in principle, this isn't a problem if we use PANTS_BOOTSTRAP_URLS, where we can specify all the necessary Pythons. The only thing is, it's all hardcoded, but I can live with that. What I'd like to clarify: 1. Why was PANTS_BOOTSTRAP_URLS invented? What do you think about creating some kind of url_template in pants.toml for Python and pants.pex, respectively? Or alternative [bootstrap].urls in pants.toml? 2. I don't quite understand why pants needs both Python 3.9 and 3.11, and not just 3.11? I downloaded the latest pants 2.29.1, compiled for 3.11 pantsbuild/pants

#16328 Using Pants in a restricted network access environment is non-trivial and not usefully documented. Issue created by jsirois As the result of a a User asking how to set up Pants to work in their environment which is network-locked-down with Artifactory as the only source for artifacts, I went through a full simulation using the example-python. There are likely further quirks for other backends Pants supports, but in this case the question concerned Python in particular. At the least this needs much more thorough documentation. At the best there is some automation and / or wizard-driven aid to make setup easier. The current docs: • https://www.pantsbuild.org/docs/restricted-internet-access • https://github.com/pantsbuild/pants/blob/main/docs/markdown/Using%20Pants/restricted-internet-access.md --- 1st I black-holed all relevant public network sources of artifacts for the Python backend by adding this line to `/etc/hosts`:

127.0.0.1	<http://files.pythonhosted.org|files.pythonhosted.org> <http://pypi.org|pypi.org> <http://github.com|github.com>

My aim was to get

./pants fmt lint check test ::

working. That failed of course and I went through alot of trial and error with insider knowledge and wound up with these changes: On the simulated corporate user side I had to modify Artifactory by making a generic repo and copying over the relevant Pex PEX release from https://github.com/pantsbuild/pex/releases/download/v2.1.90/pex. I just used the default Artifactory setup for proxying / mirroring PyPI. On the Pants side, I made these edits (The user was attempting with Pants 2.12.0 and the example-python repo was on 2.8.0 at the time): $ git diff pants pants.toml diff --git a/pants b/pants index 2e9a10c..759c580 100755 --- a/pants +++ b/pants @@ -34,9 +34,9 @@ fi PANTS_BOOTSTRAP="${PANTS_SETUP_CACHE}/bootstrap-$(uname -s)-$(uname -m)" -PEX_VERSION=2.1.42 -PEX_URL="[https://github.com/pantsbuild/pex/releases/download/v${PEX_VERSION}/pex](https://github.com/pantsbuild/pex/releases/download/v$%7BPEX_VERSION%7D/pex)" -PEX_EXPECTED_SHA256="69d6b1b1009b00dd14a3a9f19b72cff818a713ca44b3186c9b12074b2a31e51f" +PEX_VERSION=2.1.90 +PEX_URL="[https://pextest.jfrog.io/artifactory/default-generic//pantsbuild/pex/releases/download/v${PEX_VERSION}/pex](https://pextest.jfrog.io/artifactory/default-generic//pantsbuild/pex/releases/download/v$%7BPEX_VERSION%7D/pex)" +PEX_EXPECTED_SHA256="2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66" VIRTUALENV_VERSION=20.4.7 VIRTUALENV_REQUIREMENTS=$( @@ -244,7 +244,7 @@ function bootstrap_pex { local staging_dir staging_dir=$(tempdir "${PANTS_BOOTSTRAP}") cd "${staging_dir}" - curl -LO "${PEX_URL}" + curl -n -LO "${PEX_URL}" fingerprint="$(compute_sha256 "${python}" "pex")" if [[ "${PEX_EXPECTED_SHA256}" != "${fingerprint}" ]]; then die "SHA256 of ${PEX_URL} is not as expected. Aborting." @@ -270,7 +270,9 @@ function bootstrap_virtualenv { staging_dir=$(tempdir "${PANTS_BOOTSTRAP}") cd "${staging_dir}" echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt - "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex + "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \ + --no-pypi \ + --index https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/ mkdir -p "$(dirname "${bootstrapped}")" mv -f "${staging_dir}/virtualenv.pex" "${bootstrapped}" rm -rf "${staging_dir}" diff --git a/pants.toml b/pants.toml index c62c9d4..8cc2ee7 100644 --- a/pants.toml +++ b/pants.toml @@ -2,7 +2,7 @@ # Licensed under the Apache License, Version 2.0 (see LICENSE). [GLOBAL] -pants_version = "2.8.0" +pants_version = "2.12.0" backend_packages.add = [ "pants.backend.python", "pants.backend.python.lint.docformatter", @@ -27,8 +27,36 @@ root_patterns = ["/"] interpreter_constraints = [">=3.7"] # Use a constraints file. See https://www.pantsbuild.org/docs/python-third-party-dependencies. requirement_constraints = "constraints.txt" + +[python-bootstrap] # We search for interpreters on both on the $PATH and in the

$(pyenv root)/versions

folder. # If you're using macOS, you may want to leave off the <PATH> entry to avoid using the # problematic system Pythons. See # https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path. -interpreter_search_paths = ["<PATH>", "<PYENV>"] +search_path = ["<PATH>", "<PYENV>"] + +[pex-cli] +url_template = "https://%(env.ARTIFACTORY_USER)s:%(env.ARTIFACTORY_PASS)s@pextest.jfrog.io/artifactory/default-generic/pantsbuild/pex/releases/download/{version}/pex" + +[python-repos] +indexes = [ + "https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/", +] + +[black] +lockfile = "tools/black.lock" + +# Work around https://github.com/psf/black/issues/2964 +extra_requirements = "click==8.0.4" + +[pytest] +lockfile = "tools/pytest.lock" + +[isort] +lockfile = "tools/isort.lock" + +[flake8] +lockfile = "tools/flake8.lock" + +[mypy] +lockfile = "tools/mypy.lock" With all that set up, bootstrapping Pants required a 1-time:

$ PIP_INDEX_URL=<https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/> ./pants

And then I could re-gen lockfiles to pick up Artifactory urls:

$ ./pants generate-lockfiles
09:17:52.68 [INFO] Completed: Generate lockfile for isort
09:17:55.46 [INFO] Completed: Generate lockfile for flake8
09:17:56.13 [INFO] Completed: Generate lockfile for mypy
09:17:56.86 [INFO] Completed: Generate lockfile for black
09:18:09.91 [INFO] Completed: Generate lockfile for pytest
09:18:09.91 [INFO] Wrote lockfile for the resolve `pytest` to tools/pytest.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `black` to tools/black.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `flake8` to tools/flake8.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `isort` to tools/isort.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `mypy` to tools/mypy.lock

And, finally, I could achieve the initial goal:

$ ./pants fmt lint check test ::
09:18:32.32 [INFO] Completed: Format with docformatter - docformatter made no changes.
09:18:32.32 [INFO] Completed: Format with Black - black made no changes.
09:18:32.32 [INFO] Completed: Format with isort - isort made no changes.

✓ black made no changes.
✓ docformatter made no changes.
✓ isort made no changes.
09:18:32.32 [INFO] Completed: Lint with Flake8 - flake8 succeeded.

✓ black succeeded.
✓ docformatter succeeded.
✓ flake8 succeeded.
✓ isort succeeded.
09:18:32.33 [INFO] Completed: Typecheck using MyPy - mypy succeeded.
Success: no issues found in 8 source files


✓ mypy succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/greet/greeting_test.py:tests succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/translator/translator_test.py:tests succeeded.

✓ helloworld/greet/greeting_test.py:tests succeeded in 0.21s (memoized).
✓ helloworld/translator/translator_test.py:tests succeeded in 0.16s (memoized).

pantsbuild/pants

#12458 Pants lockfile generation includes un-used dists and thus un-vetted dists. Issue created by jsirois Pants uses pip-compile to generate lockfiles and pip-compile includes - apparently - all dists for a given version of a requirement whether they were actually used in the resolve or not. This is a security and stability problem. For example, if the original requirement is

foo>=1.0.0

and the IC is

CPython==3.7.*

the lockfile might contain

foo==1.0.0

with hashes for the sdist and the cp37m wheels. Say the lockfile was generated with and later tested with a CPython 3.7 interpreter with the pymalloc extension; so the wheel is what is actually resolved and tested. The sdists will then go unused and untested. As such a CPython 3.7 without the pymalloc extension lockfile consumer will resolve and build the sdist - potentially years later. In normal cases the sdist will be faithful to the cp37m wheel and generate a cp37 wheel that is ~equivalent. Even so; there is no guaranty this wheel behaves the same - there may be code that is conditional upon the pymalloc extension presence and be buggy in the non-presence branch. Worse - the wheels could be a honeypot and the sdist the trap (see: https://docs.google.com/document/d/17Y6_YjLqv6hY5APWLaRw5CMvr_FVDBIWK6WS57mioNg/edit?usp=sharing for related concerns). It appears to be the case that poetry.lock and Pipfile.lock have the same issue. pantsbuild/pants

#11965 Show progress in addition to the timer in dynamic UI swimlane lines. Issue created by jsirois In particular, when a Process execution is being reported in a swim lane, it might be nice to see something like:

⠴ [11.26s 6.1% cpu 1MB/s down 0MB/s up | Resolving ...

And then maybe even rollups in the log lines that stick above the dynamic UI when appropriate:

15:38:33.47 [INFO] Completed [26.12s 5.2% cpu 25MB down 0MB up | Building requirements.pex

pantsbuild/pants

#11963 Use the engine logging crate to setup logging in the Rust pants client. Issue created by jsirois As a follow up on #11831 switch from the current Rust pants client

env_logger

hack to our

logging

crate so we respect all pants logging configuration knobs. pantsbuild/pants

#11961 Set up group_imports = "StdExternalCrate" in rustfmt.toml Issue created by jsirois Docs here: https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#stdexternalcrate This is only available in rustfmt nightly as of

beta-2021-04-07

. pantsbuild/pants

#11960 De-dup Pants client logic between Rust and Python in favor of Rust. Issue created by jsirois As a follow up on #11831, expose the logic the rust client duplicates from the Python client via cpython so that the Python client can consume it from a central location. This includes at least the calculation of the host hash. pantsbuild/pants

#11339 Execution of user .py files in any Process requires a fresh .pyc compile Issue created by jsirois Right now we do not have a rule that implements a

.pyc

compile step and so we suffer recompilation of

.pyc

files on every run of a given

.py

source code file. It may make sense to implement compile rules that, given the target interpreter, compiles a file or set of files and leaves an output in the cas that can be used to gather up the needed

.pyc

files for composition into downstream process input digests. As always, the hard thing here to get right will likely be the performance tradoff of doing the simplest thing - compiling each source file on its own - to get maximum cache hit rates vs the performant thing which is likely compiling files in batch. pantsbuild/pants

#11324 Transitive excludes on python requirements do not work for external libraries Issue created by jsirois Transitive dependency excludes do not work for packages defined in external librarie's

requirements.txt

Given this in `3rdparty/python/BUILD`:

python_requirement_library(
   name="somelib",
   requirements=["somelib==1.0.0"]
)

python_requirement_library(
   name="someotherlib",
   requirements=["someotherlib==2.0.0"]
)

where `somelib`'s

requirements.txt

includes

someotherlib

as a requirement, i.e.:

requirements.txt

of somelib:

someotherlib==2.0.0

and this in

src/mylib/BUILD

python_library(name="base")

python_awslambda(
    name="awslambda",
    dependencies=[
        ":base",
        "3rdparty/python:somelib",
        "!!3rdparty/python:someotherlib",
    ],
    runtime="python3.8",
    handler="lambda_function:lambda_handler",
)

and running

./pants package src/mylib:awslambda

, the resulting package still includes

someotherlib

(most likely because

somelib

needs it in its own

requirements.txt

, and

pants

does not look like it's aware of that.). For my use case, I'm trying to use an external library which makes use of

boto3

and `botocore`:

aws-secretsmanager-caching

, and I need the lambda package to not contain the `boto3`and `botocore`packages, because they take too much space in the lambda package and the lambda runtime already includes them out of the box. The current workaround I have is to clone the `aws-secretsmanager-caching`library, convert it to a pants accepted build folder using BUILD files, and then the exclude works as expected, however this is not ideal since it copies source code from another library into my project. Any better way to do this ? Is this a feature that's supposed to be supported by pants ? Thanks for your feedback. pantsbuild/pants

#9736 Break out a `pantsbuild.pants.api` dist. Issue created by jsirois People wishing to extend Pants with plugins must use APIs embedded in the

pantsbuild.pants

distribution. That distribution contains much more than a plugin author needs including 3rdparty dependencies that are "fake" (they are not dependencies of the API) and yet still constrain the plugin author's code base. We should maintain a

pantsbuild.pants.api

distribution that contains just the code and dependencies needed to write a plugin. Plugin authors can depend on just that distribution. An example of the current bandaiding that's needed is in #9735. pantsbuild/pants

#9386 Pants flaky-test policy Issue created by jsirois This issue just serves to document the current Pants flaky test policy. If that policy changes over time, this issue can be updated. # Flaky Test Policy (As of 3/24/2020 as discussed here: https://groups.google.com/d/msg/pants-committers/D-9zkIa9zbQ/KO0pCwf4AAAJ): When a test flakes in CI or the test is already associated with a

flaky-test

issue, feel free to: • If the test is a python test, decorate it with: @pytest.mark.skip(reason="[pointer]") Or: @unittest.skip("[pointer]") • If the test is a rust test rust add: #[ignore] // [pointer] The

[pointer]

can just be

flaky

, but it could also be

flaky: [url]

where the url points to a github issue labelled `flaky-test` if you intend to work on the flaky test or would like to leave extra info about how the test flakes for someone who will pick up the work. If the skipped test has a

flaky-test

issue associated, please leave the issue open but further label it `test-skipped`. The issue can be closed when the test is deleted or fixed and un-skipped. If a test target is flaky due to timeout it's harder to place blame on any individual test in the suite. You may not be in a good position to skip the test, but use your judgement. At the least, please also label any associated issue with `test-timeout`. pantsbuild/pants

#9206 Pants should shade Python tool binaries. Issue created by jsirois We have all the same problems with running python tools against python code that we did with running jvm tools against jvm code. pantsbuild/pants

#7942 `regex-lint` (formerly `validate`) should allow path blocklisting Issue created by jsirois Right now, content checks are restricted to path sets built up from path allow lists. Often it would be more robust to apply a content check to all files of a certain type save some small handfull. For those cases adding to a blocklist is both more concise and robust than enumerating allow listed directories. Concretely, this sort of config today: x-path-pattern: &path-pattern content_encoding: utf8 path_patterns: # We use several python path patterns here to avoid src/python/kythe which contains 3rdparty code # that we don't have license over (python proto stubs for kythe-licensed protobuf files). python_prod: <<: *path-pattern pattern: prod/.+(?<!init)\.py$ python_source: <<: *path-pattern pattern: src/python/toolchain/.+(?<!init)\.py$ python_test: <<: *path-pattern pattern: test/.+(?<!init)\.py$ content_patterns: python_header: # NB: We match an optional shebang line here. pattern: |- ^(#![^\n]+ )?# Copyright © 20\d\d Toolchain Labs, Inc\. All rights reserved\. # # Toolchain Labs, Inc\. CONFIDENTIAL # # This file includes unpublished proprietary source code of Toolchain Labs, Inc\. # The copyright notice above does not evidence any actual or intended publication of such source code\. # Disclosure of this source code or any related proprietary information is strictly prohibited without # the express written permission of Toolchain Labs, Inc\. required_matches: python_prod: - python_header python_source: - python_header python_test: - python_header Could be written as follows with blacklists: x-path-pattern: &path-pattern content_encoding: utf8 path_patterns: # We don't have license over the python files under src/python/kythe (python proto stubs for # kythe-licensed protobuf files). python_source: <<: *path-pattern pattern: (?<!init)\.py$ exclude: src/python/kythe/.+\.py$ content_patterns: python_header: # NB: We match an optional shebang line here. pattern: |- ^(#![^\n]+ )?# Copyright © 20\d\d Toolchain Labs, Inc\. All rights reserved\. # # Toolchain Labs, Inc\. CONFIDENTIAL # # This file includes unpublished proprietary source code of Toolchain Labs, Inc\. # The copyright notice above does not evidence any actual or intended publication of such source code\. # Disclosure of this source code or any related proprietary information is strictly prohibited without # the express written permission of Toolchain Labs, Inc\. required_matches: python_source: - python_header pantsbuild/pants

#6676 Better support refactoring of targets that `provide` Issue created by jsirois on <!date^1540402165^{date_short}|2018-10-24T17:29:25Z> One strength of pants is that it makes it fairly easy to create fine grained targets. When the targets are internal - just used by pants for building and tests - pants adjusts to changes in target contents (target becomes fatter or thinner) seamlessly, allowing the developer to do the right thing. Unfortunately, once a target is published, there is a third party involved that cannot handle the refactoring - the artifact resolver in play. At this point you would hope Pants either: a. Provide useful friction to the refactor of the providing target to highlight the issues ahead. b. Provide support for publishing and/or consuming the refactored artifacts in a safe way c. Decouple target granularity from publish granularity At the time of writing, Pants provides oblique, un-useful friction, forcing the aggregator to own a dummy file; IOW a bad version of 'a'. It would be novel to support 'b' somehow. The python stack supports c but the jvm stack does not, see below: #6676 (comment) pantsbuild/pants

#6215 Execute all CI shards in a uniform Docker container and make it easy to dev against it. Issue created by jsirois Ala dtox: pex-tool/pex#526 Relates to #2162 pantsbuild/pants