mirror of
https://github.com/openai/codex.git
synced 2026-08-23 13:09:46 +00:00
Route Bazel CI through shared BuildBuddy remote config wrapper (#25156)
## Why Bazel remote configuration was selected in several CI scripts and workflow steps. That made the BuildBuddy tenant policy easy to duplicate and harder to audit, especially for fork pull requests that must not use the OpenAI tenant. This builds on [sluongng/buildbuddy-ci-host-routing](https://github.com/openai/codex/compare/main...sluongng:codex:sluongng/buildbuddy-ci-host-routing) and consolidates the policy in one place. ## What to do if this breaks you See `codex-rs/docs/bazel.md` for details. TLDR: 1. make a BuildBuddy API key and put it in `~/.bazelrc` 2. if you're an OpenAI employee, add `common --config=buildbuddy-openai-rbe` to `user.bazelrc` in the repo root Run `just bazel-test` to ensure it works. Note that `just bazel-remote-test` no longer exists, you need to select a remote configuration as documented to use RBE. ## What changed - Add `.github/scripts/run_bazel_with_buildbuddy.py` as the shared Bazel wrapper and Python library. It selects the OpenAI host only for trusted upstream GitHub Actions runs, routes keyed fork runs to the generic host, and falls back to local Bazel execution when no key is available. - Move endpoint selection into explicit `.bazelrc` configurations and update Bazel CI, query helpers, and `rusty_v8` staging to use the shared policy. Loading-phase target-discovery queries remain local. - Add wrapper and `rusty_v8` unit coverage, plus `just test-scripts` for the `.github/scripts` Python tests. - Document local Bazel usage, `user.bazelrc` setup, BuildBuddy configurations, and CI behavior in `codex-rs/docs/bazel.md`. ## Validation - `just test-scripts` - `bash -n .github/scripts/run-bazel-ci.sh .github/scripts/run-bazel-query-ci.sh .github/scripts/run-argument-comment-lint-bazel.sh scripts/list-bazel-clippy-targets.sh` - `python3 -m py_compile .github/scripts/run_bazel_with_buildbuddy.py .github/scripts/test_run_bazel_with_buildbuddy.py .github/scripts/test_rusty_v8_bazel.py .github/scripts/rusty_v8_bazel.py` - `ruff check .github/scripts/run_bazel_with_buildbuddy.py .github/scripts/test_run_bazel_with_buildbuddy.py .github/scripts/test_rusty_v8_bazel.py .github/scripts/rusty_v8_bazel.py`
This commit is contained in:
Adam Perry @ OpenAI
committed by
GitHub
GitHub
parent
859dbe2761
commit
ebb7980369
46
.bazelrc
46
.bazelrc
@@ -38,24 +38,50 @@ common:windows --test_env=WINDIR
|
||||
common --test_env=RUST_MIN_STACK=8388608 # 8 MiB
|
||||
|
||||
common --test_output=errors
|
||||
common --bes_results_url=https://app.buildbuddy.io/invocation/
|
||||
common --bes_backend=grpcs://remote.buildbuddy.io
|
||||
common --remote_cache=grpcs://remote.buildbuddy.io
|
||||
common --remote_download_toplevel
|
||||
common --nobuild_runfile_links
|
||||
# These settings tune BuildBuddy/RBE behavior but do not contact a remote
|
||||
# service unless a `buildbuddy-*` configuration below supplies an endpoint.
|
||||
common --remote_download_toplevel
|
||||
common --remote_timeout=3600
|
||||
common --noexperimental_throttle_remote_action_building
|
||||
common --experimental_remote_execution_keepalive
|
||||
common --grpc_keepalive_time=30s
|
||||
common --experimental_remote_downloader=grpcs://remote.buildbuddy.io
|
||||
|
||||
# Opt-in remote configurations selected by
|
||||
# `.github/scripts/run_bazel_with_buildbuddy.py`. Plain Bazel commands do not
|
||||
# contact BuildBuddy unless a user selects one of these configurations.
|
||||
# Use the generic host for cache, BES, and downloads without remote execution.
|
||||
common:buildbuddy-generic --bes_backend=grpcs://remote.buildbuddy.io
|
||||
common:buildbuddy-generic --bes_results_url=https://app.buildbuddy.io/invocation/
|
||||
common:buildbuddy-generic --remote_cache=grpcs://remote.buildbuddy.io
|
||||
common:buildbuddy-generic --experimental_remote_downloader=grpcs://remote.buildbuddy.io
|
||||
|
||||
# Add remote execution on the generic host.
|
||||
common:buildbuddy-generic-rbe --config=buildbuddy-generic
|
||||
common:buildbuddy-generic-rbe --config=remote
|
||||
common:buildbuddy-generic-rbe --remote_executor=grpcs://remote.buildbuddy.io
|
||||
|
||||
# Use the OpenAI tenant for cache, BES, and downloads without remote execution.
|
||||
common:buildbuddy-openai --bes_backend=grpcs://openai.buildbuddy.io
|
||||
common:buildbuddy-openai --bes_results_url=https://openai.buildbuddy.io/invocation/
|
||||
common:buildbuddy-openai --remote_cache=grpcs://openai.buildbuddy.io
|
||||
common:buildbuddy-openai --experimental_remote_downloader=grpcs://openai.buildbuddy.io
|
||||
|
||||
# Add remote execution on the OpenAI tenant.
|
||||
common:buildbuddy-openai-rbe --config=buildbuddy-openai
|
||||
common:buildbuddy-openai-rbe --config=remote
|
||||
common:buildbuddy-openai-rbe --remote_executor=grpcs://openai.buildbuddy.io
|
||||
|
||||
# This limits both in-flight executions and concurrent downloads. Even with high number
|
||||
# of jobs execution will still be limited by CPU cores, so this just pays a bit of
|
||||
# memory in exchange for higher download concurrency.
|
||||
common --jobs=30
|
||||
|
||||
# Shared remote execution policy. The endpoint-bearing `buildbuddy-*-rbe`
|
||||
# configurations include this group; CI configs override TestRunner below
|
||||
# when tests must remain local on their runner.
|
||||
common:remote --strategy=remote
|
||||
common:remote --extra_execution_platforms=//:rbe
|
||||
common:remote --remote_executor=grpcs://remote.buildbuddy.io
|
||||
common:remote --jobs=800
|
||||
# TODO(team): Evaluate if this actually helps, zbarsky is not sure, everything seems bottlenecked on `core` either way.
|
||||
# Enable pipelined compilation since we are not bound by local CPU count.
|
||||
@@ -146,15 +172,11 @@ common:ci-windows --repo_contents_cache=D:/a/.cache/bazel-repo-contents-cache
|
||||
# Linux crossbuilds don't work until we untangle the libc constraint mess.
|
||||
common:ci-linux --config=ci-bazel
|
||||
common:ci-linux --build_metadata=TAG_os=linux
|
||||
common:ci-linux --config=remote
|
||||
common:ci-linux --strategy=remote
|
||||
common:ci-linux --platforms=//:rbe
|
||||
|
||||
# On mac, we can run all the build actions remotely but test actions locally.
|
||||
common:ci-macos --config=ci-bazel
|
||||
common:ci-macos --build_metadata=TAG_os=macos
|
||||
common:ci-macos --config=remote
|
||||
common:ci-macos --strategy=remote
|
||||
common:ci-macos --strategy=TestRunner=darwin-sandbox,local
|
||||
|
||||
# On Windows, use Linux remote execution for build actions but keep test actions
|
||||
@@ -162,9 +184,7 @@ common:ci-macos --strategy=TestRunner=darwin-sandbox,local
|
||||
# still run against Windows binaries.
|
||||
common:ci-windows-cross --config=ci-windows
|
||||
common:ci-windows-cross --build_metadata=TAG_windows_cross_compile=true
|
||||
common:ci-windows-cross --config=remote
|
||||
common:ci-windows-cross --host_platform=//:rbe
|
||||
common:ci-windows-cross --strategy=remote
|
||||
common:ci-windows-cross --strategy=TestRunner=local
|
||||
common:ci-windows-cross --local_test_jobs=4
|
||||
common:ci-windows-cross --test_env=RUST_TEST_THREADS=1
|
||||
@@ -180,8 +200,6 @@ common:ci-windows-cross --extra_toolchains=//:windows_gnullvm_tests_on_msvc_host
|
||||
common:ci-v8 --config=ci
|
||||
common:ci-v8 --build_metadata=TAG_workflow=v8
|
||||
common:ci-v8 --build_metadata=TAG_os=linux
|
||||
common:ci-v8 --config=remote
|
||||
common:ci-v8 --strategy=remote
|
||||
|
||||
# Source-built Bazel V8 artifacts use the in-process sandbox by default. This
|
||||
# does not affect Cargo's default prebuilt rusty_v8 path.
|
||||
|
||||
Reference in New Issue
Block a user