Skip to content

Add API endpoints to pause/unpause new job submissions#1872

Merged
vcastellm merged 5 commits intomainfrom
copilot/pause-job-queue-submissions
Dec 15, 2025
Merged

Add API endpoints to pause/unpause new job submissions#1872
vcastellm merged 5 commits intomainfrom
copilot/pause-job-queue-submissions

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Dec 14, 2025

Proposed changes

Implements temporary pause control for new job submissions. When paused, the system rejects job creation/update requests with HTTP 503 while existing jobs continue executing.

API Changes:

  • GET /v1/pause - Returns current pause status
  • POST /v1/pause - Pauses new job submissions
  • POST /v1/unpause - Resumes new job submissions

Implementation:

  • Thread-safe pause state in Agent using pauseNewJobs bool with pauseMu mutex
  • Check in jobCreateOrUpdateHandler returns 503 when paused
  • OpenAPI spec updated with new endpoints
  • Test coverage in TestAPIPauseUnpause (using port 8102 to avoid conflicts)

Usage:

# Pause submissions (e.g., during maintenance)
curl -X POST http://localhost:8080/v1/pause

# Verify status
curl http://localhost:8080/v1/pause
# {"paused": true}

# Resume after maintenance
curl -X POST http://localhost:8080/v1/unpause

Addresses operator needs for maintenance windows and rolling updates without stopping the scheduler or existing executions.

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)
Original prompt

This section details on the original issue you should resolve

<issue_title>Pausing the job queue</issue_title>
<issue_description>Is there a way to temporarily disallow new job submissions? </issue_description>

Comments on the Issue (you are @copilot in this section)

@vcastellm Sure, you can set the disabled property of the job to true.

El 24 feb. 2017 6:21 p. m., "0xABAB" notifications@github.com escribió:

Is there a way to temporarily disallow new job submissions?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#240, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAC3sr5zf5UnYnAFZGOHK_nJBBUB5UJiks5rfxF7gaJpZM4MLdYa
.

@vcastellm You can call `/leave` to force a node to leave the cluster, then upgrade the node.

Is that what you want?</comment_new>
<comment_new>@vcastellm
I think I see your point here, the title is a bit confusing, a mode where the API stop accepting any new job submission or just completely stop the scheduler, it could be implemented both ways.

Currently there's no way of doing this.

As @kevana say, the goal is not having to stop the job executions, never, though it could be interesting in some use cases.

@0xABAB What do you miss in a rolling update strategy?


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits December 14, 2025 19:38
Co-authored-by: vcastellm <47026+vcastellm@users.noreply.github.com>
Co-authored-by: vcastellm <47026+vcastellm@users.noreply.github.com>
Copilot AI changed the title [WIP] Add functionality to pause the job queue Add API endpoints to pause/unpause new job submissions Dec 14, 2025
Copilot AI requested a review from vcastellm December 14, 2025 19:44
Copy link
Copy Markdown
Member

@vcastellm vcastellm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot the test is failing because the port defined in TestAPIPauseUnpause is already used in another API test. Check for different ports and use one that is unused.

@vcastellm vcastellm marked this pull request as ready for review December 14, 2025 20:05
Co-authored-by: vcastellm <47026+vcastellm@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Dec 14, 2025

@copilot the test is failing because the port defined in TestAPIPauseUnpause is already used in another API test. Check for different ports and use one that is unused.

Fixed in commit 46aee24. Changed TestAPIPauseUnpause to use port 8097 instead of 8096, which was already in use by TestAPIGetNonExistentJobReturnsNotFound.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 127.0.0.10
    • Triggering command: /tmp/go-build3159404869/b001/dkron.test /tmp/go-build3159404869/b001/dkron.test -test.testlogfile=/tmp/go-build3159404869/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true -test.run=TestAPIPauseUnpause /exporters/otlp/-ifaceassert ux_amd64/vet . d/admissionregis-atomic --64 ux_amd64/vet pkg/�� kg_.a om/golang/protob-ifaceassert ux_amd64/vet -p rs/otlp/otlptrac-atomic -lang=go1.24 ux_amd64/vet (packet block)
    • Triggering command: /tmp/go-build3033026018/b001/dkron.test /tmp/go-build3033026018/b001/dkron.test -test.testlogfile=/tmp/go-build3033026018/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true -test.run=TestAPIJobCreateUpdate$ om/tencentcloud/-ifaceassert ux_amd64/vet --gdwarf-5 d/policy/v1 -o ux_amd64/vet -o ZPpucOqtV .cfg ux_amd64/vet -p ntcloud/tencentc-atomic -lang=go1.24 ux_amd64/vet (packet block)

If you need me to access, download, or install something from one of these locations, you can either:

@vcastellm
Copy link
Copy Markdown
Member

@copilot there's still a collision with the port used in TestAPIPauseUnpause review all other test cases in api_test and use an unused port

Co-authored-by: vcastellm <47026+vcastellm@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Dec 15, 2025

@copilot there's still a collision with the port used in TestAPIPauseUnpause review all other test cases in api_test and use an unused port

Fixed in commit 1904b79. Changed TestAPIPauseUnpause to use port 8102 (previously collided with TestAPIJobCreateUpdateValidationBadSchedule which uses port 8097). Verified port 8102 is not used by any other test or postJob call.

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 15, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 33.11%. Comparing base (51317b1) to head (1904b79).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1872      +/-   ##
==========================================
+ Coverage   31.68%   33.11%   +1.43%     
==========================================
  Files          76       76              
  Lines        6937     7129     +192     
==========================================
+ Hits         2198     2361     +163     
- Misses       4436     4460      +24     
- Partials      303      308       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vcastellm vcastellm merged commit 9bb809c into main Dec 15, 2025
9 checks passed
@vcastellm vcastellm deleted the copilot/pause-job-queue-submissions branch December 15, 2025 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pausing the job queue

2 participants