Implementing expanded features

Oct 13, 2025 by talynt.org 66 views

Streamline Job Management for Long Running Tasks: A Comprehensive Solution

Managing long running tasks, such as evaluation, synthetic data generation, and post-training, can be a complex and time-consuming process. The existing APIs for these tasks have inconsistent behavior, making it difficult for users to work with them efficiently. In this article, we will discuss the need for a streamlined job management system and propose a comprehensive solution to address this issue.

The existing APIs for long running tasks have several limitations that make them difficult to work with. For example:

Synthetic data generation does not return job IDs at all.
Training does return job IDs, but currently locks the server to run the job, which makes API time out.
Eval is the same as training, plus uses a field name for job ID that is different from post-training (job_id vs job_uuid).

These inconsistencies make it difficult for users to work with these APIs efficiently. A common API is needed to provide consistency and make it easier for users to manage long running tasks.

The proposed design for the new job management system includes the following features:

Canary Integration for Post-Training: The first step in implementing the new job management system is to integrate the post-training scheduler with the torchtune provider. This will involve:
- Land "naive" scheduler implementation integrated with torchtune provider
- Move the scheduler module to common space for reuse
- Fix torchtune post-training integration tests
- Enable post-training integration tests in CI
- Implement job cancel verb for post-training jobs
Other APIs' Consistency: The next step is to ensure that all APIs for long running tasks have consistent behavior. This will involve:
- Adjusting the eval job API to use a consistent field name (job_id in eval vs job_uuid in post-training)
- Switching the synthetic data generation API to jobs pattern
- Not requiring passing job ID when creating a job (generate it for the user and return)
Expanded Features: The new job management system will also include several expanded features, such as:
- Exposing collected logs as a job artifact
- Implementing persistence for jobs state (server may restart and recover knowledge about jobs)
- Implementing logs streaming for jobs
- Expanding possible jobs states to reflect expanded needs (e.g. paused, resumed)
- Implementing resuming of jobs after server crash
- Considering using FastAPI BackgroundTasks for scheduler needs (a new backend?)
- Implementing jobs pause (may need addressing of hardware management story first!)
New /jobs API: The new job management system will also include a new /jobs API, which will provide a consistent way to manage long running tasks. This will involve:
- Introducing new /jobs API definition (bare spec)
- Plugging provider schedulers into /jobs API (iterate over schedulers to collect all jobs)
- Implementing /jobs DELETE API (clean up artifacts, logs etc.)
- Enabling filtering by job type for /jobs API
- Modeling dependencies between jobs in API
- Implementing job dependencies in scheduler module
- Retiring old job management APIs from post-training, eval, sdg etc. (get_artifacts, get_status, cancel etc.)

The proposed design for the new job management system has several benefits, including:

Consistency: The new system will provide a consistent way to manage long running tasks, making it easier for users to work with them.
Efficiency: The new system will reduce the complexity of managing long running tasks, making it more efficient for users.
Scalability: The new system will be designed to scale with the needs of the users, making it more flexible and adaptable.

In conclusion, the proposed design for the new job management system is a comprehensive solution that addresses the limitations of the existing APIs for long running tasks. The new system will provide a consistent way to manage long running tasks, making it easier for users to work with them. The benefits of the proposed design include consistency, efficiency, and scalability, making it a valuable addition to the existing system.

The next steps in implementing the new job management system will involve:

Implementing the canary integration for post-training: This will involve land "naive" scheduler implementation integrated with torchtune provider, move the scheduler module to common space for reuse, fix torchtune post-training integration tests, enable post-training integration tests in CI, and implement job cancel verb for post-training jobs.
Ensuring other APIs' consistency: This will involve adjusting the eval job API to use a consistent field name, switching the synthetic data generation API to jobs pattern, and not requiring passing job ID when creating a job.
Implementing expanded features: This will involve exposing collected logs as a job artifact, implementing persistence for jobs state, implementing logs streaming for jobs, expanding possible jobs states to reflect expanded needs, implementing resuming of jobs after server crash, considering using FastAPI BackgroundTasks for scheduler needs, and implementing jobs pause.
Implementing the new /jobs API: This will involve introducing new /jobs API definition, plugging provider schedulers into /jobs API, implementing /jobs DELETE API, enabling filtering by job type for /jobs API, modeling dependencies between jobs in API, implementing job dependencies in scheduler module, and retiring old job management APIs from post-training, eval, sdg etc.

By following these steps, we can implement a comprehensive job management system that provides a consistent way to manage long running tasks, making it easier for users to work with them.
Streamline Job Management for Long Running Tasks: A Comprehensive Solution

Q: What is the current state of job management for long running tasks? A: The current state of job management for long running tasks is inconsistent and complex. Different APIs have different behaviors, making it difficult for users to work with them efficiently.

Q: What are the main limitations of the current job management system? A: The main limitations of the current job management system are:

Inconsistent behavior across different APIs
Lack of consistency in field names and API calls
Difficulty in managing long running tasks due to complex API interactions

Q: What is the proposed solution to address these limitations? A: The proposed solution is a comprehensive job management system that provides a consistent way to manage long running tasks. This system will include:

A new /jobs API that provides a consistent way to manage long running tasks
A canary integration for post-training that integrates the post-training scheduler with the torchtune provider
Expanded features such as exposing collected logs as a job artifact, implementing persistence for jobs state, and implementing logs streaming for jobs

Q: What are the benefits of the proposed solution? A: The benefits of the proposed solution include:

Consistency: The new system will provide a consistent way to manage long running tasks, making it easier for users to work with them.
Efficiency: The new system will reduce the complexity of managing long running tasks, making it more efficient for users.
Scalability: The new system will be designed to scale with the needs of the users, making it more flexible and adaptable.

Q: What are the next steps in implementing the proposed solution? A: The next steps in implementing the proposed solution will involve:

Implementing the canary integration for post-training
Ensuring other APIs' consistency
Implementing expanded features
Implementing the new /jobs API

Q: What is the timeline for implementing the proposed solution? A: The timeline for implementing the proposed solution will depend on the complexity of the tasks and the resources available. However, we anticipate that the implementation will take several months to complete.

Q: How will the proposed solution be tested and validated? A: The proposed solution will be tested and validated through a combination of unit tests, integration tests, and user acceptance testing. We will also conduct regular reviews and retrospectives to ensure that the solution is meeting the needs of the users.

Q: What is the expected outcome of the proposed solution? A: The expected outcome of the proposed solution is a comprehensive job management system that provides a consistent way to manage long running tasks. This system will make it easier for users to work with long running tasks, reduce the complexity of managing these tasks, and provide a scalable solution that meets the needs of the users.

Q: What are the potential risks and challenges associated with the proposed solution? A: The potential risks and challenges associated with the proposed solution include:

Complexity of implementing the new /jobs API
Difficulty in ensuring consistency across different APIs
Potential impact on existing workflows and processes
Need for additional resources and support to implement and maintain the new system

Q: How will the proposed solution be maintained and updated? A: The proposed solution will be maintained and updated through regular reviews and retrospectives, as well as through the use of agile development methodologies. We will also conduct regular testing and validation to ensure that the solution is meeting the needs of the users.