Introducing Nitric Batch

Today, Nitric has just released Nitric Batch into preview.

Batch is a new way for data teams and developers to run high-performance workloads, such as hosting custom AI models, in serverless environments without the complexity of upfront infrastructure automation.

This initial release of Nitric Batch supports Python, Go, Nodejs and Dart, and deploys to AWS Batch, Google Cloud Batch, or any other provider through custom plugins, with Azure Batch coming soon.

Nitric Batch lets you deploy jobs that require substantial compute resources or GPU access without writing complex deployment automation. Whether you're running AI models, machine learning jobs, large-scale data processing, or handling video and image processing, Nitric Batch decouples your application from the headache of infrastructure management or any specific cloud provider.

from nitric.resources import job

hello_batch = job("generate-image")

@hello_batch(cpus=4, memory=16384, gpus=1)
async def do_something_big(data):
  print(f"Hello from Nitric Batch, here's the {data}")

Why Nitric Batch?

Deploying high-performance compute workloads at scale can be challenging. Serverless compute platforms promise to make this easier, but in there’s still a lot of glue code required to get jobs up and running reliably. This is particularly true when interacting with additional cloud services, such as queues, databases, topics, and other compute environments.

Nitric Batch changes that. With a clean, developer-first interface, you focus on the workloads, not the plumbing. Whether it's a massive dataset that needs processing or an AI model that requires GPUs to crunch through complex tasks, Nitric Batch handles the scaling, resource provisioning, and job scheduling behind the scenes. The best part? You’re able to run these workloads on your own infrastructure with any host you choose.

Access Other Nitric Resources

Batch Services, just like other Services built with Nitric, have the ability to deploy and access any other Nitric Resource, allowing you to trigger batch jobs from APIs, Schedules, access Databases, Key/Value Stores and Buckets for persistence, or work with Queues and Topics for distributed workloads.

What You Can Do with Nitric Batch

• Run AI/ML models without worrying about setting up GPU instances.
• Process terabytes of data in the cloud with minimal effort.
• Handle intensive video or image processing workloads.
• Run complex simulations or models without dealing with infrastructure overhead.

Nitric Batch lets you tap into the power of the major cloud providers' serverless compute platforms with just a few lines of code, to run big data or big compute workloads such as AI model inferencing and training. This allows applications to put custom ML/AI services into the heart of an application, whilst retaining the benefits of serverless.

Here's a basic example that creates an API for image gen with stabilityai/stable-diffusion-2-1, which stores the resulting images in a cloud bucket.

from nitric.resources import job, bucket
from nitric.resources.job import JobContext
from diffusers import DiffusionPipeline
from nitric.application import Nitric
from io import BytesIO

# create the bucket and the job
images = bucket("images").allow("read", "write")
image_generation = job("image-generation")


# register the job handler function, with resource requirements
@image_generation(cpus=4, memory=12288, gpus=1)
async def generate_sd_image(ctx: JobContext):
  prompt: str = ctx.req.data["prompt"]
  name: str = ctx.req.data["name"]

  pipeline = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1")
  if torch.cuda.is_available():
    # if cuda is available then use cuda
    print("running with cuda")
    pipeline.to("cuda")
  else:
    # otherwise we're relegated to the lowly CPU
    print("running with cpu")
    pipeline.to("cpu")

  image = pipeline(prompt).images[0]
  buffer = BytesIO()
  image.save(buffer, format="PNG")

  await images.file(f"{name}.png").write(buffer.getvalue())


Nitric.run()

In a standard Nitric service we can create an API and register the /generate route. This route accepts prompts and submits them as tasks to the image generation batch service above.

from nitric.resources import api, job
from nitric.application import Nitric
from nitric.context import HttpContext

# create the API and reference the job
main = api("main")
image_generation = job("image-generation").allow("submit")


@main.post("/generate")
async def generate_image(ctx: HttpContext):
  body = ctx.req.json
  if body is None:
    ctx.res.status = 400
    return

  prompt = body["prompt"]
  name = body["name"]

  # submit tasks to the batch service
  await image_generation.submit({
    "prompt": prompt,
    "name": name
  })

Nitric.run()

You can run this service locally with the nitric start command. Then submit prompts via the nitric local dashboard or any client.

curl -X POST http://localhost:4002/generate \
-H "Content-Type: application/json" \
-d '{ "prompt": "a drawing of a cat", "name": "cat-image" }'

When you're ready to push the code to the cloud, just create a stack and deploy it.

# create the stack - choosing AWS or GCP
nitric stack new
# then deploy
nitric up

No Overhead, No Manual Scaling

What makes this different from rolling your own solution? It is your own solution, Nitric is open source and deploys to your infrastructure, it's just a whole lot easier. You don’t need to manually provision infrastructure or write scripts to handle job scheduling, retries, or scaling. Nitric Batch does all of this for you, dynamically allocating compute resources based on your workload's needs. This means no more over-provisioning and paying for idle resources.

We're putting Nitric Batch into preview, and we're eager to see what you build with it. Whether you're working with ML, big data, or just need raw compute power without the hassle, give it a try and let us know what you think.

As always, feedback is welcome. We're just getting started with this, and your input will help shape the future direction of Nitric Batch.