simodocs
temprl.org Console

Asking questions

Images and video

Pass images and videos in media, and refer to them in the state as <image:1>, <image:2>, <video:1>, … in order, each one once.

Images

from simoai import Client, Noul, Choice, Extract, Locate, image

c = Client()
r = c.ask(
    state={"task": "Sign up for the newsletter", "screen": "<image:1>"},
    media=[image("screenshot.png")],
    questions={
        "done":   Noul("The sign-up is complete"),
        "next":   Choice("What should the agent do next?",
                         {"click": "click something", "scroll": "scroll down", "stop": "stop"},
                         args={"click": {"target": Extract("the label of the element to click")}}),
        "button": Locate("the Sign up button"),
    })

r["done"].noul           # 0.04
r["next"].choice         # "click"
r["next"].arg_values     # {"target": "Sign up"}
r["button"].center       # [912.0, 404.5]  <- pixels of YOUR screenshot.png, ready to click

image() accepts a file path, an http(s) URL (downloaded by the SDK), a data: URI, raw bytes or a PIL image:

Python
image("photo.jpg")
image("https://example.com/parcel.png")
image(open("scan.png", "rb").read())
image(pil_image)

Several images

Python
r = c.ask(state="Before: <image:1>\nAfter: <image:2>",
          media=[image("before.png"), image("after.png")],
          questions={"changed": Noul("Something changed between the two pictures"),
                     "where":   Locate("what changed", image=2)})   # locate in the second picture

Good to know

  • Size. Send images at their natural size. image() pads them on white to a shape the model reads without resampling, so small text stays legible. Locate answers are mapped back to your original pixels automatically.
  • Not found. Locate returns coordinates=None and center=None when the thing is not visible; image_size is still set. Check center is not None before clicking.
  • Placeholder order. The SDK warns when placeholders don't match media (wrong order, missing or extra), and the API rejects such requests with InvalidRequest.
  • Limits. At most 16 pictures per request, video frames included, and 25 MB per request body. Images are sent as PNG, so a few large screenshots add up. The SDK checks the 25 MB limit before uploading. See Limits.

Video

Python
from simoai import video, video_file

# From a file: sampled at 1 frame per second, at most 32 frames (needs simoai[video])
clip = video_file("entrance.mp4", fps=1.0)

# Or from frames you already have, with the rate they were sampled at
clip = video([frame1, frame2, frame3], fps=2.0)

r = c.ask(state={"camera": "Warehouse door", "clip": "<video:1>"}, media=[clip],
          questions={"entered": Noul("A person enters the restricted area"),
                     "count":   Fill("how many people appear", kind="int")})

The model sees the frames with their timestamps. The service keeps at most 16 frames per video, spread evenly. Ask "did it happen?", not "at which exact frame?".

Over HTTP, a video is {"type": "video", "frames": ["data:image/png;base64,...", ...], "fps": 1.0}.