# Images and video

Source: https://docs.temprl.org/guide/images-and-video

> Send screenshots, photos and video clips to Simo, refer to them in the state, and locate elements in pixels.

Pass images and videos in `media`, and refer to them in the state as `<image:1>`, `<image:2>`, `<video:1>`, … in order, each one once.

## Images

**Python**

```python
from simoai import Client, Noul, Choice, Extract, Locate, image

c = Client()
r = c.ask(
    state={"task": "Sign up for the newsletter", "screen": "<image:1>"},
    media=[image("screenshot.png")],
    questions={
        "done":   Noul("The sign-up is complete"),
        "next":   Choice("What should the agent do next?",
                         {"click": "click something", "scroll": "scroll down", "stop": "stop"},
                         args={"click": {"target": Extract("the label of the element to click")}}),
        "button": Locate("the Sign up button"),
    })

r["done"].noul           # 0.04
r["next"].choice         # "click"
r["next"].arg_values     # {"target": "Sign up"}
r["button"].center       # [912.0, 404.5]  <- pixels of YOUR screenshot.png, ready to click
```

**cURL**

```bash
IMG=$(base64 -w0 screenshot.png)
curl https://api.temprl.org/v1/ask \
  -H "Authorization: Bearer $SIMO_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{"model": "simo-1-pro",
 "state": {"task": "Sign up for the newsletter", "screen": "<image:1>"},
 "media": [{"type": "image", "data": "data:image/png;base64,$IMG"}],
 "questions": {
   "done": {"type": "noul", "instructions": "The sign-up is complete"},
   "button": {"type": "extract", "kind": "coordinates", "instructions": "the Sign up button",
              "image": 1, "scale": "rel1000", "max_tokens": 64}}}
EOF
```

**JavaScript**

```javascript
import { readFile } from "node:fs/promises";

const png = (await readFile("screenshot.png")).toString("base64");
const res = await fetch("https://api.temprl.org/v1/ask", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.SIMO_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "simo-1-pro",
    state: { task: "Sign up for the newsletter", screen: "<image:1>" },
    media: [{ type: "image", data: `data:image/png;base64,${png}` }],
    questions: {
      done: { type: "noul", instructions: "The sign-up is complete" },
      button: { type: "extract", kind: "coordinates", instructions: "the Sign up button", image: 1, scale: "rel1000", max_tokens: 64 },
    },
  }),
});
const { answers } = await res.json();
console.log(answers.button.center); // [x, y] in the pixels of the image sent
```

`image()` accepts a file path, an `http(s)` URL (downloaded by the SDK), a `data:` URI, raw `bytes` or a PIL image:

```python
image("photo.jpg")
image("https://example.com/parcel.png")
image(open("scan.png", "rb").read())
image(pil_image)
```

### Several images

```python
r = c.ask(state="Before: <image:1>\nAfter: <image:2>",
          media=[image("before.png"), image("after.png")],
          questions={"changed": Noul("Something changed between the two pictures"),
                     "where":   Locate("what changed", image=2)})   # locate in the second picture
```

### Good to know

- **Size.** Send images at their natural size. `image()` pads them on white to a shape the model reads without resampling, so small text stays legible. Locate answers are mapped back to your original pixels automatically.
- **Not found.** `Locate` returns `coordinates=None` and `center=None` when the thing is not visible; `image_size` is still set. Check `center is not None` before clicking.
- **Placeholder order.** The SDK warns when placeholders don't match `media` (wrong order, missing or extra), and the API rejects such requests with `InvalidRequest`.
- **Limits.** At most 16 pictures per request, video frames included, and 25 MB per request body. Images are sent as PNG, so a few large screenshots add up. The SDK checks the 25 MB limit before uploading. See [Limits](https://docs.temprl.org/models/limits).

## Video

```python
from simoai import video, video_file

# From a file: sampled at 1 frame per second, at most 32 frames (needs simoai[video])
clip = video_file("entrance.mp4", fps=1.0)

# Or from frames you already have, with the rate they were sampled at
clip = video([frame1, frame2, frame3], fps=2.0)

r = c.ask(state={"camera": "Warehouse door", "clip": "<video:1>"}, media=[clip],
          questions={"entered": Noul("A person enters the restricted area"),
                     "count":   Fill("how many people appear", kind="int")})
```

The model sees the frames with their timestamps. The service keeps at most 16 frames per video, spread evenly. Ask "did it happen?", not "at which exact frame?".

Over HTTP, a video is `{"type": "video", "frames": ["data:image/png;base64,...", ...], "fps": 1.0}`.