AI Segmentation¶

Processing → AI Segmentation turns imagery into vector features using segment-geospatial (SamGeo) and Meta's SAM 3 model. Describe what you want — "trees", "buildings", "water" — or run automatic segmentation, and GeoLibre adds the resulting polygons as a new vector layer.

Requires the Python sidecar and a model backend

Segmentation runs deep-learning models, so it needs the GeoLibre desktop app with the Python sidecar plus a running samgeo-api model server. A CUDA GPU is strongly recommended; CPU inference works but is slow. See Setup below.

How it works¶

The heavy model stack (PyTorch + SAM 3) does not run inside the GeoLibre sidecar. Instead the sidecar exposes a thin /ml reverse-proxy in front of a separate samgeo-api server (the REST server shipped with segment-geospatial). GeoLibre uploads the image to the sidecar, which forwards it to samgeo-api, runs SAM 3, and returns GeoJSON polygons.

SegmentationDialog ──image+prompt──▶ sidecar /ml/segment/* ──▶ samgeo-api (SAM 3) ──▶ GeoJSON

This keeps the GeoLibre sidecar small and lets the model server run wherever you have a GPU.

Setup¶

Install segment-geospatial with the API and SAM 3 extras in a Python environment that has a working PyTorch build (ideally CUDA):

pip install "segment-geospatial[api,samgeo3]"

Then either let GeoLibre launch the model server for you, or run it yourself:

Auto-launch (default). If samgeo-api is on the PATH of the environment the sidecar runs in, GeoLibre starts it automatically on first use.

Run it yourself / on another machine. Start the server and point GeoLibre at it:

samgeo-api --port 8000
# then run the sidecar with:
GEOLIBRE_ML_SAMGEO_URL=http://127.0.0.1:8000

Desktop app: point it at an external samgeo-api

The desktop app runs its sidecar in a managed (uv) environment that ships the ml extra but not the heavy segment-geospatial model stack, so samgeo-api is not on its PATH and auto-launch does not apply. Install segment-geospatial[api,samgeo3] in a PyTorch-capable environment, start samgeo-api there, and launch the desktop app from a shell that exports GEOLIBRE_ML_SAMGEO_URL so the sidecar proxies to it:

# in your PyTorch env
samgeo-api --port 8000

# launch the desktop app with the proxy target set
GEOLIBRE_ML_SAMGEO_URL=http://127.0.0.1:8000 npm run tauri:dev

The Tauri process passes its environment to the sidecar it spawns. Without GEOLIBRE_ML_SAMGEO_URL, the sidecar has no model backend and /ml/status reports the segmentation backend as unavailable.

Install the sidecar's optional ml extra (just an HTTP client — the models live in samgeo-api):

pip install -e "backend/geolibre_server[ml]"

Configuration¶

Environment variable	Purpose
`GEOLIBRE_ML_SAMGEO_URL`	Base URL of an already-running `samgeo-api`. When set, the sidecar proxies here and does not launch a child process.
`GEOLIBRE_ML_SAMGEO_CMD`	Command used to launch `samgeo-api` on demand (default `samgeo-api`). `--host`/`--port` are appended automatically.
`GEOLIBRE_ML_DEFAULT_MODEL`	Model the UI defaults to (default `sam3`).

Using it¶

Open Processing → AI Segmentation.
If the backend isn't reachable, the dialog shows a message and a Start server button (desktop) — click it to launch the sidecar and model server.
Choose a GeoTIFF. Pick a georeferenced raster (.tif/.tiff; PNG/JPG also work but produce pixel-space results). The polygons come back in the raster's coordinate system, so they land in the right place on the map.
Pick a mode:
- Text prompt — type what to segment (e.g. trees, buildings, cars) and set a Confidence threshold (0–1; lower finds more, higher is stricter). SAM 3 finds every matching object in the image.
- Automatic (everything) — segments all distinct objects without a prompt.
Click Segment. The result is added as a new GeoJSON polygon layer named after the prompt, and the map zooms to it.

Notes & limitations¶

SAM 3 only. GeoLibre uses the SAM 3 backend, which covers text, box/point, and automatic prompts well. (SAM 2 is intentionally not wired up.)
Desktop only. Like the raster tools, segmentation needs the desktop app and the Python sidecar; it is not available in the browser-only build.
No objects found is a normal outcome — try a different prompt or lower the confidence threshold.
Box / point prompts drawn on the map are not in the UI yet; the /ml/segment/predict endpoint already supports them for scripted use.

API (advanced)¶

The sidecar exposes these endpoints (proxied to samgeo-api):

Method	Endpoint	Description
GET	`/ml/status`	Backend availability, default model, version, and model list.
POST	`/ml/segment/text`	Text-prompt segmentation (`prompt`, `confidence_threshold`).
POST	`/ml/segment/automatic`	Automatic mask generation.
POST	`/ml/segment/predict`	Box/point prompt segmentation (`boxes`, `point_coords`, `point_labels`, `point_crs`).

Each segment/* endpoint takes a multipart file (the image) plus model_version (default sam3) and output_format (default geojson) and returns a GeoJSON FeatureCollection.