AI_Agro_Support/README.md

# AI Agro Support — Daily Pipeline

Daily agronomic advisory pipeline. For every field growing a configured crop, it
builds an 11-day weather / phenology / disease-forecast JSON, asks an LLM to
synthesise one or two vector-search queries, prefilters chemical products in SQL
Server, and retrieves up to six matching products from a local Weaviate
`ProductProfile` collection. It then enriches that JSON with full product label
information and the previous advisory, asks a second LLM to write the
farmer-facing advice, and stores the result in the `advice` table.

The pipeline is **crop-first**: `config.yaml` declares which crops and diseases
to check, and each morning's `batch` run discovers every field that grows one of
those crops and has a matching disease-forecast model configured, then runs the
full sequence above for each `(field, disease)` pair — in parallel, with one
field's failure never blocking another's.

## Prerequisites

- Python 3.12+
- Microsoft SQL Server (`DBMeteo`) reachable with ODBC Driver 17 (or 18)
- Weaviate running locally (see `docker-compose.yml`) with a populated `ProductProfile` collection
- API key for the chosen LLM provider (Gemini by default)

## Setup

```powershell
cd d:\Uni\Thesis\AI-Agro-Support
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
copy .env.example .env
# Edit .env with SQL, Weaviate, and LLM credentials
```

Start Weaviate if it is not already running:

```powershell
docker compose up -d
```

## Configuration

- [`config.yaml`](config.yaml) — the `crops:` worklist, concurrency/rate limits, retry policy,
  the 09:00 SLA deadline, LLM provider/model, and vocab paths
- [`.env`](.env) — SQL Server, Weaviate, and API keys
- [`vocab/crops.yaml`](vocab/crops.yaml) / [`vocab/diseases.yaml`](vocab/diseases.yaml) — Italian → English maps
- [`prompts/`](prompts) — the prompt registry (see below)

### `crops:` worklist

```yaml
crops:
  - crop: grapevine              # canonical English, matched via vocab/crops.yaml
    diseases:
      - model_name: PERONOSPORA  # matched against AI_agrosupport_agro_models.anmod_model
        disease: downy mildew    # canonical English, used for products/advice/prompts
```

Every morning, `batch` maps each `crop` to the fields growing it
(`AI_agrosupport_an_colture` → `AI_agrosupport_cmp_layers`), joins to
`AI_agrosupport_agro_models` for each listed `model_name`, and runs one job per
matching `(field, disease)` pair. To add a crop or disease: add an entry here, add
the Italian/English mapping to `vocab/crops.yaml` / `vocab/diseases.yaml` if it
doesn't already exist, and add a prompt directory (see below) — otherwise the
pair silently uses the generic fallback prompt.

### `worklist:`, `concurrency:`, `retry:`, `schedule:`

```yaml
worklist:
  require_enabled_model: true   # AI_agrosupport_agro_models.anmod_enabled must be 1
  field_allowlist: []           # restrict to specific field IDs while testing

concurrency:
  workers: 8                    # ThreadPoolExecutor size
  limits: { sql: 6, gemini: 4 } # see "Concurrency model" below
  gemini_requests_per_minute: 60

retry:
  attempts: 3
  initial_backoff_seconds: 2
  max_backoff_seconds: 30

schedule:
  deadline: "09:00"             # jobs not started by this local time are skipped
```

### Prompt registry (`prompts/`)

Each crop-disease pair needs its own advice prompt (its phytopathological
model's phase semantics, phenology notes, etc.). Prompts are plain Markdown
files resolved by `pipeline/prompts.py::PromptRegistry`, with a shared fallback
so a newly configured pair never fails outright — it just logs a warning and
uses a generic prompt until a dedicated one is written.

```
prompts/
  _output_format.md                          shared JSON contract, appended to every advice system prompt
  query_synthesis/_default.md                generic query-synthesis system prompt
  query_synthesis/<crop>__<disease>.md        optional per-pair override
  advice/_default/{system.md,user.md}         fallback advice prompts
  advice/<crop>__<disease>/{system.md,user.md} per-pair advice prompts
```

The `<crop>__<disease>` slug is built from the same canonical English names used
in `config.yaml`'s `crops:` block (lower-cased, non-alphanumeric runs collapsed
to `_`) — e.g. `grapevine` + `downy mildew` → `grapevine__downy_mildew`. To add a
prompt for a new pair, create `prompts/advice/<slug>/system.md` and `user.md`;
`batch` logs every configured pair still falling back to `_default` at startup.

## Run

```powershell
.\.venv\Scripts\Activate.ps1

# Daily entry point: every crop/disease pair in config.yaml, across all matching fields
python -m pipeline batch

# Historical / test dates (useful when the live window has no FASE2/FASE5/INCUBAZPRIMARIA/observation)
python -m pipeline batch --as-of 2026-06-29

# Narrow to one crop, disease, and/or field while testing
python -m pipeline batch --crop grapevine --disease "downy mildew" --field-id 4085

# Re-run a partially failed morning without redoing already-written advice
python -m pipeline batch --skip-existing-advice

# Build every job's JSON but skip LLM calls, vector search, and the DB write
python -m pipeline batch --dry-run

# Single-field debugging (keeps the pre-scaling CLI behaviour)
python -m pipeline one --field-id 4085 --disease PERONOSPORA --dry-run
```

`batch` accepts `--deadline HH:MM` and `--workers N` to override
`schedule.deadline` / `concurrency.workers` from `config.yaml` for one run.

## Concurrency model

One `ThreadPoolExecutor` (`concurrency.workers`) runs every discovered job.
Because dozens to a couple hundred jobs can run in the same morning, several
things that used to be "open it, use it, close it" per field are now shared
across the whole run (see `pipeline/resources.py`):

- **SQL**: each worker thread gets its own lazily-created `pyodbc` connection
  (connections cannot be shared across threads), kept open for the batch's
  lifetime. `concurrency.limits.sql` bounds how many jobs may run their
  SQL-heavy phase (forecast/weather/phenology/treatments lookup, or the
  product-JSON/advice-insert phase) at the same instant, independent of how
  many connections are open.
- **Weaviate**: one client is opened for the whole batch instead of one per
  field.
- **Gemini**: `near_text` search embeds through the same `GEMINI_API_KEY` as
  both LLM calls (`ProductProfile` is vectorised with text2vec-palm /
  gemini-embedding-001), so `concurrency.limits.gemini` and
  `gemini_requests_per_minute` are a single shared budget covering query
  synthesis, advice generation, *and* vector search — not two independent
  ones.
- **Products**: the whole `products` table is loaded into memory once per
  batch (`pipeline/stages/products.py::ProductIndex`) instead of re-scanned
  and re-parsed once per field, and `ProductJsonBuilder` is shared per
  `(crop, disease)` pair instead of per field.

Each of the three external boundaries — LLM, Weaviate, SQL — has its own
`tenacity` retry policy in `pipeline/retry.py` (exponential backoff with
jitter, `retry.attempts` tries), applied automatically inside
`Resources.call_llm` / `Resources.search_products` and around the
`insert_advice` transaction.

## Output

Each run writes to `output/<YYYY-MM-DD>/`:

- `field<field_id>__<crop>__<disease slug>.json` — one file per job (see fields below)
- `_run_report.json` — the operational artifact for the morning: totals by
  status plus a per-job breakdown (`status`, `duration_s`,
  `recommended_count`, `allowed_products_empty`, `message`, and, on failure,
  `error_type` / `error_message`)

Per-job payload fields:

| Field | Description |
|---|---|
| `run` | Metadata (as_of, field, crop, disease, organic, station, …) |
| `json_for_advice_generation` | Full 11-day meteorological + forecast payload, plus `date_of_today`, `allowed_products`, `applied_treatment` and `last_advice`; this is what the advice LLM receives |
| `json_for_query_synthesis` | Subset of the above, sent to the LLM for search query generation |
| `generated_queries` | One or two natural-language search queries |
| `candidate_count` | Products remaining after the prefilter |
| `recommended_products` | Up to 6 items with `product_name` + `registration_number` |
| `last_advice` | `date` (previous advisory's issuance date) + `advice_summary` + `suggested_products` of the most recent previous advisory. `advice_summary` is machine context for the next run: it is stored without relative day words, attention prefixes, or follow-the-label closers |
| `advice` | Parsed LLM output: `full_advice`, `advice_summary`, `apply_treatment`, `treatments`, `dosage`, `apply_date` |
| `sent_information` | The advice payload with every embedded product label collapsed back to a product name; mirrors `advice.sent_information` |
| `advice_inserted` | Whether the `advice` row was written |
| `status` | `ok`, `no_risk`, or `dry_run` (see `_run_report.json` for `failed` / `skipped_*`, which have no per-job file) |
| `message` | Human-readable summary of the run outcome |

`json_for_advice_generation.date_of_today` is oggi for this advisory (the run's
`as_of`, formatted `DD-MM-YYYY`). `last_advice.date` is the previous advisory's
issuance date, in the same format.

If a field's 11-day window has no `FASE2`, `FASE5`, or `INCUBAZPRIMARIA` day and
no true `observation` day (`as_of-5 .. as_of`), the product search is skipped
and `allowed_products` is empty, but advice is still generated and stored; that
job is labelled `no_risk`. The run report also counts `ok` jobs whose
`allowed_products` ended up empty anyway (for example an organic field with no
organic-certified product in the catalog yet) — watch that number as the
`products` table fills in, since a sudden jump usually means the prefilter or
the Weaviate allowlist filter is misbehaving rather than that the catalog is
genuinely empty for that combination.

The process exit code is `0` only if every job in the report is `ok`, `no_risk`,
`dry_run`, or a deliberate `skipped_*`; it is `1` if any job's status is `failed`.

## Database writes

Each non-dry-run job inserts one row into `advice` for `(date, field_id,
disease)`. That triple is the table's primary key, so any existing row for
the same triple is deleted first — re-running a day replaces its advisory
instead of failing on a duplicate key. The delete-then-insert pair runs
inside one transaction (`pipeline/db.py::transaction`) so a crash between
the two statements cannot leave a field's advisory missing for the day, and
deadlocks are retried per `retry:` in config.yaml.

The primary key's clustered index on `([date], field_id, disease)` also
lets that DELETE seek straight to the rows it needs instead of scanning the
table, so concurrent workers don't block each other on unrelated rows.

## Daily scheduling

Provide a CLI entry point only. Schedule it yourself on the deployment machine.
Advice must be ready by the `schedule.deadline` (09:00 by default), and weather
data should stay fresh, so start the run no earlier than ~07:00 — for example
via Windows Task Scheduler, daily at 07:00:

```
d:\Uni\Thesis\AI-Agro-Support\.venv\Scripts\python.exe -m pipeline batch --skip-existing-advice
```

with working directory `d:\Uni\Thesis\AI-Agro-Support`. `--skip-existing-advice`
makes a re-trigger (e.g. a scheduled retry later that morning) resumable at no
extra cost: it does one bulk check against `advice` up front and only runs jobs
that don't have a row yet.

## Observability (optional)

The pipeline can send OpenTelemetry/OpenInference traces to a self-hosted
[Arize Phoenix](https://github.com/Arize-ai/phoenix) instance for debugging a
bad advisory, comparing prompts/models later, and seeing per-job token usage
and USD cost for the two LLM stages. It is **off by default**, **fail-open**
(a down or missing Phoenix never fails `batch` or `one`), and **not** a
runtime dependency: nothing about the daily 07:00–09:00 run depends on it.

### Start Phoenix

```powershell
docker compose up -d
```

This brings up Weaviate and Phoenix together. Phoenix serves its UI (and the
OTLP/HTTP trace collector) at <http://localhost:6006>. SQLite on the
`phoenix_data` Docker volume is enough for this single-machine setup — see
"Why SQLite / why not Langfuse" below.

### Turn tracing on

Tracing is controlled by `observability:` in `config.yaml`:

```yaml
observability:
  enabled: false                                  # default: off
  project_name: ai-agro-support
  endpoint: http://localhost:6006/v1/traces
  hide_prompts: true                               # keep prompt/payload text out of spans
  max_attribute_chars: 4096                        # cap every exported string attribute
```

Any of these can be overridden per-environment without editing the file:

| Env var | Overrides |
|---|---|
| `PHOENIX_TRACING_ENABLED` | `observability.enabled` |
| `PHOENIX_COLLECTOR_ENDPOINT` | `observability.endpoint` |
| `PHOENIX_PROJECT_NAME` | `observability.project_name` |

With tracing on, `python -m pipeline one` or `batch` registers the tracer
once at startup (`pipeline/observability.py::setup_tracing`, called from
`pipeline/__main__.py` right after `load_settings()`), then instruments
`google-genai` (and `openai` / `anthropic`, if those `llm:` profiles are ever
enabled) via
[OpenInference](https://github.com/Arize-ai/openinference) auto-instrumentors.
Spans are batched and exported on a background thread, so a slow or down
collector cannot block a worker or push the run past `schedule.deadline`.

### What you get per job

Every `(field, crop, disease, as_of)` job produces one trace, rooted at a
`run_job` span, with:

- a `query_synthesis` span (gemini-2.5-flash by default) and an
  `advice_generation` span (gemini-2.5-pro), each wrapping the provider's own
  auto-instrumented LLM span with `llm.model_name`, `llm.provider`, and
  token counts (`llm.token_count.prompt` / `completion` / `total`, plus
  `completion_details.reasoning` for Gemini 2.5 thinking tokens);
- a `search_products` retriever span with the synthesized queries and the
  retrieved product IDs/distances (never product names or label content);
- `dry_run` jobs get only the `run_job` span — no LLM or retrieval children,
  since `--dry-run` skips those calls entirely.

Phoenix computes cost per span from its built-in model pricing table, which
already includes `gemini-2.5-flash` and `gemini-2.5-pro` — no custom entry
under **Settings → Models** is needed for either model as configured today.
If a model is ever renamed or swapped to one Phoenix doesn't recognise, add
it there (regex `Name Pattern`, provider `google`, per-1M-token prices).

Trace and span inputs are hidden by default (`observability.hide_prompts`):
the full `json_for_advice_generation` payload (weather, phenology, product
labels) never leaves the process as span data. Outputs — the generated
advice text — stay visible, since that is what you actually want to read
when debugging a bad advisory; `output/<date>/*.json` and
`_run_report.json` remain the authoritative operational artifacts, not
Phoenix.

### Known gap: embedding cost

`search_products`'s Weaviate `near_text` calls are embedded **inside the
Weaviate container** (`text2vec-google`), using the same `GEMINI_API_KEY` as
the two LLM calls, but that request never passes through this process's
Python `google-genai` client — so no OpenInference instrumentor can see it.
Phoenix's per-job cost therefore covers the two LLM calls only and slightly
undercounts total Gemini spend (1–2 `gemini-embedding-001` calls per job, a
small fraction of a cent at this volume). The `search_products` span still
records `query_count`, so this gap can be estimated later if it matters; it
is not worth a custom cost pipeline at this scale.

### Why SQLite / why not Langfuse or Phoenix Cloud

- **SQLite, not Postgres:** this is a single-machine, single-writer
  deployment; Phoenix officially supports SQLite on a mounted volume for
  exactly this case. Postgres would add a second container and volume for no
  benefit here (switch later via `PHOENIX_SQL_DATABASE_URL` if that changes).
- **Self-hosted, not Phoenix Cloud:** farm/field data stays in the
  deployment by design — nothing here talks to a hosted endpoint.
- **Phoenix, not Langfuse:** Langfuse's self-hosted stack needs Postgres
  *and* ClickHouse; Phoenix needs one container and ingests plain
  OTLP + OpenInference, so this pipeline isn't locked into either vendor.

## Pipeline stages

Run once per `(field, disease)` job by `pipeline/job.py::run_job`:

1. **Worklist** — crop → fields (`AI_agrosupport_an_colture` + `AI_agrosupport_cmp_layers`) → disease model (`AI_agrosupport_agro_models`); organic flag is `cmplay_imp == 3` (`pipeline/worklist.py`, run once per batch, not per job)
2. **Disease model** — per-day incubation > `FASE5` > `FASE2` resolution over as_of±5, from the worklist's known `anmod_id`; a `model_description` containing `INCUBAZPRIMARIA` is stored as `INCUBAZPRIMARIA_<model_value>` (e.g. `INCUBAZPRIMARIA_40.3`) from `AI_agrosupport_agro_model_tmp2.model_value`
3. **Observation** — `AI_agrosupport_agro_ril_pathogen` (`rilpato_layer`/`rilpato_date`/`rilpato_diffusion`); `observation` is true for `as_of` or any of the past 5 days if a row exists there with `rilpato_diffusion` other than `32` (including `NULL`)
4. **Weather** — `TDatiMeteo_D` for past days, `TDatiMeteo_D_FRC` for today/future
5. **Phenology** — carry-forward of latest observation ≤ day; future days null; if today's phase is missing, use yesterday's
6. **Applied treatments** — products sprayed over `as_of-5 .. as_of`, from `AI_agrosupport_agro_ril_operations*` with `rilop_operation = 10`
7. **LLM** — synthesise 1–2 anonymised search queries from the subset
8. **Product prefilter** — organic / crop / disease gates, then one prefilter rule against the batch-wide, in-memory `products` index (`pipeline/stages/disease.py::select_prefilter_rule`, first match wins):

   | Rule | Predicates |
   |---|---|
   | `observation` true on `as_of` or the past 5 days | `systemicity` in `{systemic, mixed}` and `eradicant_action = 1` |
   | `INCUBAZPRIMARIA` / `INCUBAZPRIMARIA_<value>` today or in the past 5 days | `systemicity` in `{systemic, mixed}` and `curative_action = 1` and `eradicant_action = 0` |
   | `FASE2` anywhere and no `FASE5` anywhere | `systemicity == contact` and `preventive_action = 1` and `curative_action = 0` and `eradicant_action = 0` |
   | `FASE5` both past/today and future | `systemicity == mixed` and `preventive_action = 1` and `curative_action = 1` and `eradicant_action = 0` |
   | `FASE5` in the future only | same predicates as the `FASE2`-only rule |
   | `FASE5` today or in the past only | `systemicity` in `{contact, mixed}` and `preventive_action = 1` and `curative_action = 0` and `eradicant_action = 0` |
   | none of the above | no FASE-specific cut; organic/crop/disease gates only |

9. **Vector search** — Weaviate `near_text` on `ProductProfile`, filtered server-side to the allowlisted `product_id`s
10. **Product JSON** — label details per product from `products`, `product_uses`, `label_chunks`, cached per `(crop, disease)`
11. **Context enrichment** — `allowed_products`, `applied_treatment` expansion, `last_advice` (with issuance `date`) from the `advice` table
12. **Advice generation** — second LLM profile returns the structured advisory
13. **Persistence** — the advisory plus its collapsed input are written to `advice` inside one transaction