AI_Agro_Support/README.md

# AI Agro Support — Daily Pipeline

Daily agronomic advisory pipeline. For every field growing a configured crop, it
builds an 11-day weather / phenology / disease-forecast JSON, asks an LLM to
synthesise one or two vector-search queries, prefilters chemical products in SQL
Server, and retrieves up to six matching products from a local Weaviate
`ProductProfile` collection. It then enriches that JSON with full product label
information and the previous advisory, asks a second LLM to write the
farmer-facing advice, and stores the result in the `advice` table.

The pipeline is **crop-first**: `config.yaml` declares which crops and diseases
to check, and each morning's `batch` run discovers every field that grows one of
those crops and has a matching disease-forecast model configured, then runs the
full sequence above for each `(field, disease)` pair — in parallel, with one
field's failure never blocking another's.

## Prerequisites

- Python 3.12+
- Microsoft SQL Server (`DBMeteo`) reachable with ODBC Driver 17 (or 18)
- Weaviate running locally (see `docker-compose.yml`) with a populated `ProductProfile` collection
- API key for the chosen LLM provider (Gemini by default)

## Setup

```powershell
cd d:\Uni\Thesis\AI-Agro-Support
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
copy .env.example .env
# Edit .env with SQL, Weaviate, and LLM credentials
```

Start Weaviate if it is not already running:

```powershell
docker compose up -d
```

## Configuration

- [`config.yaml`](config.yaml) — the `crops:` worklist, concurrency/rate limits, retry policy,
  the 09:00 SLA deadline, LLM provider/model, and vocab paths
- [`.env`](.env) — SQL Server, Weaviate, and API keys
- [`vocab/crops.yaml`](vocab/crops.yaml) / [`vocab/diseases.yaml`](vocab/diseases.yaml) — Italian → English maps
- [`prompts/`](prompts) — the prompt registry (see below)

### `crops:` worklist

```yaml
crops:
  - crop: grapevine              # canonical English, matched via vocab/crops.yaml
    diseases:
      - model_name: PERONOSPORA  # matched against AI_agrosupport_agro_models.anmod_model
        disease: downy mildew    # canonical English, used for products/advice/prompts
```

Every morning, `batch` maps each `crop` to the fields growing it
(`AI_agrosupport_an_colture` → `AI_agrosupport_cmp_layers`), joins to
`AI_agrosupport_agro_models` for each listed `model_name`, and runs one job per
matching `(field, disease)` pair. To add a crop or disease: add an entry here, add
the Italian/English mapping to `vocab/crops.yaml` / `vocab/diseases.yaml` if it
doesn't already exist, and add a prompt directory (see below) — otherwise the
pair silently uses the generic fallback prompt.

### `worklist:`, `concurrency:`, `retry:`, `schedule:`

```yaml
worklist:
  require_enabled_model: true   # AI_agrosupport_agro_models.anmod_enabled must be 1
  field_allowlist: []           # restrict to specific field IDs while testing

concurrency:
  workers: 8                    # ThreadPoolExecutor size
  limits: { sql: 6, gemini: 4 } # see "Concurrency model" below
  gemini_requests_per_minute: 60

retry:
  attempts: 3
  initial_backoff_seconds: 2
  max_backoff_seconds: 30

schedule:
  deadline: "09:00"             # jobs not started by this local time are skipped
```

### Prompt registry (`prompts/`)

Each crop-disease pair needs its own advice prompt (its phytopathological
model's phase semantics, phenology notes, etc.). Prompts are plain Markdown
files resolved by `pipeline/prompts.py::PromptRegistry`, with a shared fallback
so a newly configured pair never fails outright — it just logs a warning and
uses a generic prompt until a dedicated one is written.

```
prompts/
  _output_format.md                          shared JSON contract, appended to every advice system prompt
  query_synthesis/_default.md                generic query-synthesis system prompt
  query_synthesis/<crop>__<disease>.md        optional per-pair override
  advice/_default/{system.md,user.md}         fallback advice prompts
  advice/<crop>__<disease>/{system.md,user.md} per-pair advice prompts
```

The `<crop>__<disease>` slug is built from the same canonical English names used
in `config.yaml`'s `crops:` block (lower-cased, non-alphanumeric runs collapsed
to `_`) — e.g. `grapevine` + `downy mildew` → `grapevine__downy_mildew`. To add a
prompt for a new pair, create `prompts/advice/<slug>/system.md` and `user.md`;
`batch` logs every configured pair still falling back to `_default` at startup.

## Run

```powershell
.\.venv\Scripts\Activate.ps1

# Daily entry point: every crop/disease pair in config.yaml, across all matching fields
python -m pipeline batch

# Historical / test dates (useful when the live window has no FASE2/FASE5)
python -m pipeline batch --as-of 2026-06-29

# Narrow to one crop, disease, and/or field while testing
python -m pipeline batch --crop grapevine --disease "downy mildew" --field-id 4085

# Re-run a partially failed morning without redoing already-written advice
python -m pipeline batch --skip-existing-advice

# Build every job's JSON but skip LLM calls, vector search, and the DB write
python -m pipeline batch --dry-run

# Single-field debugging (keeps the pre-scaling CLI behaviour)
python -m pipeline one --field-id 4085 --disease PERONOSPORA --dry-run
```

`batch` accepts `--deadline HH:MM` and `--workers N` to override
`schedule.deadline` / `concurrency.workers` from `config.yaml` for one run.

## Concurrency model

One `ThreadPoolExecutor` (`concurrency.workers`) runs every discovered job.
Because dozens to a couple hundred jobs can run in the same morning, several
things that used to be "open it, use it, close it" per field are now shared
across the whole run (see `pipeline/resources.py`):

- **SQL**: each worker thread gets its own lazily-created `pyodbc` connection
  (connections cannot be shared across threads), kept open for the batch's
  lifetime. `concurrency.limits.sql` bounds how many jobs may run their
  SQL-heavy phase (forecast/weather/phenology/treatments lookup, or the
  product-JSON/advice-insert phase) at the same instant, independent of how
  many connections are open.
- **Weaviate**: one client is opened for the whole batch instead of one per
  field.
- **Gemini**: `near_text` search embeds through the same `GEMINI_API_KEY` as
  both LLM calls (`ProductProfile` is vectorised with text2vec-palm /
  gemini-embedding-001), so `concurrency.limits.gemini` and
  `gemini_requests_per_minute` are a single shared budget covering query
  synthesis, advice generation, *and* vector search — not two independent
  ones.
- **Products**: the whole `products` table is loaded into memory once per
  batch (`pipeline/stages/products.py::ProductIndex`) instead of re-scanned
  and re-parsed once per field, and `ProductJsonBuilder` is shared per
  `(crop, disease)` pair instead of per field.

Each of the three external boundaries — LLM, Weaviate, SQL — has its own
`tenacity` retry policy in `pipeline/retry.py` (exponential backoff with
jitter, `retry.attempts` tries), applied automatically inside
`Resources.call_llm` / `Resources.search_products` and around the
`insert_advice` transaction.

## Output

Each run writes to `output/<YYYY-MM-DD>/`:

- `field<field_id>__<crop>__<disease slug>.json` — one file per job (see fields below)
- `_run_report.json` — the operational artifact for the morning: totals by
  status plus a per-job breakdown (`status`, `duration_s`,
  `recommended_count`, `allowed_products_empty`, `message`, and, on failure,
  `error_type` / `error_message`)

Per-job payload fields:

| Field | Description |
|---|---|
| `run` | Metadata (as_of, field, crop, disease, organic, station, …) |
| `json_for_advice_generation` | Full 11-day meteorological + forecast payload, plus `allowed_products`, `applied_treatment` and `last_advice`; this is what the advice LLM receives |
| `json_for_query_synthesis` | Subset of the above, sent to the LLM for search query generation |
| `generated_queries` | One or two natural-language search queries |
| `candidate_count` | Products remaining after the prefilter |
| `recommended_products` | Up to 6 items with `product_name` + `registration_number` |
| `last_advice` | `advice_summary` + `suggested_products` of the most recent previous advisory |
| `advice` | Parsed LLM output: `full_advice`, `advice_summary`, `apply_treatment`, `treatments`, `dosage`, `apply_date` |
| `sent_information` | The advice payload with every embedded product label collapsed back to a product name; mirrors `advice.sent_information` |
| `advice_inserted` | Whether the `advice` row was written |
| `status` | `ok`, `no_risk`, or `dry_run` (see `_run_report.json` for `failed` / `skipped_*`, which have no per-job file) |
| `message` | Human-readable summary of the run outcome |

If neither `FASE2` nor `FASE5` appears in a field's 11-day window, the product
search is skipped and `allowed_products` is empty, but advice is still generated
and stored; that job is labelled `no_risk`. The run report also counts `ok` jobs
whose `allowed_products` ended up empty anyway (for example an organic field
with no organic-certified product in the catalog yet) — watch that number as the
`products` table fills in, since a sudden jump usually means the prefilter or
the Weaviate allowlist filter is misbehaving rather than that the catalog is
genuinely empty for that combination.

The process exit code is `0` only if every job in the report is `ok`, `no_risk`,
`dry_run`, or a deliberate `skipped_*`; it is `1` if any job's status is `failed`.

## Database writes

Each non-dry-run job inserts one row into `advice` for `(date, field_id,
disease)`. That triple is the table's primary key, so any existing row for
the same triple is deleted first — re-running a day replaces its advisory
instead of failing on a duplicate key. The delete-then-insert pair runs
inside one transaction (`pipeline/db.py::transaction`) so a crash between
the two statements cannot leave a field's advisory missing for the day, and
deadlocks are retried per `retry:` in config.yaml.

The primary key's clustered index on `([date], field_id, disease)` also
lets that DELETE seek straight to the rows it needs instead of scanning the
table, so concurrent workers don't block each other on unrelated rows.

## Daily scheduling

Provide a CLI entry point only. Schedule it yourself on the deployment machine.
Advice must be ready by the `schedule.deadline` (09:00 by default), and weather
data should stay fresh, so start the run no earlier than ~07:00 — for example
via Windows Task Scheduler, daily at 07:00:

```
d:\Uni\Thesis\AI-Agro-Support\.venv\Scripts\python.exe -m pipeline batch --skip-existing-advice
```

with working directory `d:\Uni\Thesis\AI-Agro-Support`. `--skip-existing-advice`
makes a re-trigger (e.g. a scheduled retry later that morning) resumable at no
extra cost: it does one bulk check against `advice` up front and only runs jobs
that don't have a row yet.

## Pipeline stages

Run once per `(field, disease)` job by `pipeline/job.py::run_job`:

1. **Worklist** — crop → fields (`AI_agrosupport_an_colture` + `AI_agrosupport_cmp_layers`) → disease model (`AI_agrosupport_agro_models`); organic flag is `cmplay_imp == 3` (`pipeline/worklist.py`, run once per batch, not per job)
2. **Disease model** — per-day `FASE5` > `FASE2` resolution over as_of±5, from the worklist's known `anmod_id`
3. **Weather** — `TDatiMeteo_D` for past days, `TDatiMeteo_D_FRC` for today/future
4. **Phenology** — carry-forward of latest observation ≤ day; future days null
5. **Applied treatments** — products sprayed over `as_of-5 .. as_of`, from `AI_agrosupport_agro_ril_operations*` with `rilop_operation = 10`
6. **LLM** — synthesise 1–2 anonymised search queries from the subset
7. **Product prefilter** — organic / crop / disease / FASE rules against the batch-wide, in-memory `products` index
8. **Vector search** — Weaviate `near_text` on `ProductProfile`, filtered server-side to the allowlisted `product_id`s
9. **Product JSON** — label details per product from `products`, `product_uses`, `label_chunks`, cached per `(crop, disease)`
10. **Context enrichment** — `allowed_products`, `applied_treatment` expansion, `last_advice` from the `advice` table
11. **Advice generation** — second LLM profile returns the structured advisory
12. **Persistence** — the advisory plus its collapsed input are written to `advice` inside one transaction