AI Agro Support — Daily Pipeline

Daily agronomic advisory pipeline. For every field growing a configured crop, it builds an 11-day weather / phenology / disease-forecast JSON, asks an LLM to synthesise one or two vector-search queries, prefilters chemical products in SQL Server, and retrieves up to six matching products from a local Weaviate ProductProfile collection. It then enriches that JSON with full product label information and the previous advisory, asks a second LLM to write the farmer-facing advice, and stores the result in the advice table.

The pipeline is crop-first: config.yaml declares which crops and diseases to check, and each morning's batch run discovers every field that grows one of those crops and has a matching disease-forecast model configured, then runs the full sequence above for each (field, disease) pair — in parallel, with one field's failure never blocking another's.

Prerequisites

• Python 3.12+
• Microsoft SQL Server (DBMeteo) reachable with ODBC Driver 17 (or 18)
• Weaviate running locally (see docker-compose.yml) with a populated ProductProfile collection
• API key for the chosen LLM provider (Gemini by default)

Setup

cd d:\Uni\Thesis\AI-Agro-Support
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
copy .env.example .env
# Edit .env with SQL, Weaviate, and LLM credentials

Start Weaviate if it is not already running:

docker compose up -d

Configuration

• config.yaml — the crops: worklist, concurrency/rate limits, retry policy, the 09:00 SLA deadline, LLM provider/model, and vocab paths
• .env — SQL Server, Weaviate, and API keys
• vocab/crops.yaml / vocab/diseases.yaml — Italian → English maps
• prompts/ — the prompt registry (see below)

crops: worklist

crops:
  - crop: grapevine              # canonical English, matched via vocab/crops.yaml
    diseases:
      - model_name: PERONOSPORA  # matched against AI_agrosupport_agro_models.anmod_model
        disease: downy mildew    # canonical English, used for products/advice/prompts

Every morning, batch maps each crop to the fields growing it (AI_agrosupport_an_colture → AI_agrosupport_cmp_layers), joins to AI_agrosupport_agro_models for each listed model_name, and runs one job per matching (field, disease) pair. To add a crop or disease: add an entry here, add the Italian/English mapping to vocab/crops.yaml / vocab/diseases.yaml if it doesn't already exist, and add a prompt directory (see below) — otherwise the pair silently uses the generic fallback prompt.

worklist:, concurrency:, retry:, schedule:

worklist:
  require_enabled_model: true   # AI_agrosupport_agro_models.anmod_enabled must be 1
  field_allowlist: []           # restrict to specific field IDs while testing

concurrency:
  workers: 8                    # ThreadPoolExecutor size
  limits: { sql: 6, gemini: 4 } # see "Concurrency model" below
  gemini_requests_per_minute: 60

retry:
  attempts: 3
  initial_backoff_seconds: 2
  max_backoff_seconds: 30

schedule:
  deadline: "09:00"             # jobs not started by this local time are skipped

Prompt registry (prompts/)

Each crop-disease pair needs its own advice prompt (its phytopathological model's phase semantics, phenology notes, etc.). Prompts are plain Markdown files resolved by pipeline/prompts.py::PromptRegistry, with a shared fallback so a newly configured pair never fails outright — it just logs a warning and uses a generic prompt until a dedicated one is written.

prompts/
  _output_format.md                          shared JSON contract, appended to every advice system prompt
  query_synthesis/_default.md                generic query-synthesis system prompt
  query_synthesis/<crop>__<disease>.md        optional per-pair override
  advice/_default/{system.md,user.md}         fallback advice prompts
  advice/<crop>__<disease>/{system.md,user.md} per-pair advice prompts

The <crop>__<disease> slug is built from the same canonical English names used in config.yaml's crops: block (lower-cased, non-alphanumeric runs collapsed to _) — e.g. grapevine + downy mildew → grapevine__downy_mildew. To add a prompt for a new pair, create prompts/advice/<slug>/system.md and user.md; batch logs every configured pair still falling back to _default at startup.

Run

.\.venv\Scripts\Activate.ps1

# Daily entry point: every crop/disease pair in config.yaml, across all matching fields
python -m pipeline batch

# Historical / test dates (useful when the live window has no FASE2/FASE5)
python -m pipeline batch --as-of 2026-06-29

# Narrow to one crop, disease, and/or field while testing
python -m pipeline batch --crop grapevine --disease "downy mildew" --field-id 4085

# Re-run a partially failed morning without redoing already-written advice
python -m pipeline batch --skip-existing-advice

# Build every job's JSON but skip LLM calls, vector search, and the DB write
python -m pipeline batch --dry-run

# Single-field debugging (keeps the pre-scaling CLI behaviour)
python -m pipeline one --field-id 4085 --disease PERONOSPORA --dry-run

batch accepts --deadline HH:MM and --workers N to override schedule.deadline / concurrency.workers from config.yaml for one run.

Concurrency model

One ThreadPoolExecutor (concurrency.workers) runs every discovered job. Because dozens to a couple hundred jobs can run in the same morning, several things that used to be "open it, use it, close it" per field are now shared across the whole run (see pipeline/resources.py):

• SQL: each worker thread gets its own lazily-created pyodbc connection (connections cannot be shared across threads), kept open for the batch's lifetime. concurrency.limits.sql bounds how many jobs may run their SQL-heavy phase (forecast/weather/phenology/treatments lookup, or the product-JSON/advice-insert phase) at the same instant, independent of how many connections are open.
• Weaviate: one client is opened for the whole batch instead of one per field.
• Gemini: near_text search embeds through the same GEMINI_API_KEY as both LLM calls (ProductProfile is vectorised with text2vec-palm / gemini-embedding-001), so concurrency.limits.gemini and gemini_requests_per_minute are a single shared budget covering query synthesis, advice generation, and vector search — not two independent ones.
• Products: the whole products table is loaded into memory once per batch (pipeline/stages/products.py::ProductIndex) instead of re-scanned and re-parsed once per field, and ProductJsonBuilder is shared per (crop, disease) pair instead of per field.

Each of the three external boundaries — LLM, Weaviate, SQL — has its own tenacity retry policy in pipeline/retry.py (exponential backoff with jitter, retry.attempts tries), applied automatically inside Resources.call_llm / Resources.search_products and around the insert_advice transaction.

Output

Each run writes to output/<YYYY-MM-DD>/:

• field<field_id>__<crop>__<disease slug>.json — one file per job (see fields below)
• _run_report.json — the operational artifact for the morning: totals by status plus a per-job breakdown (status, duration_s, recommended_count, allowed_products_empty, message, and, on failure, error_type / error_message)

Per-job payload fields:

Field	Description
run	Metadata (as_of, field, crop, disease, organic, station, …)
json_for_advice_generation	Full 11-day meteorological + forecast payload, plus allowed_products, applied_treatment and last_advice; this is what the advice LLM receives
json_for_query_synthesis	Subset of the above, sent to the LLM for search query generation
generated_queries	One or two natural-language search queries
candidate_count	Products remaining after the prefilter
recommended_products	Up to 6 items with product_name + registration_number
last_advice	advice_summary + suggested_products of the most recent previous advisory
advice	Parsed LLM output: full_advice, advice_summary, apply_treatment, treatments, dosage, apply_date
sent_information	The advice payload with every embedded product label collapsed back to a product name; mirrors advice.sent_information
advice_inserted	Whether the advice row was written
status	ok, no_risk, or dry_run (see _run_report.json for failed / skipped_*, which have no per-job file)
message	Human-readable summary of the run outcome

If neither FASE2 nor FASE5 appears in a field's 11-day window, the product search is skipped and allowed_products is empty, but advice is still generated and stored; that job is labelled no_risk. The run report also counts ok jobs whose allowed_products ended up empty anyway (for example an organic field with no organic-certified product in the catalog yet) — watch that number as the products table fills in, since a sudden jump usually means the prefilter or the Weaviate allowlist filter is misbehaving rather than that the catalog is genuinely empty for that combination.

The process exit code is 0 only if every job in the report is ok, no_risk, dry_run, or a deliberate skipped_*; it is 1 if any job's status is failed.

Database writes

Each non-dry-run job inserts one row into advice for (date, field_id, disease). That triple is the table's primary key, so any existing row for the same triple is deleted first — re-running a day replaces its advisory instead of failing on a duplicate key. The delete-then-insert pair runs inside one transaction (pipeline/db.py::transaction) so a crash between the two statements cannot leave a field's advisory missing for the day, and deadlocks are retried per retry: in config.yaml.

The primary key's clustered index on ([date], field_id, disease) also lets that DELETE seek straight to the rows it needs instead of scanning the table, so concurrent workers don't block each other on unrelated rows.

Daily scheduling

Provide a CLI entry point only. Schedule it yourself on the deployment machine. Advice must be ready by the schedule.deadline (09:00 by default), and weather data should stay fresh, so start the run no earlier than ~07:00 — for example via Windows Task Scheduler, daily at 07:00:

d:\Uni\Thesis\AI-Agro-Support\.venv\Scripts\python.exe -m pipeline batch --skip-existing-advice

with working directory d:\Uni\Thesis\AI-Agro-Support. --skip-existing-advice makes a re-trigger (e.g. a scheduled retry later that morning) resumable at no extra cost: it does one bulk check against advice up front and only runs jobs that don't have a row yet.

Pipeline stages

Run once per (field, disease) job by pipeline/job.py::run_job:

1. Worklist — crop → fields (AI_agrosupport_an_colture + AI_agrosupport_cmp_layers) → disease model (AI_agrosupport_agro_models); organic flag is cmplay_imp == 3 (pipeline/worklist.py, run once per batch, not per job)
2. Disease model — per-day FASE5 > FASE2 resolution over as_of±5, from the worklist's known anmod_id
3. Weather — TDatiMeteo_D for past days, TDatiMeteo_D_FRC for today/future
4. Phenology — carry-forward of latest observation ≤ day; future days null
5. Applied treatments — products sprayed over as_of-5 .. as_of, from AI_agrosupport_agro_ril_operations* with rilop_operation = 10
6. LLM — synthesise 1–2 anonymised search queries from the subset
7. Product prefilter — organic / crop / disease / FASE rules against the batch-wide, in-memory products index
8. Vector search — Weaviate near_text on ProductProfile, filtered server-side to the allowlisted product_ids
9. Product JSON — label details per product from products, product_uses, label_chunks, cached per (crop, disease)
10. Context enrichment — allowed_products, applied_treatment expansion, last_advice from the advice table
11. Advice generation — second LLM profile returns the structured advisory
12. Persistence — the advisory plus its collapsed input are written to advice inside one transaction