AI Agro Support — Daily Pipeline

Daily agronomic advisory pipeline. For every field growing a configured crop, it builds an 11-day weather / phenology / disease-forecast JSON, asks an LLM to synthesise one or two vector-search queries, prefilters chemical products in SQL Server, and retrieves up to six matching products from a local Weaviate ProductProfile collection. It then enriches that JSON with full product label information and the previous advisory, asks a second LLM to write the farmer-facing advice, and stores the result in the advice table.

The pipeline is crop-first: config.yaml declares which crops and diseases to check, and each morning's batch run discovers every field that grows one of those crops and has a matching disease-forecast model configured, then runs the full sequence above for each (field, disease) pair — in parallel, with one field's failure never blocking another's.

Prerequisites

• Python 3.12+
• Microsoft SQL Server (DBMeteo) reachable with ODBC Driver 17 (or 18)
• Weaviate running locally (see docker-compose.yml) with a populated ProductProfile collection
• API key for the chosen LLM provider (Gemini by default)

Setup

cd d:\Uni\Thesis\AI-Agro-Support
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
copy .env.example .env
# Edit .env with SQL, Weaviate, and LLM credentials

Start Weaviate if it is not already running:

docker compose up -d

Configuration

• config.yaml — the crops: worklist, concurrency/rate limits, retry policy, the 09:00 SLA deadline, LLM provider/model, and vocab paths
• .env — SQL Server, Weaviate, and API keys
• vocab/crops.yaml / vocab/diseases.yaml — Italian → English maps
• prompts/ — the prompt registry (see below)

crops: worklist

crops:
  - crop: grapevine              # canonical English, matched via vocab/crops.yaml
    diseases:
      - model_name: PERONOSPORA  # matched against AI_agrosupport_agro_models.anmod_model
        disease: downy mildew    # canonical English, used for products/advice/prompts

Every morning, batch maps each crop to the fields growing it (AI_agrosupport_an_colture → AI_agrosupport_cmp_layers), joins to AI_agrosupport_agro_models for each listed model_name, and runs one job per matching (field, disease) pair. To add a crop or disease: add an entry here, add the Italian/English mapping to vocab/crops.yaml / vocab/diseases.yaml if it doesn't already exist, and add a prompt directory (see below) — otherwise the pair silently uses the generic fallback prompt.

worklist:, concurrency:, retry:, schedule:

worklist:
  require_enabled_model: true   # AI_agrosupport_agro_models.anmod_enabled must be 1
  field_allowlist: []           # restrict to specific field IDs while testing

concurrency:
  workers: 8                    # ThreadPoolExecutor size
  limits: { sql: 6, gemini: 4 } # see "Concurrency model" below
  gemini_requests_per_minute: 60

retry:
  attempts: 3
  initial_backoff_seconds: 2
  max_backoff_seconds: 30

schedule:
  deadline: "09:00"             # jobs not started by this local time are skipped

Prompt registry (prompts/)

Each crop-disease pair needs its own advice prompt (its phytopathological model's phase semantics, phenology notes, etc.). Prompts are plain Markdown files resolved by pipeline/prompts.py::PromptRegistry, with a shared fallback so a newly configured pair never fails outright — it just logs a warning and uses a generic prompt until a dedicated one is written.

prompts/
  _output_format.md                          shared JSON contract, appended to every advice system prompt
  query_synthesis/_default.md                generic query-synthesis system prompt
  query_synthesis/<crop>__<disease>.md        optional per-pair override
  advice/_default/{system.md,user.md}         fallback advice prompts
  advice/<crop>__<disease>/{system.md,user.md} per-pair advice prompts

The <crop>__<disease> slug is built from the same canonical English names used in config.yaml's crops: block (lower-cased, non-alphanumeric runs collapsed to _) — e.g. grapevine + downy mildew → grapevine__downy_mildew. To add a prompt for a new pair, create prompts/advice/<slug>/system.md and user.md; batch logs every configured pair still falling back to _default at startup.

Run

.\.venv\Scripts\Activate.ps1

# Daily entry point: every crop/disease pair in config.yaml, across all matching fields
python -m pipeline batch

# Historical / test dates (useful when the live window has no FASE2/FASE5)
python -m pipeline batch --as-of 2026-06-29

# Narrow to one crop, disease, and/or field while testing
python -m pipeline batch --crop grapevine --disease "downy mildew" --field-id 4085

# Re-run a partially failed morning without redoing already-written advice
python -m pipeline batch --skip-existing-advice

# Build every job's JSON but skip LLM calls, vector search, and the DB write
python -m pipeline batch --dry-run

# Single-field debugging (keeps the pre-scaling CLI behaviour)
python -m pipeline one --field-id 4085 --disease PERONOSPORA --dry-run

batch accepts --deadline HH:MM and --workers N to override schedule.deadline / concurrency.workers from config.yaml for one run.

Concurrency model

One ThreadPoolExecutor (concurrency.workers) runs every discovered job. Because dozens to a couple hundred jobs can run in the same morning, several things that used to be "open it, use it, close it" per field are now shared across the whole run (see pipeline/resources.py):

• SQL: each worker thread gets its own lazily-created pyodbc connection (connections cannot be shared across threads), kept open for the batch's lifetime. concurrency.limits.sql bounds how many jobs may run their SQL-heavy phase (forecast/weather/phenology/treatments lookup, or the product-JSON/advice-insert phase) at the same instant, independent of how many connections are open.
• Weaviate: one client is opened for the whole batch instead of one per field.
• Gemini: near_text search embeds through the same GEMINI_API_KEY as both LLM calls (ProductProfile is vectorised with text2vec-palm / gemini-embedding-001), so concurrency.limits.gemini and gemini_requests_per_minute are a single shared budget covering query synthesis, advice generation, and vector search — not two independent ones.
• Products: the whole products table is loaded into memory once per batch (pipeline/stages/products.py::ProductIndex) instead of re-scanned and re-parsed once per field, and ProductJsonBuilder is shared per (crop, disease) pair instead of per field.

Each of the three external boundaries — LLM, Weaviate, SQL — has its own tenacity retry policy in pipeline/retry.py (exponential backoff with jitter, retry.attempts tries), applied automatically inside Resources.call_llm / Resources.search_products and around the insert_advice transaction.

Output

Each run writes to output/<YYYY-MM-DD>/:

• field<field_id>__<crop>__<disease slug>.json — one file per job (see fields below)
• _run_report.json — the operational artifact for the morning: totals by status plus a per-job breakdown (status, duration_s, recommended_count, allowed_products_empty, message, and, on failure, error_type / error_message)

Per-job payload fields:

Field	Description
run	Metadata (as_of, field, crop, disease, organic, station, …)
json_for_advice_generation	Full 11-day meteorological + forecast payload, plus allowed_products, applied_treatment and last_advice; this is what the advice LLM receives
json_for_query_synthesis	Subset of the above, sent to the LLM for search query generation
generated_queries	One or two natural-language search queries
candidate_count	Products remaining after the prefilter
recommended_products	Up to 6 items with product_name + registration_number
last_advice	advice_summary + suggested_products of the most recent previous advisory
advice	Parsed LLM output: full_advice, advice_summary, apply_treatment, treatments, dosage, apply_date
sent_information	The advice payload with every embedded product label collapsed back to a product name; mirrors advice.sent_information
advice_inserted	Whether the advice row was written
status	ok, no_risk, or dry_run (see _run_report.json for failed / skipped_*, which have no per-job file)
message	Human-readable summary of the run outcome

If neither FASE2 nor FASE5 appears in a field's 11-day window, the product search is skipped and allowed_products is empty, but advice is still generated and stored; that job is labelled no_risk. The run report also counts ok jobs whose allowed_products ended up empty anyway (for example an organic field with no organic-certified product in the catalog yet) — watch that number as the products table fills in, since a sudden jump usually means the prefilter or the Weaviate allowlist filter is misbehaving rather than that the catalog is genuinely empty for that combination.

The process exit code is 0 only if every job in the report is ok, no_risk, dry_run, or a deliberate skipped_*; it is 1 if any job's status is failed.

Database writes

Each non-dry-run job inserts one row into advice for (date, field_id, disease). That triple is the table's primary key, so any existing row for the same triple is deleted first — re-running a day replaces its advisory instead of failing on a duplicate key. The delete-then-insert pair runs inside one transaction (pipeline/db.py::transaction) so a crash between the two statements cannot leave a field's advisory missing for the day, and deadlocks are retried per retry: in config.yaml.

The primary key's clustered index on ([date], field_id, disease) also lets that DELETE seek straight to the rows it needs instead of scanning the table, so concurrent workers don't block each other on unrelated rows.

Daily scheduling

Provide a CLI entry point only. Schedule it yourself on the deployment machine. Advice must be ready by the schedule.deadline (09:00 by default), and weather data should stay fresh, so start the run no earlier than ~07:00 — for example via Windows Task Scheduler, daily at 07:00:

d:\Uni\Thesis\AI-Agro-Support\.venv\Scripts\python.exe -m pipeline batch --skip-existing-advice

with working directory d:\Uni\Thesis\AI-Agro-Support. --skip-existing-advice makes a re-trigger (e.g. a scheduled retry later that morning) resumable at no extra cost: it does one bulk check against advice up front and only runs jobs that don't have a row yet.

Observability (optional)

The pipeline can send OpenTelemetry/OpenInference traces to a self-hosted Arize Phoenix instance for debugging a bad advisory, comparing prompts/models later, and seeing per-job token usage and USD cost for the two LLM stages. It is off by default, fail-open (a down or missing Phoenix never fails batch or one), and not a runtime dependency: nothing about the daily 07:00–09:00 run depends on it.

Start Phoenix

docker compose up -d

This brings up Weaviate and Phoenix together. Phoenix serves its UI (and the OTLP/HTTP trace collector) at http://localhost:6006. SQLite on the phoenix_data Docker volume is enough for this single-machine setup — see "Why SQLite / why not Langfuse" below.

Turn tracing on

Tracing is controlled by observability: in config.yaml:

observability:
  enabled: false                                  # default: off
  project_name: ai-agro-support
  endpoint: http://localhost:6006/v1/traces
  hide_prompts: true                               # keep prompt/payload text out of spans
  max_attribute_chars: 4096                        # cap every exported string attribute

Any of these can be overridden per-environment without editing the file:

Env var	Overrides
PHOENIX_TRACING_ENABLED	observability.enabled
PHOENIX_COLLECTOR_ENDPOINT	observability.endpoint
PHOENIX_PROJECT_NAME	observability.project_name

With tracing on, python -m pipeline one or batch registers the tracer once at startup (pipeline/observability.py::setup_tracing, called from pipeline/__main__.py right after load_settings()), then instruments google-genai (and openai / anthropic, if those llm: profiles are ever enabled) via OpenInference auto-instrumentors. Spans are batched and exported on a background thread, so a slow or down collector cannot block a worker or push the run past schedule.deadline.

What you get per job

Every (field, crop, disease, as_of) job produces one trace, rooted at a run_job span, with:

• a query_synthesis span (gemini-2.5-flash by default) and an advice_generation span (gemini-2.5-pro), each wrapping the provider's own auto-instrumented LLM span with llm.model_name, llm.provider, and token counts (llm.token_count.prompt / completion / total, plus completion_details.reasoning for Gemini 2.5 thinking tokens);
• a search_products retriever span with the synthesized queries and the retrieved product IDs/distances (never product names or label content);
• dry_run jobs get only the run_job span — no LLM or retrieval children, since --dry-run skips those calls entirely.

Phoenix computes cost per span from its built-in model pricing table, which already includes gemini-2.5-flash and gemini-2.5-pro — no custom entry under Settings → Models is needed for either model as configured today. If a model is ever renamed or swapped to one Phoenix doesn't recognise, add it there (regex Name Pattern, provider google, per-1M-token prices).

Trace and span inputs are hidden by default (observability.hide_prompts): the full json_for_advice_generation payload (weather, phenology, product labels) never leaves the process as span data. Outputs — the generated advice text — stay visible, since that is what you actually want to read when debugging a bad advisory; output/<date>/*.json and _run_report.json remain the authoritative operational artifacts, not Phoenix.

Known gap: embedding cost

search_products's Weaviate near_text calls are embedded inside the Weaviate container (text2vec-google), using the same GEMINI_API_KEY as the two LLM calls, but that request never passes through this process's Python google-genai client — so no OpenInference instrumentor can see it. Phoenix's per-job cost therefore covers the two LLM calls only and slightly undercounts total Gemini spend (1–2 gemini-embedding-001 calls per job, a small fraction of a cent at this volume). The search_products span still records query_count, so this gap can be estimated later if it matters; it is not worth a custom cost pipeline at this scale.

Why SQLite / why not Langfuse or Phoenix Cloud

• SQLite, not Postgres: this is a single-machine, single-writer deployment; Phoenix officially supports SQLite on a mounted volume for exactly this case. Postgres would add a second container and volume for no benefit here (switch later via PHOENIX_SQL_DATABASE_URL if that changes).
• Self-hosted, not Phoenix Cloud: farm/field data stays in the deployment by design — nothing here talks to a hosted endpoint.
• Phoenix, not Langfuse: Langfuse's self-hosted stack needs Postgres and ClickHouse; Phoenix needs one container and ingests plain OTLP + OpenInference, so this pipeline isn't locked into either vendor.

Pipeline stages

Run once per (field, disease) job by pipeline/job.py::run_job:

1. Worklist — crop → fields (AI_agrosupport_an_colture + AI_agrosupport_cmp_layers) → disease model (AI_agrosupport_agro_models); organic flag is cmplay_imp == 3 (pipeline/worklist.py, run once per batch, not per job)
2. Disease model — per-day FASE5 > FASE2 resolution over as_of±5, from the worklist's known anmod_id
3. Weather — TDatiMeteo_D for past days, TDatiMeteo_D_FRC for today/future
4. Phenology — carry-forward of latest observation ≤ day; future days null; if today's phase is missing, use yesterday's
5. Applied treatments — products sprayed over as_of-5 .. as_of, from AI_agrosupport_agro_ril_operations* with rilop_operation = 10
6. LLM — synthesise 1–2 anonymised search queries from the subset
7. Product prefilter — organic / crop / disease / FASE rules against the batch-wide, in-memory products index
8. Vector search — Weaviate near_text on ProductProfile, filtered server-side to the allowlisted product_ids
9. Product JSON — label details per product from products, product_uses, label_chunks, cached per (crop, disease)
10. Context enrichment — allowed_products, applied_treatment expansion, last_advice from the advice table
11. Advice generation — second LLM profile returns the structured advisory
12. Persistence — the advisory plus its collapsed input are written to advice inside one transaction