BINS Invoice Ingestion Flow

BINS (formerly Trashy) is the Wasteology invoice management application. It receives structured invoice data automatically from the invoice-listener pipeline, which uses Google Document AI to extract fields from PDF invoices arriving via email or OneDrive.

GAPI was the legacy SharePoint-based invoice system. The migration to BINS is complete — all users are on BINS. Migration tooling is no longer maintained.

---

The Two Invoice Systems

System	Technology	Status
GAPI (legacy)	SharePoint list	Phased out — migration complete
BINS (active)	FastAPI + React + Cloud SQL PostgreSQL	Active — all users migrated

---

End-to-End Flow

PDF Invoice Arrives
  (email: invoices@wasteologygroup.com OR OneDrive folder)
  │
  ▼
invoice-listener: Listener Jobs (GCP Cloud Run Jobs)
  Every 10 min (email) / 1 hr (OneDrive)
  Publishes blob path to Pub/Sub topic
  │
  ▼
invoice-listener: HTTP Handlers (GCP Cloud Run Services)
  Receives Pub/Sub push message
  │
  ▼
orchestrator_function.py
  ├── Download PDF from Azure Blob Storage (pdfinvoices container)
  ├── Compute MD5 hash — deduplicate against GCS
  ├── Upload PDF to GCS: {GCS_PDF_BUCKET}/{md5}.pdf
  ├── Log to Azure SQL: docai.raw_invoices (md5, source, name)
  ├── Call Google Document AI → entity extraction
  ├── Save DocAI JSON to GCS: {GCS_RESPONSES_BUCKET}/{md5}
  ├── Run Service Normalization ML API on each line item
  │     → Azure SQL: docai.raw_invoice_normalized_line_items
  └── Fan out to 2 parallel queues:
        │
        ├─ Work orders queue → downstream work order system
        │     Passes MD5 + raw OCR text
        │
        └─ BINS/Trashy queue → trashy_function.py
              POST /invoices/ingest → BINS FastAPI backend
              │
              ├── Creates trashy.invoices row (Cloud SQL PostgreSQL)
              └── Creates trashy.InvoiceProcessingStatus row
                    (4-stage pipeline audit log)

---

Document AI Extracted Fields

Google Document AI extracts these fields from the PDF. The orchestrator maps them to a structured dict that is passed to all downstream queues:

Field	BINS Column	Notes
Vendor name	vendor_name
Invoice number	invoice_number
Invoice total	invoice_amount
Amount due	amount_due
Invoice date	invoice_date
Service date	service_date
Due date	due_date
Service address	service_address
Matched CieTrade location	location	Resolved via Azure address service
Assigned billing rep	assigned_to	Assigned in BINS via Azure address service
State / region	state	Resolved via Azure address service
Account number	account_number
Purchase order	purchase_order
Retail pricing notes	retail_pricing_notes

---

Line Item ML Classification

Each line item description is sent to the Service Normalization API during ingestion:

Endpoint: https://service-normalization.agreeablesmoke-44fc50ff.centralus.azurecontainerapps.io/predict/service_info/

Input: {"raw_text": "<line item description>"}

Output per line item:

Label	Description
service_desc_label + _proba	Service description category + confidence
size_label + _proba	Container/equipment size + confidence
material_label + _proba	Material type + confidence
container_label + _proba	Container type + confidence
is_valid + _proba	Whether the line item is valid/parseable + confidence

Results stored in docai.raw_invoice_normalized_line_items on Azure SQL Server.

Pathfinder project

This classifier was built as part of the Pathfinder initiative. Confidence scores are stored in docai.raw_invoice_normalized_line_items but are not actively used in the current BINS UI or any automated workflow. The data is available as a potential signal for future complexity classification.

Complexity signal

The is_valid_proba and _proba scores per line item are the best existing ML signal for invoice complexity. Invoices with many low-confidence or invalid line items are harder to process. See Invoice Complexity Classification for how to use this.

---

Data Written Per Invoice

System	Table / Location	What
Azure SQL Server	docai.raw_invoices	MD5, source, filename (dedup audit)
Azure SQL Server	docai.raw_invoice_normalized_line_items	ML labels + confidence per line item
GCS	{GCS_PDF_BUCKET}/{md5}.pdf	Raw PDF copy
GCS	{GCS_RESPONSES_BUCKET}/{md5}	Document AI JSON response
Cloud SQL PostgreSQL	trashy.invoices	Full structured invoice record
Cloud SQL PostgreSQL	trashy.InvoiceProcessingStatus	4-stage pipeline audit log

---

Pipeline Stage Audit (trashy.InvoiceProcessingStatus)

Every uploaded invoice gets a row tracking its progress through 4 independent pipeline stages:

Column	Possible Values	What It Tracks
status_storage	PENDING / SUCCESS / FAILED	PDF uploaded to GCS
status_doc_ai	PENDING / SUCCESS / FAILED	Document AI OCR completed
status_database_insert	PENDING / SUCCESS / FAILED / DUPLICATE	Invoice row written to BINS DB
status_webservice_call	PENDING / SUCCESS / FAILED	Address + assignment webservice call
last_error_message	text	Error detail for last failed stage
retry_count	integer	Number of manual retries
created_at	timestamp	When ingestion was first attempted
updated_at	timestamp	Last state change

Admin view: GET /admin/processing-status?status=FAILED

---

BINS Backend Reporting Endpoints

Method	Endpoint	Description
POST	/invoices/query	Paginated filtered invoice list
POST	/invoices/export	XLSX export — sync (≤1K rows) or async via GCS (>1K rows)
GET	/admin/processing-status	All pipeline status records; filter with ?status=FAILED
GET	/admin/processing-status/{md5}	Single invoice pipeline status
POST	/admin/processing-status/{md5}/retry	Reset failed stage back to PENDING
GET	/status/pipeline	GCP Cloud Function + Scheduler health (admin only)

No aggregate stats today

There is no /api/stats, /api/counts, or GROUP-BY-status endpoint in the BINS backend. Aggregate reporting requires either exporting data or querying the trashy.invoices table directly.

---

GCP Infrastructure

All invoice-listener components run on GCP project academic-torch-405913:

Resource Type	Name	Schedule / Trigger
Cloud Run Job	wasteologyinvoice-email-queued-listener	Every 10 min
Cloud Run Job	wasteologyinvoice-onedrive-listener-queued	Every 1 hour
Cloud Run Job	wasteologyinvoice-error-email-processor	Every 1 hour
Cloud Run Service	wasteololgy-email-handlers	Pub/Sub push
Cloud Run Service	wasteololgy-onedrive-handlers	Pub/Sub push
Cloud Run Service	wasteologyinvoice-mailgun-handler	Pub/Sub push
Cloud Function	db_inserter	Pub/Sub trigger
Cloud Function	doc_ai_processor	Pub/Sub trigger
Cloud Function	webservice_caller	Pub/Sub trigger

Active services

The active Cloud Run Services are the email, OneDrive, and trashy handlers, plus the recently added mailgun handler. Some Cloud Functions listed above may be inactive — confirm with the BINS team before modifying them.

See GCP Infrastructure for full networking, VPC, and Terraform details.