---
type: redmine-readme
project: Canofy Process Standardization
subject: NSDL SEZ BOE/DTA Lambda pipeline: Phase A (fetch, QC, stage) and Phase B (deliver to Frappe)
story_key: nsdl-lambda-core-pipeline
generated_at: 2026-10-15T16:16:51+05:30
story_type: feature
attachments:
  - file: nsdl-lambda-core-pipeline_20261015-1616__NSDL_SEZ_BOE_Automation_Blueprint.md.attach
    description: Architecture and design rationale this build followed (Blueprint)
  - file: nsdl-lambda-core-pipeline_20261015-1616__NSDL_SEZ_BOE_Automation_BuildSpec_ClaudeCode.md.attach
    description: Phase-by-phase build specification (Phases 0-11)
---

# NSDL SEZ BOE/DTA Lambda pipeline: Phase A (fetch, QC, stage) and Phase B (deliver to Frappe)

> An AWS Lambda pipeline that downloads SEZ trade data from the NSDL portal, validates it per client, and creates the matching BOE / DTA documents with PDF attachments in each client's Frappe/Canofy site.

## 1. Summary
The existing `fw_verify` script did this for one client from a developer machine. This story turns it into a multi-client, scheduled AWS service. **Phase A** logs in to NSDL, exports the Bill of Entry (BE) and DTA Sale reports, routes each row to a client by IEC code, runs six QC checks per client and stages the records in DynamoDB. **Phase B** delivers one client's staged records to that client's Frappe site and attaches the NSDL print PDF. It lives in the repo `canofy-NSDL-data-extraction` (`app/phase_a_fetch`, `app/phase_b_deliver`, `shared`, `template.yaml`) and is deployed to the TEST stack `nsdl-sez-boe-automation-test` in `ap-south-1`. PROD is not deployed.

## 2. Requirement
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | Port the proven `fw_verify` logic instead of rewriting it | user (first message) | Done |
| R2 | Several clients from one NSDL export, routed by IEC; an IEC under two clients is a hard error | Blueprint | Done |
| R3 | Rolling date window computed from the run date, not from config | Build Spec | Done (later changed, see Decisions) |
| R4 | Per-client QC (six checks) and gating before anything is delivered | Build Spec | Done |
| R5 | Phase A stages into DynamoDB and never talks to Frappe | Build Spec | Done |
| R6 | Phase B pushes with each client's own scoped Frappe credentials | Build Spec | Done |
| R7 | Credentials only in AWS Secrets Manager; non-secret constants as Lambda env vars | user | Done |
| R8 | Monitoring: alert topic, dead-letter queue, alarms, stuck-record watchdog | Build Spec Phase 8 | Done (one alert not built, see section 8) |
| R9 | Never invent values the spec marks as human-confirmed (retention days, cron time, credentials) | user | Done |

Requirement that changed: R3 started as a 7-day window ending on the run date; the window later ended the day before the run date (NSDL data lags a day). That change is recorded in `nsdl-aws-deploy-lambda-hardening`.

## 3. Decisions
| Decision | Options considered | Why this one | Date |
|---|---|---|---|
| Config strategy | Config file, Secrets Manager for everything, env vars | Credentials in Secrets Manager; tuning constants as env vars in `template.yaml` | 2026-09-12 |
| Shared code packaging | Lambda Layer, Docker context at repo root | `DockerContext: .` with the Dockerfile in each app folder; no layer to publish | 2026-09-13 |
| Required deploy parameters | Invent defaults, require a value | `S3RetentionDays` and the cron expression are required with no default | 2026-09-13 |
| Secret name | Unsuffixed (spec), suffixed | `nsdl-boe-automation-secrets-<Environment>` so TEST and PROD can coexist | 2026-09-13 |
| Phase B runtime | Spec's 300 s zip function, container image with Playwright | Image with a 900 s timeout, because Phase B performs a real NSDL login and print-to-PDF | 2026-09-13 |
| Phase B query scope | STAGED only (spec), STAGED and PUSHED | A record that is pushed but fails its PDF attach must be retried | 2026-09-13 |
| Phase B trigger | Synchronous API Gateway call, asynchronous | API Gateway REST caps synchronous calls at 29 s; asynchronous returns 202 and Phase B updates the audit record itself | 2026-09-13 |
| Testing | Local only, real AWS | Test against deployed Lambdas and real AWS resources; no dry-run mode exists | 2026-09-13 |

## 4. Logic & Workflow
Phase A (daily, EventBridge): `run_date (default today UTC) -> window -> NSDL login -> export BE and DTA -> validate files -> store in S3 + run control -> parse -> route rows by IEC -> six QC checks per client -> stage as STAGED`

1. The event may carry `run_date` and `force_refresh`; a same-day re-run reuses the stored export unless `force_refresh` is set.
2. Both exports are checked: file exists, is readable as NSDL's tab-delimited text, and has the columns the parser needs.
3. Each row's IEC is normalised (stripped, zero-padded to 10 digits if numeric) and mapped to a client through the reverse map built from the secret. Unmapped IECs are logged and skipped.
4. Records already in a terminal status in DynamoDB are excluded, then six QC checks run per client and pipeline: header count, invoice count, item count, shipment count, sum checks (quantity, duty, assessable value) and IEC consistency within a request.
5. A passing client's records are staged as `STAGED`; strict mode withholds a client whose QC fails.

Phase B (per client, started by an API call or an invoke with `client_name`): `read secret -> query STAGED/PUSHED -> create in Frappe (skip if exists) -> NSDL login -> fetch print PDF -> attach -> mark ATTACHED`

1. Look up the client's Frappe URL and API credentials in the secret.
2. For each record, check Frappe by `request_id` first and create it only if missing; mark the record `PUSHED`.
3. Log in to NSDL once, fetch each pending record's print PDF, attach it, mark `ATTACHED` (a TTL is set only at this terminal status).
4. A failed push or attach leaves the status unchanged, so the next run retries it; there is no permanent-failure state.
5. If a `sync_run_name` was given, update that audit document with the result.

## 5. Files & Components Changed
| File / resource | Type | Change | Purpose |
|---|---|---|---|
| `app/phase_a_fetch/*` | Lambda code | New | Phase A: download, parse, route, QC, stage |
| `app/phase_b_deliver/*` | Lambda code | New | Phase B: deliver and attach |
| `app/watchdog/lambda_handler.py` | Lambda code | New | Hourly scan for records stuck in STAGED over 24 h |
| `shared/` (iec, secrets_loader, json_utils, logger, frappe_client) | Shared code | New | IEC routing, secret loading, Frappe REST client |
| `template.yaml` | SAM template | New | S3, three DynamoDB tables, three functions, API Gateway, SNS, SQS, alarms, secret |
| `events/*.json` | Test events | New | Payloads for manual invokes |
| `README.md` | Docs | New | Secret shape, deploy parameters, testing, monitoring |

## 6. Configuration & Deployment
- **Secret** `nsdl-boe-automation-secrets-<ENV>`: `NSDL_ACCOUNTS.<account>` (username, password, entity id) and `CLIENTS.<client>` (IEC codes, Frappe URL, API key and secret). Values are entered by a human, never committed.
- **Env vars** (names only): `RAW_FILES_BUCKET`, `RUN_CONTROL_TABLE`, `STAGED_RECORDS_TABLE`, `RUN_GATE_TABLE`, `NSDL_BOE_SECRET_NAME`, `NSDL_BASE_URL`, `NSDL_MODULE_BE`, `NSDL_MODULE_DTA`, `NSDL_ACCOUNT_KEY`, `STEP2_QC_STRICT`, `STAGED_TTL_DAYS`, PDF pause settings.
- **Parameters** with no default: `S3RetentionDays`, `DailyScheduleCronExpression`.
- **Deploy:** `sam build` then `sam deploy` (container images). After deploy, populate the secret and subscribe a real endpoint to the alerts topic.

## 7. Testing
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
| Phase A real run | Live NSDL, run date 2026-09-13 | Records staged, QC pass | 64 `eap` records staged: BE 31 requests / 161 invoices / 227 items; DTA 33 / 140 / 210; all QC checks passed | Pass |
| Phase B real run | `eap` TEST Frappe site | All staged records delivered | 64 of 64 delivered; 31 BE PDFs attached | Pass |
| Idempotency stress test | All 64 reset to STAGED, Frappe emptied | No duplicates after a timeout and retry | One Frappe document per request after the first run timed out at 900 s and was retried | Pass |

## 8. Known Limitations & Open Items
- No dry-run mode: any invoke of Phase A logs in to the real NSDL portal, and any Phase B invoke writes to the Frappe site in the secret.
- Blueprint's per-client "Phase B gate fails" alert has no infrastructure; the alerts topic has no subscribers until someone subscribes.
- DynamoDB status is treated as the truth: a Frappe document deleted by hand is not re-created unless its record is reset.
- A full-size batch can exceed the 900 s limit; the batch cap is in `nsdl-pdf-attachment-be-dta`.
- Not confirmed: PROD deployment and the production Frappe deployment of the audit doctype.

## 9. Status
| Item | Value |
|---|---|
| Branch | `main` (initial commit) |
| Commits | `2e9ba74` Initial commit: NSDL SEZ BOE/DTA automation Lambda pipeline (2026-09-15) |
| PR | None (direct initial push) |
| Version | `VERSION` file added later (see `nsdl-github-actions-deployment`) |
| Deployed to | TEST stack `nsdl-sez-boe-automation-test`, `ap-south-1` |

## 10. References
Related stories: `nsdl-aws-deploy-lambda-hardening`, `nsdl-pdf-attachment-be-dta`, `nsdl-frappe-trigger-sync-run`, `nsdl-architecture-documentation`, `nsdl-github-actions-deployment`. Attachments: Blueprint and Build Spec (see front matter).