type: redmine-readme
project: Canofy Process Standardization
subject: NSDL SEZ BOE/DTA Lambda pipeline: Phase A (fetch, QC, stage) and Phase B (deliver to Frappe)
story_key: nsdl-lambda-core-pipeline
generated_at: 2026-10-15T16:16:51+05:30
story_type: feature
attachments:

• file: nsdl-lambda-core-pipeline_20261015-1616__NSDL_SEZ_BOE_Automation_Blueprint.md.attach
  description: Architecture and design rationale this build followed (Blueprint)
• file: nsdl-lambda-core-pipeline_20261015-1616__NSDL_SEZ_BOE_Automation_BuildSpec_ClaudeCode.md.attach
  description: Phase-by-phase build specification (Phases 0-11)

NSDL SEZ BOE/DTA Lambda pipeline: Phase A (fetch, QC, stage) and Phase B (deliver to Frappe)

An AWS Lambda pipeline that downloads SEZ trade data from the NSDL portal, validates it per client, and creates the matching BOE / DTA documents with PDF attachments in each client's Frappe/Canofy site.

1. Summary

The existing fw_verify script did this for one client from a developer machine. This story turns it into a multi-client, scheduled AWS service. Phase A logs in to NSDL, exports the Bill of Entry (BE) and DTA Sale reports, routes each row to a client by IEC code, runs six QC checks per client and stages the records in DynamoDB. Phase B delivers one client's staged records to that client's Frappe site and attaches the NSDL print PDF. It lives in the repo canofy-NSDL-data-extraction (app/phase_a_fetch, app/phase_b_deliver, shared, template.yaml) and is deployed to the TEST stack nsdl-sez-boe-automation-test in ap-south-1. PROD is not deployed.

2. Requirement

Requirement that changed: R3 started as a 7-day window ending on the run date; the window later ended the day before the run date (NSDL data lags a day). That change is recorded in nsdl-aws-deploy-lambda-hardening.

3. Decisions

4. Logic & Workflow

Phase A (daily, EventBridge): run_date (default today UTC) -> window -> NSDL login -> export BE and DTA -> validate files -> store in S3 + run control -> parse -> route rows by IEC -> six QC checks per client -> stage as STAGED

1. The event may carry run_date and force_refresh; a same-day re-run reuses the stored export unless force_refresh is set.
2. Both exports are checked: file exists, is readable as NSDL's tab-delimited text, and has the columns the parser needs.
3. Each row's IEC is normalised (stripped, zero-padded to 10 digits if numeric) and mapped to a client through the reverse map built from the secret. Unmapped IECs are logged and skipped.
4. Records already in a terminal status in DynamoDB are excluded, then six QC checks run per client and pipeline: header count, invoice count, item count, shipment count, sum checks (quantity, duty, assessable value) and IEC consistency within a request.
5. A passing client's records are staged as STAGED; strict mode withholds a client whose QC fails.

Phase B (per client, started by an API call or an invoke with client_name): read secret -> query STAGED/PUSHED -> create in Frappe (skip if exists) -> NSDL login -> fetch print PDF -> attach -> mark ATTACHED

1. Look up the client's Frappe URL and API credentials in the secret.
2. For each record, check Frappe by request_id first and create it only if missing; mark the record PUSHED.
3. Log in to NSDL once, fetch each pending record's print PDF, attach it, mark ATTACHED (a TTL is set only at this terminal status).
4. A failed push or attach leaves the status unchanged, so the next run retries it; there is no permanent-failure state.
5. If a sync_run_name was given, update that audit document with the result.

5. Files & Components Changed

6. Configuration & Deployment

• Secret nsdl-boe-automation-secrets-<ENV>: NSDL_ACCOUNTS.<account> (username, password, entity id) and CLIENTS.<client> (IEC codes, Frappe URL, API key and secret). Values are entered by a human, never committed.
• Env vars (names only): RAW_FILES_BUCKET, RUN_CONTROL_TABLE, STAGED_RECORDS_TABLE, RUN_GATE_TABLE, NSDL_BOE_SECRET_NAME, NSDL_BASE_URL, NSDL_MODULE_BE, NSDL_MODULE_DTA, NSDL_ACCOUNT_KEY, STEP2_QC_STRICT, STAGED_TTL_DAYS, PDF pause settings.
• Parameters with no default: S3RetentionDays, DailyScheduleCronExpression.
• Deploy: sam build then sam deploy (container images). After deploy, populate the secret and subscribe a real endpoint to the alerts topic.

7. Testing

8. Known Limitations & Open Items

• No dry-run mode: any invoke of Phase A logs in to the real NSDL portal, and any Phase B invoke writes to the Frappe site in the secret.
• Blueprint's per-client "Phase B gate fails" alert has no infrastructure; the alerts topic has no subscribers until someone subscribes.
• DynamoDB status is treated as the truth: a Frappe document deleted by hand is not re-created unless its record is reset.
• A full-size batch can exceed the 900 s limit; the batch cap is in nsdl-pdf-attachment-be-dta.
• Not confirmed: PROD deployment and the production Frappe deployment of the audit doctype.

9. Status

10. References

Related stories: nsdl-aws-deploy-lambda-hardening, nsdl-pdf-attachment-be-dta, nsdl-frappe-trigger-sync-run, nsdl-architecture-documentation, nsdl-github-actions-deployment. Attachments: Blueprint and Build Spec (see front matter).