NSDL SEZ BOE/DTA Automation

What this system does, in one pass

Every SEZ/FTWZ unit's import (Bill of Entry) and domestic-tariff-area sale (DTA Sale) transactions are recorded on NSDL's government SEZ Online portal, one entry at a time, with no bulk export API — only a browser UI. This automation replaces the manual "log in, export, re-key into Frappe, print, re-attach the PDF" routine with two scheduled Lambda functions either side of a DynamoDB staging table.

Architecture at a glance

Every resource in the deployed CloudFormation stack, and how data moves between them.

flowchart LR
    NSDL[["NSDL SEZ Online Portal
(external, no API)"]]

    subgraph AWS["AWS · nsdl-sez-boe-automation-test"]
      direction TB
      EB["EventBridge
cron, daily"] --> PA["PhaseAFunction
Playwright + EasyOCR"]
      PA --> S3["S3 RawFilesBucket
raw BE/DTA exports"]
      PA --> RCT["RunControlTable
per-day idempotency"]
      PA --> SRT[("StagedRecordsTable
per-record status")]
      PA --> RGT["RunGateTable
per-client QC summary"]

      APIGW["API Gateway
/sez/deliver-pending"] --> PB["PhaseBFunction
Playwright + EasyOCR"]
      PB --> SRT
      PB -.async failure.-> DLQ["SQS DLQ"] --> ALM1["CloudWatch Alarm"]

      WD["WatchdogFunction
hourly"] --> SRT
      WD --> SNS(["SNS AlertsTopic"])
      ALM1 --> SNS
      ALM2["CloudWatch Alarm
PhaseA Errors"] --> SNS
      PA -.errors.-> ALM2

      SEC[("Secrets Manager
NSDL creds + per-client Frappe creds")]
      PA --> SEC
      PB --> SEC
    end

    PA <--> NSDL
    PB <--> NSDL
    PB <--> FRAPPE["Client Frappe / Canofy site
FTWZ BOE Import · FTWZ DTA Sale"]
    FRAPPE -- "daily scheduled job" --> APIGW
    SNS --> HUMAN["Email / Slack
(human subscribes post-deploy)"]

    style SRT fill:#9a5f1b22,stroke:#9a5f1b
    style NSDL fill:#1f4e5c22,stroke:#1f4e5c
    style FRAPPE fill:#1f4e5c22,stroke:#1f4e5c
          

A record's life, in one row of StagedRecordsTable

Every NSDL request (identified by its request_id) becomes exactly one row, hashed on request_id with a client_name-status-index GSI for Phase B's per-client queries.

stateDiagram-v2
    [*] --> STAGED: Phase A stages a new,
QC-passed request_id
    STAGED --> QC_MISMATCH: check_6 IEC consistency
failed for this group
    STAGED --> PUSHED: Phase B — Frappe record
created or found (get_record)
    PUSHED --> ATTACHED: Phase B — NSDL PDF
fetched and attached
    STAGED --> STAGED: push failed —
error_type/retry_count set,
status unchanged, retried next run
    PUSHED --> PUSHED: PDF fetch/attach failed —
same retry-in-place pattern
    QC_MISMATCH --> [*]
    ATTACHED --> [*]
          

Phase A — Fetch

app/phase_a_fetch/ · one Lambda, one daily invocation, every client at once.

sequenceDiagram
    participant EB as EventBridge
    participant PA as PhaseAFunction
    participant RCT as RunControlTable
    participant NSDL as NSDL Portal
    participant S3 as S3
    participant Parse as Parser + QC
    participant SRT as StagedRecordsTable
    participant RGT as RunGateTable

    EB->>PA: cron daily · {force_refresh:false}
    PA->>RCT: get_run_control(run_date)
    alt already PASSED today
        RCT-->>PA: reuse existing S3 files
    else not yet run
        PA->>NSDL: login (Playwright + EasyOCR CAPTCHA)
        PA->>NSDL: export BILL OF ENTRY + DTA SALE
        PA->>S3: store raw exports
        PA->>RCT: mark PASSED
    end
    PA->>Parse: read_nsdl_export, partition_by_client(IEC map)
    Parse->>Parse: 6 QC checks per client per pipeline
    Parse->>SRT: put_staged_record (new request_ids only)
    Parse->>RGT: put_run_gate (per-client QC summary)
          

Trigger & the date window

EventBridge's Schedule target can only carry a static Input, so it always sends {"force_refresh": false} — no run_date. lambda_handler.py defaults it to today's UTC date, then computes the actual NSDL query window:

to_date   = run_date - 1 day      # NSDL's own data lags by one day
from_date = to_date - 7 days       # rolling 7-day window
# entered into NSDL's date fields as DD/MM/YYYY

Login & export

nsdl_download.py drives a real headless Chromium session: fills username/password, solves the CAPTCHA via ocr_engine.py (four OpenCV preprocessing variants fed to EasyOCR, confidence-weighted character-by-character consensus across variants), retries login up to MAX_LOGIN_ATTEMPTS times, then exports both modules (NSDL_MODULE_BE=17, NSDL_MODULE_DTA=26) to tab-delimited .xls files.

Parse, partition, six QC checks

nsdl_parser.py groups rows by REQUEST ID, resolves each row's CLIENT IEC against the secret's reverse IEC→client map (an IEC mapped to two different clients aborts the entire run — the one error class this pipeline never resolves silently), and runs six checks per pipeline per client:

A failed check 6 group is staged with status QC_MISMATCH and excluded from the batch. Any other failing check either aborts staging for that client/pipeline (STEP2_QC_STRICT=true, the default) or stages anyway with a logged warning.

Phase B — Deliver

app/phase_b_deliver/ · one Lambda, invoked independently per client.

sequenceDiagram
    participant FR as Client Frappe site
    participant NSR as NSDL Sync Run doc
    participant APIGW as API Gateway
    participant PB as PhaseBFunction
    participant SRT as StagedRecordsTable
    participant CAN as Canofy (Frappe API)
    participant NSDL as NSDL Portal

    FR->>NSR: insert(status=Triggered)
    FR->>APIGW: POST /sez/deliver-pending
X-Frappe-Client, {sync_run_name}
    APIGW->>PB: invoke — Event (async), 202 immediately
    PB->>SRT: query_actionable_by_client (STAGED + PUSHED)
    loop each STAGED record
        PB->>CAN: get_record → found? reuse : create_record
        PB->>SRT: mark_pushed(frappe_docname)
    end
    PB->>NSDL: login (once for the whole batch)
    loop pending PDFs, capped at MAX_PDF_ATTACHES_PER_INVOCATION
        PB->>NSDL: fetch print-PDF for request_id
        PB->>CAN: attach_pdf
        PB->>SRT: mark_attached(ttl)
        Note over PB,NSDL: every 15th download: 120s
keep-alive-scroll pause
    end
    PB->>NSR: update_audit_doc(status, counts)
          

Why the call is asynchronous

API Gateway's REST integration caps a synchronous Lambda proxy at 29 seconds. A real NSDL login plus throttled PDF downloads routinely runs minutes. The integration sets X-Amz-Invocation-Type: Event, so Frappe's scheduled job gets a bare 202 {"accepted": true} the instant AWS accepts the trigger — the actual delivery result arrives later, written back onto the same NSDL Sync Run document.

Idempotent push to Frappe

Before creating anything, Phase B calls frappe.get_record(doctype, request_id). Only if Frappe has nothing for that request_id does it call create_record. This is the only check that touches Frappe's real state — DynamoDB's own status is what decides whether Phase B even looks at a record at all.

PDF fetch & attach — one path for BE and DTA

pdf_fetch.py drives Search Request → request_id → Print on NSDL and renders the result to PDF via the browser's own print-to-PDF. BE and DTA requests use the identical navigation — an earlier build assumed DTA had no printable document at all; that assumption was wrong and has since been corrected (see Hardening).

Every PDF_PAUSE_EVERY (15) downloads, the batch pauses PDF_PAUSE_SECONDS (120s) — not idle, but scrolling the page gently, because a fully idle tab was observed tripping NSDL's own session timeout mid-run. A hard MAX_PDF_ATTACHES_PER_INVOCATION (30) cap keeps one invocation inside Lambda's 900s ceiling on a large backlog; anything past the cap stays PUSHED for the next invocation.

Watchdog

app/watchdog/ · a plain zip Lambda, no browser needed.

Every hour, scans StagedRecordsTable for rows still STAGED more than 24 hours after staged_at, groups the count by client, and publishes one SNS message naming every affected client. One check, two real failure modes: a client's Frappe scheduler has silently stopped calling Phase B at all, or Phase B is repeatedly failing to push a specific record (which — by design — never changes that record's status, so it looks identical to "nobody's asking" from here).

The Frappe side

What lives in the client's own Canofy/Frappe app (canofy_tcs), not in this AWS stack.

flowchart LR
    HOOKS["hooks.py
scheduler_events.daily"] --> JOB["trigger_nsdl_sync()"]
    JOB --> CFG[("API Configuration - Canofy
doc: NSDL SEZ DELIVER PENDING")]
    CFG -.reads url / api_key / api_secret
+ X-Frappe-Client header.-> JOB
    JOB --> NSR["NSDL Sync Run
insert(status=Triggered)"]
    JOB --> POST["make_post_request()"]
    POST --> AWS["AWS API Gateway"]
    AWS -. later, async .-> NSR2["Phase B updates the
same NSDL Sync Run doc"]
          

Deliberately not routed through the existing Canofy API Call auto-dispatch controller — that controller only ever fires for API configs whose name/service contains "EWAY" (hardcoded), so it would silently do nothing for an NSDL config without editing shared, already-in-production code. nsdl_sync_scheduled_job.py instead reuses the two genuinely generic pieces: frappe.integrations.utils.make_post_request and the API Configuration - Canofy doctype as a config store.

Configuration reference

Every value that shapes this system's behavior, and where it lives.

CloudFormation parameters — human-confirmed, never guessed

PhaseAFunction environment

PhaseBFunction environment

Secrets Manager shape — nsdl-boe-automation-secrets-{Environment}

{
  "NSDL_ACCOUNTS": {
    "siddhartha_ftwz": {
      "NSDL_USERNAME": "",
      "NSDL_PASSWORD": "",
      "entity_id": ""            // NSDL's own SEZ-unit ID, shown per-row in every export
    }
  },
  "CLIENTS": {
    "eap": {
      "client_iec_codes": ["300014953", "0300014953"],
      "FRAPPE_URL": "https://eapx.canofy.tech",
      "api_key": "",
      "api_secret": ""
    }
  }
}

Real credentials are never generated or guessed by this codebase — populated post-deploy via aws secretsmanager put-secret-value. An IEC listed under two clients aborts the whole Phase A run rather than resolving silently.

Hardening notes — Docker-local vs. real Lambda

Every one of these passed cleanly under sam local invoke and then failed the first time it ran as the actually-deployed Lambda — the gap between a normal Docker container and AWS Lambda's sandboxed execution environment. Fixed and verified against real invocations, in the order found.

Setup checklist — what's done, what's left

1. 1
   Deploy the AWS stack donesam build && sam deploy with S3RetentionDays and DailyScheduleCronExpression supplied.
2. 2
   Populate real secrets doneNSDL TEST credentials + the eap client's Frappe URL/API key/secret, via put-secret-value.
3. 3
   Verify Phase A + Phase B end-to-end doneReal NSDL login, real export, real Frappe push, real PDF attach — all verified against the actual deployed Lambdas, not just local invokes.
4. 4
   Add NSDL Sync Run + register the scheduled job done — local dev benchMigrated onto the local frappe.local dev site. Not yet deployed to whatever actually hosts eapx.canofy.tech in production.
5. 5
   Deploy those same file changes to production Frappe outstandingReal production hosting path for canofy_tcs is not yet identified from this session.
6. 6
   Create the API Configuration - Canofy record outstandingNamed NSDL SEZ DELIVER PENDING, with the real deployed API URL, the real API Gateway key value, and {"X-Frappe-Client": "eap"} in its headers field.
7. 7
   Subscribe to AlertsTopic outstandingNobody receives Watchdog/error alarms until a real email or Slack endpoint is subscribed.

Known gaps, worth deciding on deliberately