Project

General

Profile

Download (5.2 KB)

Feature #154 » README.md

README: Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap - Akshath Shettigara, 10/15/2026 04:35 PM

 

type: redmine-readme
project: Canofy Process Standardization
subject: Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap
story_key: nsdl-pdf-attachment-be-dta
generated_at: 2026-10-15T16:16:51+05:30
story_type: feature
attachments:

  • file: nsdl-pdf-attachment-be-dta_20261015-1616__nsdl_browser.py
    description: Phase B browser helpers including the keep-alive pause

Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap

Phase B now fetches the NSDL print PDF and attaches it to every delivered record, for DTA Sale as well as BOE, and limits how many PDFs one Lambda run handles so a large backlog cannot hit the 900 s limit.

1. Summary

The first Phase B version attached a PDF only to BE (FTWZ BOE Import) records and marked DTA (FTWZ DTA Sale) records done as soon as they were created, copying a note in the reference pipeline that DTA has no print PDF. The user questioned this and supplied an updated reference pipeline showing that NSDL's Search Request -> Print flow is the same for DTA. DTA records now go through the same PDF step. A keep-alive pause protects the NSDL session during cooldowns, and a batch cap defers the surplus to the next invocation.

2. Requirement

# Requirement Source Status
R1 DTA Sale documents get the NSDL print PDF attached user (2026-09-15) Done
R2 Pausing between downloads must not let the NSDL session expire updated reference pipeline Done
R3 One run must not exceed the 900 s Lambda limit on a large backlog user: "Yes, tune the pause/batch size to avoid the 900s timeout" Done
R4 A retried or duplicated run must not create duplicate documents idempotency test Done (verified)

3. Decisions

Decision Options considered Why this one Date
Add DTA PDF step Keep DTA without PDF; ask for NSDL selectors; port updated reference The updated reference shows the same print button on both request types 2026-09-15
Handle the cooldown Shorter pause; keep 120 s every 15 downloads with scrolling A shorter pause risks a silent NSDL session expiry mid-batch 2026-09-15
Avoid the 900 s limit Shrink pauses; cap items per invocation The design is resumable, so leftover items stay PUSHED for the next run 2026-09-15
Cap value Not specified by the user 30 per invocation, as an env var default; adjustable 2026-09-15

The cap applies per invocation, and each invocation serves one client, so clients do not share a budget.

4. Logic & Workflow

push record -> status PUSHED -> (batch of up to 30) -> NSDL login -> for each: search request ID -> Print -> PDF -> attach to the record's DocType -> ATTACHED

  1. A pushed BE or DTA record stays PUSHED until its PDF is attached.
  2. Each pending entry carries its DocType, so the attach call targets the right document type.
  3. After every 15 downloads the browser idles for 120 s while scrolling, so NSDL does not log the session out.
  4. At most MAX_PDF_ATTACHES_PER_INVOCATION records are processed per run; the rest are processed by the next run.

5. Files & Components Changed

File / resource Type Change Purpose
app/phase_b_deliver/lambda_handler.py Code DTA goes through the PDF step; DocType in pending entries; batch cap Core change
app/phase_b_deliver/nsdl_browser.py Code Added idle_keepalive_pause Keep session alive
app/phase_b_deliver/pdf_fetch.py, dynamo.py Code Docstrings corrected Remove the wrong DTA claim
template.yaml SAM template MAX_PDF_ATTACHES_PER_INVOCATION: "30" on PhaseBFunction Configurable cap

6. Configuration & Deployment

Setting Value Where
PDF_PAUSE_EVERY 15 Lambda env var
PDF_PAUSE_SECONDS 120 Lambda env var
MAX_PDF_ATTACHES_PER_INVOCATION 30 Lambda env var

Deployed to the TEST stack as a PhaseBFunction-only change.

7. Testing

Test Data / Site Expected Actual Status
Build check Local Builds cleanly Built Pass
DTA PDF flow 3 DTA records reset to PUSHED, TEST Frappe site 3 PDFs attached attached_pdfs: 3, no errors Pass
Full re-run with timeout 64 records, Frappe emptied One document per request 45 PDFs before the 900 s limit; the retry attached the other 19 and created nothing new Pass
Duplicate check on Frappe Same site 1 document per request ID 1 each Pass

8. Known Limitations & Open Items

  • 30 DTA documents delivered before this change have no PDF; the user was asked whether to back-fill them. Answer not confirmed.
  • The AWS CLI retried the long synchronous invoke; later runs used asynchronous invokes.
  • The cap value of 30 is untuned against real backlog sizes. Not confirmed.

9. Status

Item Value
Branch main
Commits 2e9ba74 (initial commit contains this state)
PR None
Deployed to TEST stack, ap-south-1

10. References

Related stories: nsdl-lambda-core-pipeline, nsdl-aws-deploy-lambda-hardening, nsdl-boe-pdf-print-layout (PDF page layout), nsdl-data-parity-dta-be.

(1-1/3)