Actions

Copy link

Feature #154

open

AS AS

Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap

Feature #154: Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap

Added by Akshath Shettigara 7 days ago.

Status:

New

Priority:

Normal

Assignee:

Akshath Shettigara

Target version:

Backlogs

Start date:

10/15/2026

Due date:

% Done:

Estimated time:

Description

Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap¶

Phase B now fetches the NSDL print PDF and attaches it to every delivered record, for DTA Sale as well as BOE, and limits how many PDFs one Lambda run handles so a large backlog cannot hit the 900 s limit.

1. Summary¶

The first Phase B version attached a PDF only to BE (FTWZ BOE Import) records and marked DTA (FTWZ DTA Sale) records done as soon as they were created, copying a note in the reference pipeline that DTA has no print PDF. The user questioned this and supplied an updated reference pipeline showing that NSDL's Search Request -> Print flow is the same for DTA. DTA records now go through the same PDF step. A keep-alive pause protects the NSDL session during cooldowns, and a batch cap defers the surplus to the next invocation.

2. Requirement¶

#	Requirement	Source	Status
R1	DTA Sale documents get the NSDL print PDF attached	user (2026-09-15)	Done
R2	Pausing between downloads must not let the NSDL session expire	updated reference pipeline	Done
R3	One run must not exceed the 900 s Lambda limit on a large backlog	user: "Yes, tune the pause/batch size to avoid the 900s timeout"	Done
R4	A retried or duplicated run must not create duplicate documents	idempotency test	Done (verified)

3. Decisions¶

Decision	Options considered	Why this one	Date
Add DTA PDF step	Keep DTA without PDF; ask for NSDL selectors; port updated reference	The updated reference shows the same print button on both request types	2026-09-15
Handle the cooldown	Shorter pause; keep 120 s every 15 downloads with scrolling	A shorter pause risks a silent NSDL session expiry mid-batch	2026-09-15
Avoid the 900 s limit	Shrink pauses; cap items per invocation	The design is resumable, so leftover items stay `PUSHED` for the next run	2026-09-15
Cap value	Not specified by the user	30 per invocation, as an env var default; adjustable	2026-09-15

The cap applies per invocation, and each invocation serves one client, so clients do not share a budget.

4. Logic & Workflow¶

push record -> status PUSHED -> (batch of up to 30) -> NSDL login -> for each: search request ID -> Print -> PDF -> attach to the record's DocType -> ATTACHED

A pushed BE or DTA record stays PUSHED until its PDF is attached.
Each pending entry carries its DocType, so the attach call targets the right document type.
After every 15 downloads the browser idles for 120 s while scrolling, so NSDL does not log the session out.
At most MAX_PDF_ATTACHES_PER_INVOCATION records are processed per run; the rest are processed by the next run.

5. Files & Components Changed¶

File / resource	Type	Change	Purpose
`app/phase_b_deliver/lambda_handler.py`	Code	DTA goes through the PDF step; DocType in pending entries; batch cap	Core change
`app/phase_b_deliver/nsdl_browser.py`	Code	Added `idle_keepalive_pause`	Keep session alive
`app/phase_b_deliver/pdf_fetch.py`, `dynamo.py`	Code	Docstrings corrected	Remove the wrong DTA claim
`template.yaml`	SAM template	`MAX_PDF_ATTACHES_PER_INVOCATION: "30"` on PhaseBFunction	Configurable cap

6. Configuration & Deployment¶

Setting	Value	Where
`PDF_PAUSE_EVERY`	15	Lambda env var
`PDF_PAUSE_SECONDS`	120	Lambda env var
`MAX_PDF_ATTACHES_PER_INVOCATION`	30	Lambda env var

Deployed to the TEST stack as a PhaseBFunction-only change.

7. Testing¶

Test	Data / Site	Expected	Actual	Status
Build check	Local	Builds cleanly	Built	Pass
DTA PDF flow	3 DTA records reset to `PUSHED`, TEST Frappe site	3 PDFs attached	`attached_pdfs: 3`, no errors	Pass
Full re-run with timeout	64 records, Frappe emptied	One document per request	45 PDFs before the 900 s limit; the retry attached the other 19 and created nothing new	Pass
Duplicate check on Frappe	Same site	1 document per request ID	1 each	Pass

8. Known Limitations & Open Items¶

30 DTA documents delivered before this change have no PDF; the user was asked whether to back-fill them. Answer not confirmed.
The AWS CLI retried the long synchronous invoke; later runs used asynchronous invokes.
The cap value of 30 is untuned against real backlog sizes. Not confirmed.

9. Status¶

Item	Value
Branch	`main`
Commits	`2e9ba74` (initial commit contains this state)
PR	None
Deployed to	TEST stack, `ap-south-1`

10. References¶

Related stories: nsdl-lambda-core-pipeline, nsdl-aws-deploy-lambda-hardening, nsdl-boe-pdf-print-layout (PDF page layout), nsdl-data-parity-dta-be.

Pushed to Redmine on 15 Oct 2026, 16:35 from nsdl-pdf-attachment-be-dta_20261015-1616_20261015-163538.md (README created 15 Oct 2026, 16:35).

Files

Download all files

README.md (5.2 KB) README.md	README: Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap	Akshath Shettigara, 10/15/2026 04:35 PM
nsdl-pdf-attachment-be-dta_20261015-1616__nsdl_browser.py (7.82 KB) nsdl-pdf-attachment-be-dta_20261015-1616__nsdl_browser.py	Phase B browser helpers including the keep-alive pause	Akshath Shettigara, 10/15/2026 04:35 PM
nsdl_browser.py (7.82 KB) nsdl_browser.py	Python script: from __future__ import annotations	Akshath Shettigara, 10/15/2026 04:35 PM

No data to display