Feature #152
openPort the updated NSDL login, OTP mailbox and CAPTCHA engine, and treat missing BOE/DTA data as normal
0%
Description
Port the updated NSDL login, OTP mailbox and CAPTCHA engine, and treat missing BOE/DTA data as normal¶
The Lambdas now use the latest login, OTP and CAPTCHA logic from the reference project, and a module with no data in the window no longer stops the run.
1. Summary¶
After the first live runs for the new wud client, three problems showed up: late OTP mails invalidated by an early resend, logins rejected without a recorded reason, and a Bill of Entry export that timed out when NSDL had no data. The user supplied updated reference files for the login, OTP mailbox and CAPTCHA engine. They were ported into shared/ and into both Lambdas, and the download and transform steps now treat "NSDL sent no file" as a normal result. Verified live on the TEST stack on 2026-10-12.
2. Requirement¶
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | Review the reference changes to CAPTCHA, login and OTP and update the Lambdas | user (2026-10-12) | Done |
| R2 | Never submit an OTP from a mail that arrived for an older, already invalidated request | observed failures | Done |
| R3 | Do not resend just because a mail is slow | observed failures | Done |
| R4 | Log every NSDL portal message on a failed login | observed failure | Done |
| R5 | A module with no data in the window must not stop the other module | observed failure and user's explanation | Done |
| R6 | Reference config.yaml setting wait_seconds = 540 maps to a Lambda setting |
user's change list | Done |
3. Decisions¶
| Decision | Options considered | Why this one | Date |
|---|---|---|---|
| Source of the changes | First zip; the "NSDL" folder | The first zip was byte-for-byte the earlier one; the folder held the changes | 2026-10-12 |
| Fresh mail rule | Newest mail; sent after the specific Login or Resend click | A late mail of an old OTP was being submitted and rejected | 2026-10-12 |
| Resend policy | Resend after a short wait; never resend because mail is slow | Each resend invalidates the earlier OTP | 2026-10-12 |
| OTP wait | 90 s, 240 s, 540 s | Reference setting; capped by the OTP's 10-minute validity | 2026-10-12 |
| Empty export | Timeout error; "no data" | NSDL sends no file when a module has no requests | 2026-10-12 |
| Not ported |
main.py of the reference |
Only affects the reference's command-line tool | 2026-10-12 |
| Local OCR test | Run locally; skip | The local Python had no OpenCV build; the real test is a Lambda run | 2026-10-12 |
4. Logic & Workflow¶
Login: fill credentials -> read CAPTCHA -> confident? submit : request a new image -> click Login (anchor time) -> wait for a mail sent after the anchor -> submit the OTP once -> result classified and logged
- The CAPTCHA engine reads cleaned variants, votes per character and accepts only a confident 5-character reading; otherwise it asks NSDL for a new image.
- A mail is accepted only if both its own date and the original "Sent:" time inside a forwarded mail are after the click. Rejected mails are logged with the reason.
- The wait lasts up to the configured seconds and never triggers a resend on its own.
- Credential rejection and lockout pages stop the run at once. Every portal message is classified and logged, and an audit file and screenshots are written.
Export: BE and DTA are exported independently. If NSDL sends no file for a module, that module gets no S3 key, the run continues, and the result shows a status per module (exports).
5. Files & Components Changed¶
| File / resource | Type | Change | Purpose |
|---|---|---|---|
shared/otp_mailbox.py |
Code | Freshness rules, one-time use, completed-pull confirmation | Correct OTP pickup |
shared/nsdl_login.py |
Code | New login class, message classification, settings validation | Login |
app/phase_a_fetch/ocr_engine.py, app/phase_b_deliver/ocr_engine.py
|
Code | New CAPTCHA engine; /tmp model directory kept |
CAPTCHA |
app/phase_a_fetch/nsdl_download.py, lambda_handler.py, transform.py, dynamo.py
|
Code | Per-module export and "no data" flow | Missing data |
app/phase_a_fetch/requirements.txt, app/phase_b_deliver/requirements.txt
|
Config | Add tzdata, python-dateutil
|
Time zone handling |
template.yaml |
SAM template | New settings, remove MAX_CAPTCHA_READ_ATTEMPTS
|
Configuration |
README.md, architecture document |
Docs | Updated | Documentation |
6. Configuration & Deployment¶
| Setting | Value | Notes |
|---|---|---|
NSDL_OTP_WAIT_SECONDS |
540 | Capped by the OTP validity (10 minutes) |
NSDL_OTP_RESEND_ON_TIMEOUT |
false | No resend when mail is slow |
NSDL_CAPTCHA_EXPECTED_LENGTH |
5 | Only a 5-character reading is submitted |
TREAT_EXPORT_TIMEOUT_AS_NO_DATA |
true | An export timeout is treated as no data |
NSDL_OTP_MAX_RESENDS, NSDL_OTP_MAX_SUBMIT_ATTEMPTS
|
1, 2 | Unchanged from nsdl-email-otp-login
|
Both images rebuild because the requirements changed. Deployed through the pipeline (nsdl-github-actions-deployment). With a 540 s wait, Phase B (login, wait and 30 PDFs) can exceed the 900 s limit if mail is very slow; lowering MAX_PDF_ATTACHES_PER_INVOCATION is the stated remedy.
7. Testing¶
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
| Local OTP, mailbox and config checks | Replay of the 2026-10-12 morning timings | Late and stale mails rejected, config errors named | 30 checks passed | Pass |
Ruff, sam validate --lint, compile |
Repo | Clean | Clean | Pass |
| Local CAPTCHA engine | Local Python | Run | Not run (no OpenCV build for the local Python) | Not tested locally |
| Live Phase A, 2026-10-12 | Real NSDL, TEST stack | Login, export, QC | CAPTCHA accepted on the second image; OTP after 109 s, no resend; BE "no data" and the run continued; DTA 516 rows; eap 43 and wud 3 DTA staged; all QC passed |
Pass |
| First attempt of that run |
eapx Frappe site |
Reachable | No answer within 30 s, before NSDL was contacted | Fail, transient |
| Phase B after the update |
wud, eap
|
Records and PDFs |
wud 6 and 6 PDFs; eap 43 records, no PDFs because no OTP mail arrived within 9 minutes |
Partial |
8. Known Limitations & Open Items¶
- A second login about 2.5 minutes after the first on the same NSDL account got no OTP mail. Cause not confirmed: NSDL throttling or the mail forwarding path.
- The Frappe site used for the OTP mailbox may time out on slow responses (see
nsdl-otp-mailbox-resilience). - The CAPTCHA engine was verified only on the deployed Lambda.
- The reference folders extracted during the work contain logs and a
config.yaml; they were not read and sit outside the repo. Deleting them is left to the user.
9. Status¶
| Item | Value |
|---|---|
| Branch | feature/AI-TBD-nsdl-otp-login |
| Commits | 25d6ea0 |
| PR | Created and merged by the user; number not confirmed |
| Version |
1.1.0-beta (a bump was offered; not confirmed) |
| Deployed to | TEST stack |
10. References¶
Related stories: nsdl-email-otp-login, nsdl-otp-mailbox-resilience, nsdl-data-parity-dta-be.
Pushed to Redmine on 15 Oct 2026, 16:35 from nsdl-login-otp-captcha-update_20261015-1616_20261015-163536.md (README created 15 Oct 2026, 16:35).
Files
No data to display