Project

General

Profile

Feature #132 » NSDL_SEZ_LOGIN_FIXES_AND_SETUP.md

Framework Setup - Rajesh Gotur, 10/14/2026 10:39 AM

 

NSDL SEZ Automation — Login, OTP, CAPTCHA & BOE/DTA Fixes

Fix report and setup guide

This document explains:

  1. the problems that were reported,
  2. the root cause of each problem,
  3. what was changed to fix it,
  4. how to set up and run the fixed framework,
  5. how the fixes were tested.

Only the login (CAPTCHA + OTP) and the BOE/DTA export handling were changed.
Everything after login (parsing, QC, Frappe document creation, PDF attach) is
unchanged.


1. Problems reported

# Problem
1 The framework picked the wrong / older OTP mail, both when pushing mail from Gmail to the Frappe Communication doctype and when reading the OTP from Communication.
2 The log almost always said "no OTP within 90 sec", and then a Resend loop started.
3 Resend OTP was clicked without checking that the right OTP had been used, and the reason for a failure was not captured.
4 The CAPTCHA was often read wrongly, and the whole login was slow.
5 When no BOE file existed for the IEC and dates, the whole flow stopped with an error. DTA was never tried.

2. Root-cause analysis

The analysis was done by reading the full framework and checking it against
Frappe v15's own email code (frappe/email/doctype/email_account/email_account.py
and frappe/email/receive.py). After that, the problems were reproduced by
running the original code against a mock NSDL portal and a mock Frappe.

2.1 Wrong / older OTP picked (problems 1 and 2)

Root cause Explanation
90-second wait was too short The mail travels NSDL → manager's mailbox → forward → Gmail → Frappe pull. That routinely takes 60–120 s. In the log of 10-Oct, a successful OTP arrived after 82 s, which is just under the limit.
Resend made it worse After 90 s the framework clicked Resend, so NSDL issued a new OTP and the first one became invalid. The first mail then arrived a few seconds later. It was not in the snapshot, so it was treated as "new" and the invalid OTP was submitted. NSDL rejected it, and the cycle repeated until "NSDL rejected 3 OTPs".
"New" was decided by creation time creation is the moment Frappe imported the mail, not when the mail was sent. Frappe's Email Account uses UNSEEN sync and imports every unread mail in Gmail. Every old unread OTP mail was therefore imported with a brand-new creation and looked fresh.
pull_emails can silently do nothing When Frappe's scheduler is already pulling the account, pull_emails returns HTTP 200 with "Emails are already being pulled from this account" and imports nothing. The scheduler job then imported mails after the snapshot was taken, so they looked new.
No date checks at all The mail's real date (communication_date, from the mail's Date header) was never checked. Neither was the original "Sent:" time inside the forwarded mail, or the 10-minute validity. A late forward of an old OTP mail was accepted.

2.2 Resend and missing error reasons (problem 3)

Root cause Explanation
Resend on a timer Resend was clicked because the mail was slow, not because an OTP had failed.
JavaScript alerts lost NSDL shows errors such as Invalid OTP or Invalid Captcha as JS alert() popups. Playwright auto-dismisses alerts when no handler is registered, so the reason was lost. Only a 1,000-character page dump was logged.
Resend result not checked Nothing verified that the Resend click worked. A "limit reached" message, for example, went unnoticed.
OTP page reload misread After a wrong OTP, NSDL reloads the OTP page. The old code saw "still on OTP page" before the error message had loaded.

2.3 CAPTCHA slow and inaccurate (problem 4)

Root cause Explanation
Slow OCR call EasyOCR readtext() runs the heavy text-detection network on 4 image variants each time: about 1.15–1.4 s per read on CPU.
Model loaded late The OCR model (several seconds to load) was loaded only when the first CAPTCHA appeared.
Weak filtering Filters worked on greyscale only, so the grey noise curve and the border went into the OCR. The NSDL letters are dark coloured; the noise is mid-grey.
Detector dropped letters The text detector often missed thin letters such as i, which gave 4-letter readings.
Bad guesses were submitted No CAPTCHA refresh was used, so a doubtful reading was still submitted. Each wrong guess cost a full page reload.
CAPTCHA error not detected The "Invalid Captcha" alert was lost (see 2.2). A wrong password would also have been retried 8 times, which risks locking the account.
Fixed sleeps wait_for_timeout(1500) after every action slowed the whole login.

2.4 Missing BOE file stopped everything (problem 5)

Root cause Explanation
Download timeout raised an error When NSDL has no BOE data, no file is produced. expect_download waited 30 s and raised an exception.
One error stopped both modules That exception broke out of the export loop, so DTA was never exported.
Both files were mandatory Step 1 validation and Step 2 both required the BE and DTA files to exist.

3. What was fixed

3.1 OTP — always the newest, valid OTP

Files: utils/otp_mailbox.py, utils/nsdl_login.py

The OTP request time is now recorded

When Login or Resend OTP is clicked, the exact time is stored. This is
the OTP request time. It is measured on the Frappe server clock, using
the HTTP Date header, so a wrong clock on the PC does not matter.

Before every OTP request (Login / Resend)

1. Pull the mailbox and WAIT until Frappe confirms a real, completed pull
   (if Frappe says "already being pulled", wait and pull again)
   → every old unread mail in Gmail is imported NOW, before the OTP is requested
2. Snapshot the existing OTP Communications (baseline)
3. Record the request time
4. Click Login / Resend

Validations every mail must pass

A mail is used only if all of these are true:

# Validation Stops
1 Not in the baseline snapshot Mails that existed before the request
2 Communication name, mail Message-ID and OTP value never used before Re-using an OTP
3 communication_date (the mail's own date) ≥ request time Old unread mails imported late
4 Original "Sent:" / "Date:" time inside the forwarded mail ≥ request time A new forward of an old NSDL mail
5 OTP is still inside the 10-minute validity Expired OTPs
6 Subject contains SEZOnline Authentication Unrelated mails
7 Exactly one OTP sentence (<OTP> is your One Time Password) Forwards that quote two OTP mails
8 User ID in the mail = NSDL_USERNAME OTPs of other portal users

If several mails pass, the newest is used.

The clock tolerance (default 20 s) is automatically limited to half the gap
between two requests
. So after a Resend, a late mail of the cancelled OTP
can never pass for the new one.

Every rejected mail is written to the log once, with the reason. Example:

[otp] Ignoring mail 'FW: SEZOnline Authentication' (j3b0bmbk27): forwarded copy of an
older NSDL mail (original sent 2026-10-12 15:32:27, OTP request #1 was at 2026-10-12 15:36:27).

3.2 New OTP and Resend flow

OTP page reached
   │
   ▼
wait for a valid OTP mail — up to 9 minutes (never past the 10-min validity)
   │
   ├─ no valid mail at all → STOP with a clear reason
   │                          (lists every mail seen and why it was rejected)
   │
   ▼
check before submit: 6 digits, still valid, still on OTP page
   │
   ▼
fill OTP → verify the value really landed in the box → Submit
   │
   ├─ accepted → home page ✔
   │
   └─ rejected → capture NSDL's reason (alert OR page label) + screenshot
                 │
                 ├─ lockout message?                       → STOP immediately
                 ├─ 3 rejected OTPs?                       → STOP (NSDL locks at 5)
                 ├─ another valid OTP already in mailbox?  → submit that one (no resend)
                 └─ otherwise → wait the 30 s cooldown → click Resend
                                → check the Resend result (limit / block / page lost)
                                → wait for the new OTP again

Key rules:

  • Resend is used only after NSDL rejects an OTP. It is never used just
    because the mail is slow. (Optional: resend_on_timeout: true allows one
    resend when no mail at all arrived within the validity.)
  • Every portal message, including JavaScript alerts, is captured and
    classified as one of: captcha, credentials, otp_invalid,
    otp_expired, resend_limit, lockout, otp_sent.
  • Wrong User ID / password stops at once. It is not retried 8 times, to
    avoid an account lock.
  • The whole login is written to data/logs/<run_date>/login_audit.jsonl:
    request times, which mail was submitted, each portal message and its
    classification.

Example login_audit.jsonl:

{"time": "...", "event": "otp_requested", "sequence": 1, "at": "2026-10-12T10:07:30+00:00"}
{"time": "...", "event": "otp_submitted", "communication": "vh07zwkg1l"}
{"time": "...", "event": "otp_rejected", "kind": "otp_invalid", "portal_message": "Invalid OTP. Please enter the correct OTP"}
{"time": "...", "event": "otp_resend", "sequence": 2, "portal_message": "OTP has been resent successfully", "kind": "otp_sent"}
{"time": "...", "event": "otp_accepted", "sequence": 2, "communication": "x1k9..."}

3.3 CAPTCHA — accurate and fast

File: utils/ocr_engine.py

CAPTCHA image (151 x 63)
   │
   ▼  1. FILTER
   crop grey border
   "darkness" map from the brightest colour channel
      → dark navy / brown / purple letters become black
      → mid-grey noise curves + speckled background become white
   Otsu threshold (with a floor the grey noise cannot reach)
   remove tiny speckles
   │
   ▼  2. RE-SPACE
   cut out every letter and place it on a clean canvas with even gaps
   (5 gap sizes x grey/binary)  +  4 variants of the original layout
   │
   ▼  3. RECOGNISE
   EasyOCR recogniser only (the slow detector is skipped; it also dropped letters)
   OCR model is loaded in the background while the login page opens
   │
   ▼  4. VOTE
   majority vote across all variants → per-character confidence vote
   upper/lower case of c o s u v w x z decided by letter height
   a letter with a dot = i / j
   │
   ▼  5. VALIDATE
   alphanumeric, expected length (5), enough variants agree?
      no  → click the CAPTCHA refresh and read a new image (no wasted login)
      yes → submit
   │
   ▼
   rejected by NSDL? → image saved in captcha_rejected/ (for tuning) → new attempt

Measured result (151 NSDL-style CAPTCHAs: the real sample plus generated
images in the same style):

Before After
Exact reads 50 % 70 %
Reads ignoring upper/lower case 70 % 86 %
Time per read 1.15 s 0.32 s

Other speed-ups:

  • The fixed 1.5 s sleeps were replaced by smart waits that react as soon as
    NSDL answers.
  • The mailbox pull and snapshot happen before the CAPTCHA is read, so the
    CAPTCHA is submitted while it is fresh.

3.4 Missing BOE or DTA — the flow no longer stops

Files: pipeline/step1_download.py, pipeline/step2_transform.py, pipeline/base.py

  • BOE and DTA are now exported independently. A problem in one never
    stops the other.
  • A module counts as "no data" when any of these happens:
    • NSDL shows a "No Records Found" popup or page message;
    • no file arrives within 30 s and NSDL shows no error;
    • the file has only the header row.
  • After download, rows are counted per configured client IEC:
BOE rows for IEC DTA rows for IEC Result
yes yes Both are processed
none yes Logs No BOE records for IEC ... → DTA is processed
yes none Logs No DTA records for IEC ... → BOE is processed
none none Logs There are no BOE and no DTA files/records for IEC ... Nothing to process — stopping here. → run stops cleanly (exit code 0)
  • Step 2 now accepts a missing BOE or DTA file and simply skips that pipeline.
  • A real export failure (navigation error, unexpected portal error) still
    fails Step 1, with a screenshot, after the other module has been tried.

3.5 Files changed

File Change
utils/otp_mailbox.py Rewritten: request-time anchoring, all mail validations, completed-pull check, server clock, time zones
utils/nsdl_login.py Rewritten: alert capture, response detection, OTP / Resend flow, error classification, audit log
utils/ocr_engine.py Rewritten: new filters, re-spacing, fast recognition, voting, validation, background model load
pipeline/step1_download.py Independent BOE / DTA export, no-data detection, per-IEC validation
pipeline/step2_transform.py Accepts a missing BOE or DTA file
pipeline/base.py Clean stop when there is nothing to process
config/config.py, config/*.yaml New settings and their validation
main.py --otp-check now also shows mail date, original sent time and age
requirements.txt Added python-dateutil, tzdata (needed for time zones on Windows)
README.md Updated login / OTP / CAPTCHA / missing-report sections

Not changed: nsdl_parser.py, frappe_client.py, step3_pdf_attach.py,
state/store.py, nsdl_browser.py, json_utils.py, logger.py.


4. Setup guide

4.1 Install

cd NSDL_SEZ_AUTOMATION
python -m venv venv
.\venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m playwright install chromium

tzdata (in requirements.txt) is required on Windows, because Python on
Windows has no time-zone database of its own.

4.2 OTP mailbox (nsdlotp@gmail.com)

  1. Turn on 2-Step Verification for the Gmail account and create an App Password.
  2. Make sure IMAP is enabled in Gmail settings.
  3. Nobody should open this mailbox in Gmail web, the phone app or Outlook.
    Frappe imports only unread mail (UNSEEN), so a mail read there is never imported.

4.3 Forwarding rule (manager's mailbox)

  • Condition: from donotreply-ndmlsupport@ndml.in or subject contains SEZOnline Authentication
  • Action: forward to nsdlotp@gmail.com
  • Use a server-side rule (Outlook "Rules" on Exchange / Microsoft 365), so
    mails are forwarded immediately, even when Outlook is closed.
  • The NSDL user's email ID must be verified on the NSDL portal.

4.4 Frappe Email Account

Open /app/email-account/new:

Field Value
Email Address nsdlotp@gmail.com
Email Account Name e.g. NSDL SEZ OTP → this is OTP_EMAIL_ACCOUNT
Password Gmail App Password
Enable Incoming ✔
Use IMAP ✔
Incoming Server imap.gmail.com, Use SSL ✔
IMAP Folder table one row: INBOX
Append To empty
Email Sync Option UNSEEN

Save, click Pull Emails once, and check /app/communication.

API user: needs read permission on Communication and Email Account
(System Manager has both). Generate the API key and secret from the User record.

4.5 config/config.yaml

Per environment:

environments:
  TEST:
    NSDL_USERNAME: "canofy01"
    NSDL_PASSWORD: "********"
    FRAPPE_URL: "http://127.0.0.1:8000"
    FRAPPE_API_KEY: "xxxxxxxx"
    FRAPPE_API_SECRET: "xxxxxxxx"
    OTP_EMAIL_ACCOUNT: "NSDL SEZ OTP"   # Email Account NAME, not the email address

The old log showed Could not read Email Account 'nsdlotp@gmail.com': not found.
OTP_EMAIL_ACCOUNT must be the Email Account Name (e.g. NSDL SEZ OTP),
not the email address.

New and changed settings under common (the defaults are safe):

  max_login_attempts: 8
  nsdl_response_timeout_seconds: 30   # max wait for NSDL to answer Login / OTP submit
  nsdl_export_timeout_seconds: 30     # max wait for an export file before "no data"
  treat_export_timeout_as_no_data: true

  nsdl_otp:
    otp_validity_seconds: 600         # NSDL OTP validity (10 min)
    wait_seconds: 540                 # was 90 — wait for the mail up to 9 min
    submit_margin_seconds: 30         # don't submit an OTP in its last 30 s
    resend_on_timeout: false          # resend ONLY after NSDL rejects an OTP
    drain_timeout_seconds: 45         # wait for a running Frappe pull to finish
    clock_skew_tolerance_seconds: 20
    verify_forwarded_sent_time: true  # check the original "Sent:" time in forwards
    frappe_time_zone: ""              # "" = read from Frappe, e.g. "Asia/Kolkata"
    mail_time_zone: ""                # zone of the forwarded "Sent:" line ("" = Frappe's)
    resend_cooldown_seconds: 32
    max_resends: 3
    max_submit_attempts: 3

  captcha:
    expected_length: 5                # letters in the NSDL CAPTCHA (null = 4-8)
    min_agreement: 0.34
    max_reads_per_attempt: 4

Unsafe values (for example wait_seconds above the validity, a cooldown under
30 s, or 5+ submit attempts) are rejected when the config loads.

4.6 Check the OTP connection (no NSDL login)

Forward one real NSDL OTP mail to nsdlotp@gmail.com, then run:

python main.py --otp-check --environment TEST

Expected output:

Frappe Email Account 'NSDL SEZ OTP' is reachable and receiving mail.
Latest OTP mail : FW: SEZOnline Authentication
Communication   : 0a1b2c3d4e (imported 2026-10-12 15:47:19)
Mail date       : 2026-10-12T10:15:18+00:00 (UTC)
Original sent   : 2026-10-12T10:15:18+00:00 (UTC)
Age             : 121s
Extracted OTP   : 658204

4.7 Run

Task Command
Offline test (no NSDL, no Frappe writes) python main.py --run-date 2026-09-12 --environment TEST --dry-run
Step 1 only python main.py --run-date 2026-09-12 --environment TEST --steps 1
Step 2 only python main.py --run-date 2026-09-12 --environment TEST --steps 2
Step 3 only python main.py --run-date 2026-09-12 --environment TEST --steps 3
All steps python main.py --run-date 2026-09-12 --environment TEST
Custom date range add --from-date 01/09/2026 --to-date 05/09/2026
Status python main.py --run-date 2026-09-12 --environment TEST --status

Exit codes: 0 = success (including a clean "no BOE/DTA data" stop), 1 = a
step failed validation, 2 = config/setup error or gate refused.

4.8 Where to look when something fails

File Contains
data/logs/<run_date>/run.log Full log, including every ignored mail and the reason
data/logs/<run_date>/login_audit.jsonl Login timeline: OTP request times, submitted mail, portal messages
data/logs/<run_date>/*.png Screenshot at every stop / rejection
data/logs/<run_date>/captcha_rejected/ CAPTCHA images NSDL rejected (file name = what was read)
Message Meaning / action
No valid OTP mail arrived for request #N ... Mails seen but rejected — ... Read the listed reasons. "No new NSDL OTP mail reached Frappe" means: check the forwarding rule, that nobody opens the Gmail inbox, and Email Account → Pull Emails / Error Log.
NSDL rejected N OTPs (last reason: ...) NSDL's own reason is shown; compare login_audit.jsonl with the real mail (--otp-check).
NSDL reports a login block Cooling period — wait 30 minutes.
NSDL refused the OTP resend (limit reached) NSDL resend limit — wait and re-run.
NSDL rejected the User ID / password Fix NSDL_PASSWORD. The login is deliberately not retried.
Forwarded 'Sent:' time ... is after the mail date Set nsdl_otp.mail_time_zone to the manager mailbox's zone.
There are no BOE and no DTA files/records for IEC ... Not an error: nothing to process for those dates.

5. Testing performed

5.1 Test setup

A mock NSDL portal and a mock Frappe were built to behave like the
real ones:

  • Mock NSDL portal: login, CAPTCHA images in NSDL style, OTP page,
    Resend, JS alerts, red notices, trade-data export.
  • Mock Frappe: Email Account, pull_emails (including the "already being
    pulled" case), Communication API, and late background imports.

The real framework code ran against these mocks with Playwright + Chromium.

5.2 Login / OTP scenarios — 15 of 15 passed

# Scenario Expected and verified
1 Normal login 1 OTP submitted, accepted
2 Old unread OTP mails waiting in Gmail Old mails ignored, correct OTP used
3 Old mail imported late by Frappe (after the request) Rejected by mail date
4 New forward of an old NSDL mail Rejected by original "Sent:" time
5 Frappe "already being pulled" + late import Waits for a real pull; old mail rejected
6 Very slow mail (longer than the old 90 s rule) Waits, no resend, accepted
7 Wrong OTP rejected, another valid OTP already waiting Uses it without resend
8 Wrong OTP rejected (alert style) Reason captured → Resend → new OTP accepted
9 Wrong OTP rejected (page-label style) Same as 8
10 No mail ever arrives Clean stop with explanation, no resend
11 Lockout message Stops immediately
12 Resend limit message Captured, stops
13 Wrong password Stops after 1 try (no retry)
14 CAPTCHA rejected twice Image kept, retried, logged in
15 resend_on_timeout: true One resend, new OTP accepted

Before/after proof: scenarios 3, 4 and the slow-mail case were also run on
the original code. It reproduced exactly the reported problem: wrong OTPs
submitted, Resend loop, and finally "NSDL rejected 3 OTPs".

5.3 BOE / DTA scenarios (Step 1 live against the mock + Step 2) — 6 of 6 passed

Scenario Result
BOE "No Records" popup, DTA file DTA processed (55 records)
BOE file, DTA no answer (timeout) BOE processed (32 records)
Both "no records" Clean stop with the "no BOE and no DTA" message
Both files Both processed (87 records)
BOE file with header only, DTA file DTA processed
Files present but no rows for the IEC Clean stop with the IEC message

5.4 Unit checks — 21 of 21 passed

These cover:

  • OTP extraction, including forwarding headers and an ambiguous double OTP;
  • forwarded-time parsing (Outlook and Gmail formats, explicit time zones);
  • rejection of stale and expired OTPs;
  • the minute-precision boundary;
  • config validation of unsafe values.

5.5 Regression

  • Dry run: a full --dry-run of the original and the fixed code gave
    identical results: same 87 request records, same Frappe POST payloads,
    and Step 3 at 87/87.
  • Untouched files: the post-login files are byte-for-byte unchanged.
  • Lint: clean (pyflakes).
  • Clean zip: the packaged zip was extracted fresh and ran successfully.

6. Known limits / things to confirm on the real portal

  • Real-world testing: the fixes were tested against mocks, not the live
    NSDL portal and Gmail, because a live login sends a real OTP to the manager.
    The first live run should be watched, and login_audit.jsonl checked.
  • CAPTCHA upper/lower case: if NSDL's CAPTCHA is not case-sensitive,
    the effective accuracy is the 86 % figure.
  • CAPTCHA refresh icon: if the NSDL page has none, a doubtful reading is
    submitted anyway. A rejection then reloads the page, which is slower but
    still works. A refresh icon can be set with captcha.refresh_selector.
  • Empty BOE/DTA export: the exact way NSDL reports this is handled in all
    common forms (popup, page message, no file within 30 s, header-only file).
  • Frappe without pull_emails (older than v15): mails arrive only on the
    Frappe scheduler interval, so the OTP may take several minutes.
(1-1/3)