Feature #132
openUpdating the Login Automation and Captcha Handling as per new guidelines of NSDL WEBSITE
Description
User Story: Update NSDL SEZ Automation for Mandatory 2FA, CAPTCHA Handling, and Reliable Login¶
Project: NSDL SEZ Automation
Work type: Enhancement / Change Request
Status: Implemented; mock-based verification documented
Primary change: Adapt the existing automation to the SEZ Online portal's updated login flow, which now requires two-factor authentication (2FA).
1. User Story¶
As an operations user running the NSDL SEZ automation,
I want the existing automation to handle the updated SEZ Online login process—including CAPTCHA and email OTP verification—reliably,
so that the automation can continue downloading Bill of Entry (BOE) and DTA Sale data and complete its existing Frappe processing workflow without being blocked by the new authentication steps.
Business context¶
The automation had already been developed to access SEZ Online, download BOE and DTA Sale data, transform the downloaded data, create the relevant Frappe records, and attach BOE PDFs.
The portal's login process changed. The previous login implementation was no longer sufficient because a successful login now required an additional OTP verification step. The automation therefore needed an updated login flow, a dependable way to retrieve OTP emails, and more robust CAPTCHA recognition and error handling.
The changes were designed to update the login and report-download reliability while preserving the existing downstream parsing, quality checks, Frappe document creation, and PDF attachment behavior.
2. Why This Change Was Required¶
Official portal change¶
The official SEZ Online website displays a notice that the system has migrated to a 2FA-based login process. NDML's official System Notice No. NDML/SEZ Online/2026/88, dated 16 September 2026, states that login based on two-factor authentication would become mandatory effective 23 September 2026. The notice also asked users to verify their registered mobile numbers and email addresses.
The official SEZ Online Release Note v3.0 (May 2026), under Important Notice No. 86, describes the login flow as User ID + Password + CAPTCHA, followed by an OTP sent to the user's verified mobile number and email address. It states that the OTP is valid for 10 minutes, resend is available after a 30-second wait, and five incorrect OTP attempts can cause a 30-minute login restriction.
Official references
- SEZ Online official login page
- Official Important Notices page
- System Notice No. 88 — Two-Factor Authentication (PDF)
- Release Note v3.0 — 2FA feature details (PDF)
Source distinction: The mandatory 2FA timeline and portal behavior above come from official SEZ Online / NDML material. The implementation details and test results below come from the supplied automation code and its fix report.
3. Existing Workflow and Updated Workflow¶
Previous login concept¶
Open SEZ Online
↓
Enter User ID + Password
↓
Read and enter CAPTCHA
↓
Login
↓
Continue to report download
This flow did not adequately handle the new OTP verification stage.
Updated login concept¶
flowchart TD
A[Open SEZ Online] --> B[Enter User ID and Password]
B --> C[Read CAPTCHA]
C --> D{CAPTCHA reading valid?}
D -- No --> E[Refresh CAPTCHA and read again]
E --> C
D -- Yes --> F[Prepare OTP mailbox baseline]
F --> G[Submit login]
G --> H{Portal response}
H -- CAPTCHA rejected --> E
H -- Credentials rejected --> I[Stop and report error]
H -- OTP page --> J[Wait for a fresh, valid OTP email]
J --> K[Validate OTP age, request time, user ID and mail]
K --> L[Submit OTP]
L --> M{Login accepted?}
M -- Yes --> N[Continue to requested NSDL task]
M -- No --> O[Capture portal reason and audit event]
O --> P{Lockout, resend limit or credentials issue?}
P -- Yes --> I
P -- No --> Q[Use another valid OTP or resend under controlled rules]
Q --> J
The updated implementation uses a shared login module so the relevant pipeline steps use the same CAPTCHA, OTP, and portal-response handling logic.
4. What Was Implemented¶
4.1 Frappe Email Account for OTP retrieval¶
A Frappe Email Account was configured to receive forwarded NSDL OTP emails. The automation reads the received email through Frappe's Communication records rather than relying on a person to copy the OTP manually.
Email route¶
flowchart LR
A[SEZ Online sends OTP email] --> B[Registered / manager mailbox]
B --> C[Server-side forwarding rule]
C --> D[Dedicated OTP mailbox]
D --> E[Frappe Email Account pulls mail]
E --> F[Communication record]
F --> G[Automation reads and validates OTP]
G --> H[OTP entered into SEZ Online]
Configuration responsibilities¶
- Configure a dedicated OTP mailbox and enable IMAP access.
- Configure a Frappe Email Account with incoming mail enabled.
- Set the Email Sync Option to
UNSEEN, as required by the implementation. - Configure server-side forwarding for the relevant SEZ Online authentication email.
- Give the automation's Frappe API user the required read access to Email Account and Communication.
- Set
OTP_EMAIL_ACCOUNTto the Frappe Email Account Name, not merely the email address.
Credentials and tokens must remain in protected configuration and must not be committed to source control or written into the story.
4.2 Correct OTP selection and freshness validation¶
The earlier implementation could pick an old OTP or time out before the email arrived. A simple Frappe record creation timestamp was not enough to prove that the underlying email was new, because Frappe may import an older unread email later.
The updated implementation anchors each OTP to the login/resend request and validates the email before submitting its OTP.
| Validation | Purpose |
|---|---|
| Mail is not present in the pre-request baseline | Excludes emails already known before the request |
| Request time is recorded | Establishes when the portal OTP was requested |
| Communication's email date is checked | Prevents a late-imported old email from appearing fresh |
Original forwarded Sent: / Date: time is checked |
Helps reject a new forward of an old OTP message |
| OTP remains within the 10-minute validity window | Avoids submitting expired OTPs |
| Expected authentication subject is present | Excludes unrelated emails |
| Exactly one valid OTP sentence is present | Avoids ambiguous forwards containing multiple OTP messages |
OTP email's User ID matches configured NSDL_USERNAME
|
Avoids using an OTP intended for another portal user |
| Previously used mail/message/OTP values are rejected | Prevents accidental OTP reuse |
If multiple emails pass the checks, the newest valid OTP is selected.
4.3 Updated OTP wait, submit, and resend logic¶
The OTP wait was increased from 90 seconds to up to 9 minutes (540 seconds), while remaining inside the portal's documented 10-minute OTP validity period.
The revised behavior is designed to avoid requesting a replacement OTP just because an email is delayed.
Request OTP
↓
Wait for fresh OTP email (up to configured limit)
↓
Validate the mail and OTP
↓
Submit OTP
↓
┌───────────────────────────────┐
│ Accepted? │
└───────────────┬───────────────┘
Yes │ No
↓ │ ↓
Continue │ Capture actual error
│ ↓
│ Check lockout / limit
│ ↓
│ Use another valid OTP if available
│ ↓
│ Otherwise resend only under allowed rules
└──────────────────────────
Additional controls include:
- Resend is not triggered simply because email delivery is slow.
- Portal responses are checked after login, OTP submission, and resend.
- OTP rejection reasons are captured from JavaScript alerts or page messages.
- Wrong credentials stop the login instead of triggering repeated attempts.
- Lockout and resend-limit messages cause a controlled stop.
- OTP request, submission, rejection, and acceptance events are recorded in
login_audit.jsonl.
4.4 New CAPTCHA recognition and handling¶
The CAPTCHA handler was redesigned because the previous OCR flow was both slow and prone to incorrect readings.
The revised logic in utils/ocr_engine.py uses a multi-stage process:
flowchart TD
A[Capture CAPTCHA image] --> B[Crop border and reduce grey noise]
B --> C[Create filtered / thresholded variants]
C --> D[Re-space characters and prepare variants]
D --> E[Run OCR recognition]
E --> F[Compare readings and vote per character]
F --> G{Length and agreement acceptable?}
G -- No --> H[Refresh CAPTCHA and read again]
H --> A
G -- Yes --> I[Submit CAPTCHA with login]
I --> J{Portal accepts CAPTCHA?}
J -- Yes --> K[Continue login]
J -- No --> L[Record rejection and retry with a fresh CAPTCHA]
L --> A
The new handling includes:
- Color-aware filtering to separate dark letters from grey background noise.
- Border cropping and removal of small speckles.
- Multiple image variants to improve recognition.
- Character re-spacing to help recognition of thin or closely spaced letters.
- OCR recognition with a voting strategy across variants.
- Validation of expected character length and agreement before submission.
- CAPTCHA refresh when the reading is uncertain, where the page exposes a usable refresh control.
- Saving rejected CAPTCHA samples for later investigation.
- Loading the OCR model in the background and replacing fixed delays with response-aware waits where possible.
Measured OCR comparison in the supplied test set¶
The fix report describes testing across 151 NSDL-style CAPTCHA images, combining a real sample with generated images in the same style.
| Measure | Before | After |
|---|---|---|
| Exact reads | 50% | 70% |
| Reads when case is ignored | 70% | 86% |
| Time per read | 1.15 seconds | 0.32 seconds |
These are results from the documented test set, not an official NSDL accuracy guarantee. The portal's displayed CAPTCHA is case-sensitive, so exact-case recognition remains important.
4.5 Portal errors, retries, and auditability¶
The updated login module distinguishes portal responses instead of treating every failed login as the same error.
| Response category | Intended handling |
|---|---|
| Invalid CAPTCHA | Capture the reason and retry with a fresh CAPTCHA |
| Invalid credentials | Stop immediately; do not repeatedly retry the password |
| Invalid or expired OTP | Record the rejection and follow the controlled OTP flow |
| OTP sent / resend successful | Continue waiting for a new valid OTP |
| Resend limit reached | Stop and report the portal's response |
| Lockout / cooling period | Stop immediately and preserve diagnostic evidence |
Screenshots and structured audit events help explain why a login attempt failed.
5. Related Report-Download Reliability Fix¶
Alongside the login changes, the BOE and DTA export handling was updated so that missing data in one report does not prevent the other report from being attempted.
Previous behavior¶
Attempt BOE export
↓
No BOE file / timeout
↓
Exception stops the export step
↓
DTA export is not attempted
Updated behavior¶
flowchart TD
A[Start report export] --> B[Attempt BOE export]
A --> C[Attempt DTA export]
B --> D[Classify BOE result]
C --> E[Classify DTA result]
D --> F[Validate records for configured IEC]
E --> G[Validate records for configured IEC]
F --> H[Process available BOE data]
G --> I[Process available DTA data]
H --> J{Any relevant data available?}
I --> J
J -- Yes --> K[Continue with available pipeline(s)]
J -- No --> L[Stop cleanly with a no-data message]
The implementation treats common no-data situations as a no-data result, including a portal “No Records Found” message, a download timeout without a portal error, or a file containing only a header row. A genuine portal/navigation failure remains an error and is logged.
This allows the automation to process whichever dataset is available for the configured IEC and date range.
6. Scope of Change¶
Updated modules¶
| File / area | Responsibility |
|---|---|
utils/nsdl_login.py |
Shared login flow, CAPTCHA/OTP transitions, portal message classification, resend handling, audit trail |
utils/otp_mailbox.py |
Frappe Email Account access, mailbox pull/snapshot, OTP extraction and freshness checks |
utils/ocr_engine.py |
CAPTCHA image processing, OCR voting, validation and retry support |
pipeline/step1_download.py |
Independent BOE/DTA exports and no-data detection |
pipeline/step2_transform.py |
Skips a missing BOE or DTA pipeline where appropriate |
pipeline/base.py |
Clean stop when no relevant BOE/DTA records exist |
config/config.py, config/*.yaml
|
OTP, CAPTCHA, wait, retry and validation settings |
main.py |
OTP mailbox check command and pipeline execution options |
requirements.txt |
Additional date/time-zone dependencies used by the implementation |
README.md |
Updated setup and troubleshooting documentation |
Preserved downstream behavior¶
The supplied fix report states that the following components were not changed:
pipeline/nsdl_parser.pypipeline/frappe_client.pypipeline/step3_pdf_attach.pystate/store.pyutils/nsdl_browser.pyutils/json_utils.pyutils/logger.py
The change was therefore focused on authentication, CAPTCHA/OTP handling, and BOE/DTA export robustness—not a redesign of the complete downstream pipeline.
7. End-to-End Automation Context¶
The login update supports the existing three-step automation:
flowchart TD
A[Step 1: Sign in to SEZ Online] --> B[CAPTCHA + email OTP / 2FA]
B --> C[Download BOE and DTA reports]
C --> D{Step 1 validation passes?}
D -- No --> X[Stop and log reason]
D -- Yes --> E[Step 2: Parse, filter, QC and transform]
E --> F[Create Frappe documents]
F --> G{Step 2 validation passes?}
G -- No --> X
G -- Yes --> H[Step 3: Retrieve BOE PDFs from NSDL]
H --> I[Attach PDFs to corresponding Frappe records]
I --> J[Record results and completion state]
The framework's validation gates and persisted run state remain in place. Later steps require earlier steps to have passed their validation requirements.
8. Acceptance Criteria¶
| ID | Acceptance criterion | Evidence / status |
|---|---|---|
| AC-01 | Login supports User ID, password, CAPTCHA, and the new OTP stage | Implemented in shared login module |
| AC-02 | OTP is retrieved from Frappe Email Account / Communication | Implemented; mailbox check is available |
| AC-03 | Old, expired, reused, or mismatched OTP emails are rejected | Covered by OTP validation logic and unit checks |
| AC-04 | OTP waiting respects the portal validity window | Configured to wait up to 540 seconds within the 600-second validity |
| AC-05 | Resend does not occur just because mail delivery is slow | Implemented in the controlled resend flow |
| AC-06 | Portal alerts and visible error messages are captured | Implemented in login handling and audit logs |
| AC-07 | CAPTCHA recognition uses preprocessing, multiple readings, and validation | Implemented in OCR engine |
| AC-08 | CAPTCHA rejection can trigger a fresh CAPTCHA attempt | Implemented where a usable refresh control is available |
| AC-09 | Invalid credentials and lockout messages stop safely | Implemented in response classification |
| AC-10 | BOE and DTA exports are attempted independently | Implemented in Step 1 |
| AC-11 | A missing BOE report does not prevent processing available DTA data, and vice versa | Covered by mock scenarios |
| AC-12 | Existing parsing, QC, Frappe creation, and PDF attachment logic remains unchanged | Stated in the supplied fix report |
| AC-13 | Failures leave useful logs and screenshots for diagnosis | Login audit, run log and diagnostic artifacts are supported |
9. Verification and Test Results¶
The supplied fix report documents tests against a mock NSDL portal and mock Frappe service, using the real framework code with Playwright and Chromium.
| Test group | Result reported |
|---|---|
| Login / OTP scenarios | 15 of 15 passed |
| BOE / DTA scenarios | 6 of 6 passed |
| Unit checks | 21 of 21 passed |
| Full dry-run regression | Same 87 request records and same Frappe POST payloads; Step 3 reported 87/87 |
Scenarios included delayed email delivery, old unread OTP messages, late imports, forwarded copies of old OTP emails, OTP rejection, resend behavior, lockout responses, wrong passwords, CAPTCHA rejection, missing BOE/DTA data, and no relevant records for the configured IEC.
Important verification limitation¶
The supplied report explicitly states that the login fixes were tested against mocks, not a live NSDL portal and live mailbox. Therefore:
- The documented automated test results should not be described as proof of a successful live production login.
- A controlled live verification should confirm the registered email forwarding, Frappe Email Account, actual portal messages, and current CAPTCHA behavior.
- Any live test should be supervised and should avoid repeated attempts that could trigger the portal's lockout policy.
10. Operational Setup Dependencies¶
The updated login depends on the following being configured correctly:
- SEZ Online user: registered email and mobile details are verified as required by the portal.
- Email forwarding: the relevant NSDL authentication email is forwarded to the dedicated OTP mailbox.
- Frappe Email Account: incoming IMAP is configured and email pulling works.
- Frappe permissions: the API user can read the Email Account and Communication records required by the framework.
-
Application configuration:
OTP_EMAIL_ACCOUNTmatches the Frappe Email Account's configured name. - Runtime dependencies: Python requirements and Playwright Chromium are installed.
- Configuration safety: usernames, passwords, API secrets, and mailbox credentials are stored securely and are not committed to the repository.
The supplied setup guide includes the following commands for installing the Python dependencies and Playwright browser:
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m playwright install chromium
To check the OTP mailbox integration without initiating an NSDL login:
python main.py --otp-check --environment TEST
Run the command with the environment name configured for the local setup. Do not paste real credentials or OTPs into issue trackers or documentation.
11. Risks and Safeguards¶
| Risk | Safeguard |
|---|---|
| Delayed OTP email | Longer bounded wait and request-time validation |
| Old OTP imported late | Baseline snapshot and original email-time checks |
| Forwarded email contains an older OTP | Validate the original forwarded sent time and require an unambiguous OTP |
| OTP expires before submission | Check OTP age and keep a margin before expiry |
| Repeated wrong OTPs cause lockout | Cap submissions and stop on lockout indicators |
| CAPTCHA OCR is uncertain | Validate readings and refresh/retry where possible |
| Wrong password repeatedly submitted | Stop immediately on credential rejection |
| One report has no data | Attempt BOE and DTA independently |
| Failure is difficult to diagnose | Structured login audit, run logs and screenshots |
12. Definition of Done¶
- Shared login implementation updated for the portal's OTP stage.
- Frappe Email Account / Communication integration implemented for OTP retrieval.
- OTP freshness, expiry, user matching and reuse checks implemented.
- OTP wait and resend behavior revised.
- Portal alerts and error reasons captured.
- CAPTCHA OCR and retry handling redesigned.
- BOE and DTA export handling made independent for no-data cases.
- Existing downstream pipeline behavior preserved according to the fix report.
- Mock login, BOE/DTA and unit-test results documented.
- Live NSDL login verification confirmed after deployment (not established by the supplied test report).
13. Outcome¶
The existing NSDL SEZ automation was adapted to the portal's mandatory 2FA login process without redesigning its downstream business pipeline. A Frappe Email Account now provides the automation with a route to retrieve OTP messages, while the login code validates OTP freshness, handles portal responses, and controls resend and retry behavior.
The CAPTCHA handler was redesigned to improve recognition speed and reliability, and BOE/DTA exports were decoupled so that missing data in one report does not unnecessarily block the other. Audit logs, screenshots, validation checks, and mock-based regression tests improve diagnosability and reduce the chance of repeated invalid login attempts.
The key deliverable is a revised authentication and download layer that supports the changed SEZ Online login process while retaining the existing parse → QC → Frappe document creation → BOE PDF attachment workflow.
14. Source References¶
Official portal sources¶
-
SEZ Online login page — displays the portal's 2FA migration notice:
https://sezonline-ndml.co.in/ -
SEZ Online Important Notices — lists Notice No. 88, Two-Factor Authentication (2FA), dated 16 September 2026:
https://sezonline-ndml.co.in/ImpNotices.html -
System Notice No. NDML/SEZ Online/2026/88 — mandatory 2FA effective 23 September 2026:
https://sezonline-ndml.co.in/User/SEZ_Online_System%20Notice_88.pdf -
Release Note v3.0, Important Notice No. 86 — official 2FA flow, OTP validity, resend timing and lockout behavior:
https://sezonline-ndml.co.in/User/Release_Note_v3.0_for_Units_Developers_Codevelopers_86.pdf
Implementation evidence¶
-
NSDL_SEZ_LOGIN_FIXES_AND_SETUP(1).md— supplied implementation, setup, root-cause analysis and test report. -
NEW_NSDL_SEZ_AUTOMATION.zip— supplied automation source code. -
codes to execute(1).txt— supplied command reference.
These implementation files are the basis for the changes and test results documented in this story. Official portal references establish the external login change; they do not independently verify the automation's implementation or test results.
Files