Skip to content

Gil's scraper changes draft#2508

Draft
gilgardosh wants to merge 3 commits intomainfrom
gils-scraper-changes-draft
Draft

Gil's scraper changes draft#2508
gilgardosh wants to merge 3 commits intomainfrom
gils-scraper-changes-draft

Conversation

@gilgardosh
Copy link
Copy Markdown
Collaborator

No description provided.

@gilgardosh gilgardosh self-assigned this Sep 16, 2025
@gilgardosh gilgardosh temporarily deployed to accounter-fullstack September 16, 2025 06:33 — with GitHub Actions Inactive
@gilgardosh gilgardosh marked this pull request as draft September 16, 2025 06:33
@gilgardosh gilgardosh temporarily deployed to accounter-fullstack September 16, 2025 06:33 — with GitHub Actions Inactive
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @gilgardosh, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the robustness and data handling capabilities of the Max and Isracard-Amex scrapers. It introduces more flexible data schemas for Max transactions to gracefully handle missing or null data, refines the transaction insertion logic, and adds explicit validation for transaction amounts in the Isracard-Amex scraper to prevent processing malformed data.

Highlights

  • Max Scraper Schema Flexibility: Numerous fields across the Max transaction schemas (transactionMerchantDataSchema, transactionDealDataSchema, maxTransactionSchema, resultSchema) have been updated to be nullable or optional. This change accommodates scenarios where these data points might be missing or explicitly null in the scraped responses, preventing validation failures.
  • Max Scraper Data Normalization and Insertion Logic: The NormalizedTransaction type in the Max scraper has been updated to align with the newly nullable fields. Crucially, the transaction insertion process now includes a filter to ensure that only transactions with a valid actualPaymentAmount are processed, enhancing data integrity.
  • Isracard-Amex Transaction Validation Enhancement: The isTransactionNew function in the Isracard-Amex scraper now includes explicit checks for NaN values in paymentSum and paymentSumOutbound. If invalid amounts are detected, an error is thrown. Additionally, improved error logging has been added for the existingTransactionsChecker.run() method.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several changes to improve the robustness of the scrapers, primarily for Max and Isracard/Amex. The changes include making Zod schemas more lenient to handle nullable and optional fields from the Max API, which prevents validation failures. Corresponding TypeScript types have been updated for consistency. Additionally, a check for NaN transaction amounts has been added for Isracard/Amex, and Max transactions without a payment amount are now filtered out before being inserted into the database. My review includes two main points. First, I've identified a redundant error handling block in the Isracard/Amex scraper that can be simplified. Second, I've pointed out a potential issue in the Max scraper where filtering logic might unintentionally discard valid transactions with a zero amount, and I've suggested a safer alternative.

Comment on lines +323 to +326
.catch(e => {
logger.error(e);
throw e;
});
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The .catch() block here is redundant. The entire await existingTransactionsChecker.run(...) call is already wrapped in a try...catch block (lines 302-353). If the promise from run() rejects, it will be caught by the outer catch block, which already logs the error. Adding this inner .catch() just to log and re-throw the same error results in duplicate logging.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 16, 2025

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package Version Info
@accounter/client 0.0.7-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/green-invoice-graphql 0.8.1-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/hashavshevet-mesh 0.2.7-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/israeli-vat-scraper 0.1.7-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/modern-poalim-scraper 0.10.1-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/payper-mesh 0.2.7-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/pcn874-generator 0.6.2-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/server 0.0.7-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/shaam-uniform-format-generator 0.2.1-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎
@accounter/shaam6111-generator 0.1.3-alpha-20250916121934-9a6d7df081f77b13548a6a7e5478a1361f5c91e8 npm ↗︎ unpkg ↗︎

Base automatically changed from scraper-schema-extensions to main September 16, 2025 12:14
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@gilgardosh gilgardosh temporarily deployed to accounter-fullstack September 16, 2025 12:15 — with GitHub Actions Inactive
@gilgardosh gilgardosh temporarily deployed to accounter-fullstack September 16, 2025 12:15 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant