Skip to content

[Tech Scoping] CRON to be created to solve for backup of tickets #180

Open
Open
[Tech Scoping] CRON to be created to solve for backup of tickets#180
Assignees
karntrehan

Description

@tanishk2907
tanishk2907
opened on Dec 9, 2024

The bug of tickets from the community not showing up on our website in cases of old organizations has been identified. Details of the issue can be found [here].

After conducting an RCA, we decided to set up a CRON job to periodically push all available ticket information into our servers at a defined frequency (to be determined). This will serve as a backup mechanism to the existing webhooks and not as a replacement

Key Considerations:

  1. CRON frequency: Identify an optimal schedule to balance data freshness with server load.
  2. Data consistency: Ensure CRON jobs do not create duplicate entries or overwrite legitimate updates from the webhook.

Success Criteria:

  1. Mechanism in place to differentiate tickets marked out of scope/'to be deleted' earlier by PT coming in with new CRON.
  2. To Maintain a source-of-truth table to reconcile data discrepancies post CRON processing.
  3. Logs to be added for all CRON runs - success and failures

Activity

  1. tanishk2907
    assigned
    karntrehan
    on Dec 9, 2024

  2. karntrehan commented on Dec 19, 2024

    @karntrehan
    karntrehan
    on Dec 19, 2024
    Contributor

    Tech flow:

    1. Generate JWT - Once
    2. Get installations - Once
    3. Generate token - (X installations) 300
    4. Get Repos - (X installations) 300
    5. Get issues with label in each repo - (X installations X Repos) 900
    6. Get PRs for each repo - (X installations X Repos) 900
    7. Filter PRs from 6 linked to issues in 5
    8. Update DB with issues
    9. Update DB with PRs
    10. Update DB with points to assignees of issues for closed PRs
    11. Generate report: Success / Failure: Time taken, orgs found (delta + cumulative), repos found(delta + cumulative), issues found(delta + cumulative), PRs found(delta + cumulative)
    12. Send report: via Email, Slack / Discord (to be decided by PT / @tanishk2907)

    Gotchas:
    Rate limits.

    Running: Once a day at 2 am IST

    To be added to: https://github.com/Code4GovTech/server repo.

  3. karntrehan
    changed the title [-]CRON to be created to solve for backup of tickets[/-] [+][Tech Scoping] CRON to be created to solve for backup of tickets[/+] on Dec 19, 2024

  4. Shreyash-work-em commented on Jan 6, 2025

    @Shreyash-work-em
    Shreyash-work-em
    on Jan 6, 2025
    Contributor

    Updates -

    • Work's in progress
    • No roadblocks here
    • Will be ready for staging in the next 2-3 days except the report part.
    • As Report platform is yet to be confirmed.

  5. tanishk2907 commented on Jan 9, 2025

    @tanishk2907
    tanishk2907
    on Jan 9, 2025
    Author

    CRON report top be shared via discord (new updates channel to be setup in the C4GT server only visible to the c4gt core and engg team)

    Can be shared for every run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions