Skip to content

Memory tester adjustments - #315

Open
alan-george-lk wants to merge 1 commit into
mainfrom
alan/feature-memory-limits
Open

alan-george-lk wants to merge 1 commit into
mainfrom
alan/feature-memory-limits

Conversation

@alan-george-lk

Copy link
Copy Markdown
Collaborator
  • Adjusts failing threshold to 10 threads
  • Comments on PR when test fails

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment on lines +464 to +465
env:
GITHUB_STEP_SUMMARY: ${{ runner.temp }}/memory-tester-report.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Missing memory report breaks passing CI

On successful memory runs, GITHUB_STEP_SUMMARY retains the runner-generated path because the runner overwrites step-level environment values. The tester never creates the expected report, so Summarize memory results fails and no artifact reaches PR commenting.

Learn more

The Actions runner exports its GitHub context variables immediately before executing the shell, replacing the GITHUB_STEP_SUMMARY value declared in the step's env mapping. The memory tester writes its report to that variable, so its output goes to the runner's summary file instead of the persistent report. On success, the summarization step does not create the persistent report before trying to read it. On failure, it creates a report containing only the generic failure message, omitting the scenario results.

Example: All nine scenarios pass. Their table appears in the memory step's normal summary, but the next step fails with a missing-file error and the upload step finds no report.

Recommended fix: Remove the step-level GITHUB_STEP_SUMMARY override and export GITHUB_STEP_SUMMARY="$RUNNER_TEMP/memory-tester-report.md" inside the memory step's run script before invoking the tester. This assigns the path after the runner has prepared the shell environment. Validate passing, advisory, and failing runs to confirm that the artifact contains the scenario table and that successful checks leave CI green.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant