Skip to content

Industrial Automation Challenge Track 1 grader format clarification #502

Closed as not planned
Closed as not planned
Industrial Automation Challenge Track 1 grader format clarification#502
Labels
stale

Description

@Harshul-24
Harshul-24
opened on Aug 13, 2026

Ready-to-send note: Track 1 hosted grader issue

Subject: Industrial Automation Challenge Track 1 grader format clarification

Hello,

We have an AIF KG-governed Industrial Automation Challenge submission package.
Track 2 has been accepted and scored successfully:

  • competition: industrial-automation-challenge-track-2
  • submission ref: 55368732
  • status: SubmissionStatus.COMPLETE
  • public score: 1.00000

Track 1 appears blocked by the hosted grader rather than by local generation.
We tried the public starter-kit format, the visible Kaggle metadata-style
format, and numeric answer encodings. All variants reach Kaggle upload, but
the hosted evaluation fails.

Observed Track 1 refs:

Ref Submission shape Result
55368741 3,048-row full test file Submission has 3048 rows but 2758 were expected
55368816 290-row multi_choice_question_answering tail only Submission has 290 rows but 2758 were expected
55368833 2,758-row id,answer with option letters Submission column answer must be a number
55368957 2,758-row numeric answer, A=0 Evaluation metric raised an unexpected error
55368978 2,758-row numeric answer, A=1 Evaluation metric raised an unexpected error
55408937 2,758-row metadata-preserving numeric answer, A=0 Submission column option_desc must be a number
55408938 2,758-row metadata-preserving numeric answer, A=1 Submission column option_desc must be a number
55488733 2,758-row public eight-column metadata schema with letter option Submission column option must be a number
55488747 2,758-row all-numeric eight-column schema, A=0 Evaluation metric raised an unexpected error
55488774 2,758-row sequential-ID all-numeric eight-column schema, A=0 Solution and submission values for id do not match
55488794 2,758-row all-numeric eight-column schema, A=1 Evaluation metric raised an unexpected error
55488832 2,758-row minimal id,option numeric schema, A=0 Evaluation metric raised an unexpected error
55488849 2,758-row minimal id,option numeric schema, A=1 Evaluation metric raised an unexpected error

Could you please clarify the required Track 1 submission schema and answer
encoding, or check whether the hosted Track 1 metric has an internal error?

For reproducibility, a private GitHub handoff repository is available:

https://github.com/Harshul-24/aif-kg-industrial-automation-challenge

The private GitHub handoff repository is shared with biswasprateek. As of the
latest refresh on 2026-08-13 12:35 CDT, pending invitations were empty and
biswasprateek had write permission.

The main local documentation is:

  • docs/research/industrial-automation-challenge/LEADERBOARD_SUBMISSION_REPORT.md
  • docs/research/industrial-automation-challenge/COMPLETION_AUDIT.md
  • docs/research/industrial-automation-challenge/AIF_KG_ASSETOPS_TRACK2_RESEARCH_SUMMARY.md
  • docs/research/industrial-automation-challenge/runs/kaggle-live-status-20260813/README.md

Thank you.

Activity

  1. Harshul-24 commented on Aug 27, 2026

    @Harshul-24
    Harshul-24
    on Aug 27, 2026
    Author

    2026-08-27 follow-up with current Kaggle evidence:

    I rechecked the live Track 1 state today.

    • The Track 1 public leaderboard still has valid scored entries, but the latest accepted/scored entries visible on the downloaded leaderboard are from before the Aug 1 Kaggle discussion reporting the same submission-format error.
    • Current Kaggle discussion 731782 reports the same breakage pattern: id,answer submissions that previously worked now fail with Submission column answer must be a number; no organizer clarification is visible in that thread.
    • The Track 1 Evaluation page still documents the eight-column id,option,option_desc,model_name,model_param_size,model,fine_tune_yes_or_no,model_reasoning_traces format with letter-valued option.
    • The official IBM/AssetOpsBench@ijcai_2026_competition starter kit and QA.md still document id,answer, where answer is the selected option letter.
    • The hosted grader currently expects 2,758 rows, so the 290 extra multi_choice_question_answering rows in the Track 1 test JSONL are not part of the live grader target.
    • 2,758-row files using the JSONL q... IDs pass the row-id gate; sequential IDs fail the row-id gate.
    • Letter-valued answer/option files are rejected as nonnumeric; numeric answer/option probes reach the metric and then fail with Evaluation metric raised an unexpected error.

    This still looks like a Track 1 grader/schema mismatch introduced after earlier leaderboard submissions were scored. Track 2 remains accepted for the same team with public/private score 1.00000.

    Could you please provide either:

    1. the exact currently accepted Track 1 CSV schema and numeric encoding, or
    2. confirmation that Track 1 late submissions/code handoff should be evaluated manually by the organizers?

  2. github-actions commented on Sep 27, 2026

    @github-actions
    github-actionsbot
    on Sep 27, 2026 – with GitHub Actions

    This issue has been inactive for 30 days. It will be closed in 7 days unless there is further activity.

  3. github-actions
    added
    stale
    on Sep 27, 2026

  4. github-actions commented on Oct 4, 2026

    @github-actions
    github-actionsbot
    on Oct 4, 2026 – with GitHub Actions

    Closed due to 30 days of inactivity. Please reopen if this is still relevant.

  5. github-actions
    closed this as not plannedon Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stale

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions