Skip to content

iaas-service-roles: intermittent mysql role failures (auth_socket rejection, and deploys stuck in draft) #2

Description

@yakari007

Summary

Related to #1 (same root cause suspected, but filing separately since this
is about reproducibility/flakiness rather than the auth_socket mechanism
itself). Re-running the exact same repro from #1 (same image, same
create-wizard payload shape with service_roles.mysql.config.root_password
set) 5 times in a row produced 3 different outcomes:

  1. FAIL — ERROR 1698 (28000): Access denied for user 'root'@'localhost'
    (the auth_socket issue described in mysql service role: root_password config is not honored for password login (auth_socket left active) #1)
  2. FAIL — same error, reproduced again
  3. FAIL — unrelated symptom: the VM never left draft status within the
    deploy poll timeout (600s) — looked like a deploy-pipeline stall, not an
    auth issue at all
  4. FAIL — same draft-stuck symptom as run 3
  5. PASS — role installed correctly, mysql -u root -p'<password>' -e 'SELECT 1' succeeded with the configured password

So the exact same root_password config that failed twice in a row with an
auth_socket-style rejection worked fine on the fifth attempt, with no
change to the request.

Two separate flakiness patterns here

  • Auth outcome is racy: suggests the auth-plugin-switch step in the
    mysql role's install script may be order-dependent (e.g. running before
    mysql-server's own postinst has finished setting up root@localhost,
    then losing that race) rather than deterministically broken every time.
  • Deploy pipeline stalls: two runs never even reached running status
    before timing out at draft, which is unrelated to the mysql role and
    points at something intermittent in VM deploy itself (queue backlog,
    compute pool capacity, etc.).

Repro

Same as #1: POST /leo/iaas/virtual-machines/create-wizard with
iaas_repository_image_id set to the Ubuntu 24.x "with PlusClouds Service"
image and service_roles: { "mysql": { "config": { "root_password": "<pw>" } } },
repeated several times back to back.

Found via leo4.tester's iaas-service-roles automated test scenario
(IAAS_ONLY_SERVICE_ROLE=mysql to isolate the role).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions