.. _working_with_logs:

=============================================
How To Read The Logs And Understand The Issue
=============================================


Logs are stored on swift OBS and they expire after ~1 week. The logs are can be
accessed from multiple locations:

  - Zulip notifications:


    .. image:: faq_images/zulip_notification_links.jpg


  - Alerts in Alerta


    .. image:: faq_images/alerta_alerts_detail.png


  - Tables in dashboards


    .. image:: faq_images/dashboard_log_links.jpg


The logs contain whole ansible playbook output and help to analyze the problem
in detail.
For example following log detail describes the failed scenario for ECS deployment::

    2023-05-17 21:08:09.038955 | TASK [server_create_delete : Try connecting]
    2023-05-17 21:08:09.485569 | localhost | ERROR
    2023-05-17 21:08:09.485862 | localhost | {
    2023-05-17 21:08:09.485922 | localhost |   "changed": true,
    2023-05-17 21:08:09.485950 | localhost |   "cmd": [
    2023-05-17 21:08:09.485984 | localhost |     "ssh",
    2023-05-17 21:08:09.486016 | localhost |     "-o",
    2023-05-17 21:08:09.486052 | localhost |     "UserKnownHostsFile=/dev/null",
    2023-05-17 21:08:09.486076 | localhost |     "-o",
    2023-05-17 21:08:09.486097 | localhost |     "StrictHostKeyChecking=no",
    2023-05-17 21:08:09.486118 | localhost |     "linux@80.158.60.117",
    2023-05-17 21:08:09.486138 | localhost |     "-i",
    2023-05-17 21:08:09.486160 | localhost |     "~/.ssh/scenario2a-162b6915911748c5809474be69d2a3b3-kp.pem"
    2023-05-17 21:08:09.486192 | localhost |   ],
    2023-05-17 21:08:09.486221 | localhost |   "delta": "0:00:00.127394",
    2023-05-17 21:08:09.486242 | localhost |   "end": "2023-05-17 21:08:09.454247",
    2023-05-17 21:08:09.486262 | localhost |   "invocation": {
    2023-05-17 21:08:09.486283 | localhost |     "module_args": {
    2023-05-17 21:08:09.486314 | localhost |       "_raw_params": "ssh -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' linux@80.158.60.117 -i ~/.ssh/scenario2a-162b6915911748c5809474be69d2a3b3-kp.pem",
    2023-05-17 21:08:09.486373 | localhost |       "_uses_shell": false,
    2023-05-17 21:08:09.486397 | localhost |       "argv": null,
    2023-05-17 21:08:09.486428 | localhost |       "chdir": null,
    2023-05-17 21:08:09.486455 | localhost |       "creates": null,
    2023-05-17 21:08:09.486487 | localhost |       "executable": null,
    2023-05-17 21:08:09.486513 | localhost |       "removes": null,
    2023-05-17 21:08:09.486533 | localhost |       "stdin": null,
    2023-05-17 21:08:09.486553 | localhost |       "stdin_add_newline": true,
    2023-05-17 21:08:09.486573 | localhost |       "strip_empty_ends": true,
    2023-05-17 21:08:09.486593 | localhost |       "warn": false
    2023-05-17 21:08:09.486613 | localhost |     }
    2023-05-17 21:08:09.486633 | localhost |   },
    2023-05-17 21:08:09.486657 | localhost |   "msg": "non-zero return code",
    2023-05-17 21:08:09.486689 | localhost |   "rc": 255,
    2023-05-17 21:08:09.486713 | localhost |   "start": "2023-05-17 21:08:09.326853",
    2023-05-17 21:08:09.486734 | localhost |   "stderr": "Pseudo-terminal will not be allocated because stdin is not a terminal.\r\nWarning: Permanently added '80.158.60.117' (ED25519) to the list of known hosts.\r\nlinux@80.158.60.117: Permission denied (publickey).",
    2023-05-17 21:08:09.486755 | localhost |   "stderr_lines": [
    2023-05-17 21:08:09.486776 | localhost |     "Pseudo-terminal will not be allocated because stdin is not a terminal.",
    2023-05-17 21:08:09.486808 | localhost |     "Warning: Permanently added '80.158.60.117' (ED25519) to the list of known hosts.",
    2023-05-17 21:08:09.486834 | localhost |     "linux@80.158.60.117: Permission denied (publickey)."
    2023-05-17 21:08:09.486855 | localhost |   ]
    2023-05-17 21:08:09.486875 | localhost | }

In this case it seems that deployed ECS doesn't contain injected public SSH key
which can point to issue with cloud init or metadata server.

The playbooks can be run also manually on any OTC tenant and can be used
for further investigation and analysis.

