On Premises Pro Deployment Kibana Issues (openvidu-server_1 | Waiting for kibana in 'http://127.0.0.1/kibana' 'URL'.)

Hey all, having issues deploying OpenVidu Pro - deploying on on-premises instances, with Ubuntu 22.04 LTS, and the most recent docker on the APT repository. Deploying the media node goes without issue, but the master node is getting stuck on the following

openvidu-server_1 | Waiting for kibana in 'http://127.0.0.1/kibana' 'URL'. This may take some minutes, please be patient...

I’ve left the server for 30 minutes, and tried a variety of configurations, based on suggestions seen on the forum, with the same result. At current the only change I’ve made is adding the KMS Uri, but that did not affect the issue.

Check with this command if elasticsearch and kibana are running

docker ps

Also check all of these things:

  • Make sure to configure not only OPENVIDU_SECRET, but also ELASTICSEARCH_PASSWORD in /opt/openvidu/.env
  • Review nginx container logs (openvidu-proxy container), maybe the nginx container is not running properly.

This waiting process is to ensure two things. That nginx is running properly and check if Kibana is available.

One of those two things is failing

docker ps shows kibana, and the nginx container are starting, but the nginx container logs show a sleugh of the following error

2022/06/10 11:42:55 [warn] 91#91: *25 upstream server temporarily disabled while reading response header from upstream, client: 127.0.0.1, server: <redacteddomain>, request: "HEAD /kibana HTTP/1.1", upstream: "http://[::1]:5601/", host: "127.0.0.1"

I tried changing from the domain to the IP, which allowed for the server to at least start (but I was unable to connect, receiving timeout issues) - I’m not quite sure why the domain would cause these issues, or if it’s something certificate based - I was going for letsencrypt and moved to selfsigned when testing IP. The A Record is correct, and i’m pointing to the correct subdomain. Is it possible there’s another issue I’m missing?

May I ask what parameters are you using in your /opt/openvidu/.env file?

Also check the logs of Kibana. What nginx is saying is that kibana is responding with something weird.

Regards.

Kibana logs don’t show anything of concern, with the latest message being {"type":"log","@timestamp":"2022-06-12T17:49:58Z","tags":["info","http","server","Kibana"],"pid":6,"message":"http server running at http://0:5601"} although, I did note the 0 in the http address looks off - is it possible kibana isn’t running on the right address?

Parameters I’ve changed follow:

  • DOMAIN_OR_PUBLIC_IP=video.domain.com
  • OPENVIDU_PRO_LICENSE= (valid license)
  • OPENVIDU_SECRET= (alphanumerical password)
  • CERTIFICATE_TYPE=letsencrypt
  • LETSENCRYPT_EMAIL= (valid email)
  • OPENVIDU_PRO_CLUSTER_MODE=manual
  • OPENVIDU_PRO_CLUSTER_ENVIRONMENT=on_premise
  • KMS_URIS=[“ws://(IP):8888/kurento”]
  • OPENVIDU_PRO_CLUSTER_MEDIA_NODES=1
  • ELASTICSEARCH_PASSWORD= (alphabetical password)

The 0 means it is listening in all addresses. Kibana should be visible through http://127.0.0.1:5601, Is it?

Also see the beginning (not the end) of the nginx log, maybe there was something wrong in the letsencrypt verifcation.

Make sure for letsencrypt to work that you have port 80 and 443 TCP open.

Regards

You’ll have to forgive me, my local debugging isn’t the best - I curled http://127.0.0.1:5601 and got the following response

root@apollo:~# curl -v http://127.0.0.1:5601
*   Trying 127.0.0.1:5601...
* Connected to 127.0.0.1 (127.0.0.1) port 5601 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5601
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /kibana/login?next=%2Fkibana%2F
< kbn-name: kibana
< kbn-license-sig: 81b83128bf0aa155363cbc5d03ef3130847721d50ee6e532a51a3f2061fa94d2
< kbn-xpack-sig: 877aa6a271f3fd9a33182dad3c4c289b
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 0
< Date: Sun, 12 Jun 2022 19:19:55 GMT
< Connection: keep-alive
<
* Connection #0 to host 127.0.0.1 left intact

So from that response I’m assuming that it responded without issue.

As for the nginx log, I’m noticing a lot of issues relating to letsencrypt - below is the log I received after a few seconds, after I disabled ufw for testing.

root@apollo:/opt/openvidu# docker logs ffeacc90bb23

  =======================================
  =          INPUT VARIABLES            =
  =======================================

  Config NGINX:
    - Http Port: 80
    - Https Port: 443
    - Worker Connections: 10240
    - Allowed Access in Openvidu Dashboard: all
    - Allowed Access in Openvidu API: all
    - Support deprecated API: true
    - Redirect www to non-www: false

  Config Openvidu Application:
    - Domain name: video.ourdomain.com
    - Certificated: letsencrypt
    - Letsencrypt Email: oliver@olivernichol.co.uk
    - Openvidu Application: true
    - Openvidu Application Type: PRO

  =======================================
  =       CONFIGURATION NGINX           =
  =======================================

  Configure video.ourdomain.com domain...
    - New configuration: letsencrypt video.ourdomain.com
    - Old configuration: letsencrypt video.ourdomain.com

2022/06/12 20:51:24 [error] 24#24: *1 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
    - Requesting LetsEncrypt certificate...127.0.0.1 - - [12/Jun/2022:20:51:24 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for video.ourdomain.com
2022/06/12 20:51:25 [error] 23#23: *2 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:25 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:26 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:26 [error] 24#24: *3 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:27 [error] 24#24: *4 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:27 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:28 [error] 24#24: *5 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:28 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:29 [error] 24#24: *6 open() "/etc/nginx/html/nginx_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /nginx_status HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:29 +0000] "GET /nginx_status HTTP/1.1" 404 146 "-" "Go-http-client/1.1" "-"
2022/06/12 20:51:29 [error] 24#24: *7 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:29 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:30 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:30 [error] 24#24: *8 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:31 [error] 24#24: *9 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:31 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:32 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:32 [error] 24#24: *10 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:33 [error] 24#24: *11 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:33 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:34 [error] 24#24: *12 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:34 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:35 [error] 24#24: *13 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:35 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: video.ourdomain.com
  Type:   connection
  Detail: 78.141.207.255: Fetching http://video.ourdomain.com/.well-known/acme-challenge/MKHoDMS3i9gNBPi7bqBMKsDvX9LkT3Dj3mq1rTSZt-I: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
2022/06/12 20:51:36 [error] 24#24: *14 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:36 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"


  =======================================
  =          ALLOWED ACCESS             =
  =======================================

  Adding rules...
    - Public IPv4 for rules: 78.141.206.217

  Finish Rules:
    Openvidu Dashboard:
                - allow all;
    Openvidu API:
                - allow all;

  =======================================
  =         START OPENVIDU PROXY        =
  =======================================

Restarting nginx
2022/06/12 20:51:36 [emerg] 86#86: cannot load certificate "/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2022/06/12 20:51:37 [error] 24#24: *15 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:37 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:38 [error] 24#24: *16 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:38 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:39 +0000] "GET /nginx_status HTTP/1.1" 404 146 "-" "Go-http-client/1.1" "-"
2022/06/12 20:51:39 [error] 24#24: *17 open() "/etc/nginx/html/nginx_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /nginx_status HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:39 [error] 24#24: *18 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:39 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:40 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:40 [error] 23#23: *19 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:41 [error] 23#23: *20 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:41 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:42 [error] 23#23: *21 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:42 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:43 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:43 [error] 23#23: *22 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:44 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:44 [error] 23#23: *23 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:45 [error] 23#23: *24 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:45 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
127.0.0.1 - - [12/Jun/2022:20:51:46 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:46 [error] 23#23: *25 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:46 [emerg] 88#88: cannot load certificate "/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
127.0.0.1 - - [12/Jun/2022:20:51:47 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:47 [error] 23#23: *26 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:48 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:48 [error] 23#23: *27 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:49 +0000] "GET /nginx_status HTTP/1.1" 404 146 "-" "Go-http-client/1.1" "-"
2022/06/12 20:51:49 [error] 23#23: *28 open() "/etc/nginx/html/nginx_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /nginx_status HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:49 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:49 [error] 23#23: *29 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:50 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:50 [error] 23#23: *30 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
2022/06/12 20:51:51 [error] 23#23: *31 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:51 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:52 [error] 23#23: *32 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:52 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:53 [error] 23#23: *33 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:53 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:54 [error] 23#23: *34 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:54 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:55 [error] 24#24: *35 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:55 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:56 [emerg] 90#90: cannot load certificate "/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/video.ourdomain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2022/06/12 20:51:56 [error] 24#24: *36 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:56 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
2022/06/12 20:51:57 [error] 24#24: *37 open() "/etc/nginx/html/kibana" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "HEAD /kibana HTTP/1.1", host: "127.0.0.1"
127.0.0.1 - - [12/Jun/2022:20:51:57 +0000] "HEAD /kibana HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"

I’ve trimmed it down a fair whack but I don’t the limit of what would be useful - I can see Lets Encrypt is failing because it can’t find the challenge files, but I’m unsure as to what could be interrupting it - the firewall is completely disabled at this stage, I’ve checked with my VPS and I don’t have any blocked ports, could file permissions be a issue?

Edit: Yes, I’m aware I’ve leaked both an IP and my email in that one - this is why discourse’s edit history infuriates me. I’ve decided not to care. (They’ll be getting swapped out in the near future anyway)

You can delete history edits I think. Anyways I’ve removed revisions.

Do you have port 80 open?

Let’s try something…

  1. Remove /opt/openvidu/certificates folder.
  2. Rerun, but make sure to not use docker-compose command run instead this:
sudo su
cd /opt/openvidu
./openvidu start

Regards,
Carlos

No dice, same logs as before - and as far as I’m aware port 80 is accessible, running on a VPS with firewall completely disabled, services on the same network have no issue communicating through 80 - could the fact that pretty much all the issues are 404 mean it could just be a bad download of the master node? I’ll try reinstalling the whole thing and let you know if that brings any luck.

Edit: Nope, no dice - only possibilities I can think of left at this stage:
. Docker or Docker compose version - I am using the APT repository instead of download a tarball, I could try that.
. OS (Ubuntu Server 22.04 LTS - we’re using on premises because we need finer control over the ability to reflash and terminate our services on a daily basis.)
. I haven’t paid enough attention and there’s a wider port block on my VPS - I’ll double check.

Sincerely, I am out of ideas with your VPS :thinking:

Did you check with netcat if ports are reachable through the public IP? We are missing something.

Oh bloody hell, just solved it - domain records had tweaked themselves, hadn’t spotted it because the record had saved half the changes but not the other half - sorry for taking up so much of your time, just glad it’s finally over now. if I have any other issues I’ll just throw it in here.

1 Like

No problem :slight_smile:

It is good to reach the forum so people can see how to troubleshoot some strange errors.