AWS deploy (v.2.15.0, CloudFormation) - LetsEncrypt initial failure

So I used the CloudFormation template to setup from scratch. I checked the other topics and can’t find a similar one.

I selected “letsencrypt”, and setup an EIP and created a domain name for that.

The CloudFormation template completed without errors, however I am unable to connect on port 80 or 443 to the server.

In the troubleshooting documents, it referenced supervisorctl, which doesn’t seem to be found.

When I look at the logs, I see:

Attaching to openvidu_nginx_1
nginx_1            | 
nginx_1            |   =======================================
nginx_1            |   =          INPUT VARIABLES            =
nginx_1            |   =======================================
nginx_1            | 
nginx_1            |   Config NGINX:
nginx_1            |     - Http Port: 80
nginx_1            |     - Https Port: 443
nginx_1            |     - Allowed Access in Openvidu Dashboard: all
nginx_1            |     - Allowed Access in Openvidu API: all
nginx_1            | 
nginx_1            |   Config Openvidu Application:
nginx_1            |     - Domain name: redacted
nginx_1            |     - Certificated: letsencrypt
nginx_1            |     - Letsencrypt Email: redacted
nginx_1            |     - Openvidu Application: true
nginx_1            |     - Openvidu Application Type: CE
nginx_1            | 
nginx_1            |   =======================================
nginx_1            |   =       CONFIGURATION NGINX           =
nginx_1            |   =======================================
nginx_1            | 
nginx_1            |   Configure redacted domain...
nginx_1            |     - New configuration: letsencrypt
nginx_1            |     - Old configuration: none
nginx_1            | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx_1            | Plugins selected: Authenticator webroot, Installer None
nginx_1            | An unexpected error occurred:
nginx_1            | The JWS was signed by a public key the server does not support :: rpc error: code = Unknown desc = failed to select one blockedKeys: dial tcp 10.0.13.44:3306: connect: cannot assign requested address
nginx_1            | Please see the logfiles in /var/log/letsencrypt for more details.
nginx_1            |     - Requesting LetsEncrypt certificate...
nginx_1            | 
nginx_1            |   =======================================
nginx_1            |   =          ALLOWED ACCESS             =
nginx_1            |   =======================================
nginx_1            | 
nginx_1            |   Adding rules...
nginx_1            | 
nginx_1            |   Finish Rules:
nginx_1            |     Openvidu Dashboard: 
nginx_1            | 		- allow all;
nginx_1            |     Openvidu API: 
nginx_1            | 		- allow all;
nginx_1            | 
nginx_1            |   =======================================
nginx_1            |   =         START OPENVIDU PROXY        =
nginx_1            |   =======================================
nginx_1            | 
nginx_1            | 2020/10/28 17:04:18 [emerg] 65#65: cannot load certificate "/etc/letsencrypt/live/redacted/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/redacted/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1            | nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/redacted/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/redacted/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1            | ==> /var/log/nginx/access.log <==
nginx_1            | 2020/10/28 17:08:43 [error] 18#18: *1 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 114.35.128.31, server: , request: "GET / HTTP/1.1", host: "127.0.0.1"
nginx_1            | 2020/10/28 17:10:37 [error] 19#19: *2 open() "/etc/nginx/html/robots.txt" failed (2: No such file or directory), client: 13.65.119.198, server: , request: "HEAD /robots.txt HTTP/1.0"
nginx_1            | 13.65.119.198 - - [28/Oct/2020:17:10:37 +0000] "HEAD /robots.txt HTTP/1.0" 404 0 "-" "-" "-"
nginx_1            | 66.183.72.234 - - [28/Oct/2020:17:17:20 +0000] "GET /dashboard HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" "-"
nginx_1            | 2020/10/28 17:17:20 [error] 19#19: *4 open() "/etc/nginx/html/dashboard" failed (2: No such file or directory), client: 66.183.72.234, server: , request: "GET /dashboard HTTP/1.1", host: "redacted"
nginx_1            | 66.183.72.234 - - [28/Oct/2020:17:17:20 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://redacted/dashboard" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" "-"
nginx_1            | 2020/10/28 17:17:20 [error] 19#19: *5 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 66.183.72.234, server: , request: "GET /favicon.ico HTTP/1.1", host: "redacted", referrer: "http://redacted/dashboard"
nginx_1            | 2020/10/28 17:26:23 [emerg] 7#7: cannot load certificate "/etc/letsencrypt/live/redacted/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/redacted/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1            | nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/redacted/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/redacted/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

Which looks like it failed on getting the certificate, and now nginx is not connecting because it’s missing the certificates.

I tried updating the CloudFormation with the self signed option to see if that would solve the issue, but it seems like it didn’t update the variable on the server (ie the .env file still has letsencrypt). CloudFormation reports that it ran clean though.

I then ssh’d into the instance and changed the .env to have self signed. Then I restarted with ./openvidu restart.

Once I did that, I was able to connect to 80 & 443 and see the home page (after I accepted the self signed certificate)

I then modified the .env to have letsencrypt and ./openvidu restart.

At this point, it did successfully create the LetsEncrypt certificate, and I’m able to connect fine.

I just document this in case it’s affecting anyone else; basically the initial install failed, and I had to muck with it.

Yes, this is a known issue. Some parameters can not be changed in the CloudFormation because a lot of scripts use them, and CloudFormation is not able to modify all these files.

I’ll document that the infrastructure is not possible to update from the CloudFormation Control Panel. Anyways, the letsencrypt certificate usually works fine on the first run. It could probably be a problem of latency on the DNS with a domain name recently created or something like that, and the lesencrypt could not verify such domain.

Anyways, a lot of errors with the nginx-proxy of openvidu would be fixed in the next version.

Regards,
Carlos