This avoids logging accept errors on shutdown and removes
a bit of duplication. It also fixes some goimports lint warnings.