Packer/Ansible: Unable to acquire dpkg lock

Today I ran into a rather odd issue attempting to patch a base image using Ansible and Packer. Randomly and sporadically, my playbook would fail with the error of:

Could not get lock /var/lib/dpkg/lock-frontend

If you ever try to run Ansible on Ubuntu 16.10 and later, be aware that Unattended Upgrades is enabled by default. On boot of new Packer bake instances, I noticed that it would sometimes lock apt to do security updates by default. I spent a good part of my day tracking this down as to why sometimes it would work and other times it would not; turns out it was a race condition (could apt finish faster then next step). Seeing as it’s 2018 and we should not be afraid of security fixes, I didn’t want to disable this because this is useful for security and such.

To get around this, I created a role called prerun, which does the following task:

# Check for unattended-upgrades
- name: Wait for automatic system updates to complete
  shell: while pgrep unattended; do sleep 10; done;

After including this in roles that used apt, my error went away. One of my builds took almost 30 seconds to get past this; which would have otherwise failed. Hope this helps another poor soul out there. 🙂

Source: https://github.com/ansible/ansible/issues/25414#issuecomment-401212950

EDIT:

Another way as I was shown with Packer (to avoid adding additional Ansible roles) is to run a pre and post script in between your Ansible run! 

In your base Packer JSON provisioners section, you would do:

{
  "script": "scripts/startup.sh",
  "type": "shell"
},
{
  "extra_arguments": [
    "-vv"
  ],
  "playbook_dir": "playbooks",
  "playbook_file": "playbooks/example.yml",
  "type": "ansible-local"
},
{
  "script": "scripts/postrun.sh",
  "type": "shell"
}

In startup.sh, you would need a section to do whatever you need (most likely install ansible and such):

# Ensure to disable u-u to prevent breaking later
sudo systemctl mask unattended-upgrades.service
sudo systemctl stop unattended-upgrades.service

# Ensure process is in fact off:
echo "Ensuring unattended-upgrades are in fact disabled"
while systemctl is-active --quiet unattended-upgrades.service; do sleep 1; done

Finally, in postrun.sh (and whatever else you need, like delete ansible dir and such):

sudo systemctl unmask unattended-upgrades.service
sudo systemctl start unattended-upgrades.service

EDIT 2:

Another solution could be to add this in you Ansible roles before package installation:

- name: Wait for /var/lib/dpkg/lock-frontend to be released   
  shell: while lsof /var/lib/dpkg/lock-frontend ; do sleep 10; done;

Thanks Gordon Kirkland for the additional solution!

Hope one of these many solutions helps others stuck with the same problem!