Today I ran into a rather odd issue attempting to patch a base image using Ansible and Packer. Randomly and sporadically, my playbook would fail with the error of:
Could not get lock /var/lib/dpkg/lock-frontend
If you ever try to run Ansible on Ubuntu 16.10 and later, be aware that Unattended Upgrades is enabled by default. On boot of new Packer bake instances, I noticed that it would sometimes lock apt to do security updates by default. I spent a good part of my day tracking this down as to why sometimes it would work and other times it would not; turns out it was a race condition (could apt finish faster then next step). Seeing as it’s 2018 and we should not be afraid of security fixes, I didn’t want to disable this because this is useful for security and such.
To get around this, I created a role called prerun, which does the following task:
# Check for unattended-upgrades - name: Wait for automatic system updates to complete shell: while pgrep unattended; do sleep 10; done;
After including this in roles that used apt, my error went away. One of my builds took almost 30 seconds to get past this; which would have otherwise failed. Hope this helps another poor soul out there. 🙂
Source: https://github.com/ansible/ansible/issues/25414#issuecomment-401212950
EDIT:
Another way as I was shown with Packer (to avoid adding additional Ansible roles) is to run a pre and post script in between your Ansible run!
In your base Packer JSON provisioners section, you would do:
{ "script": "scripts/startup.sh", "type": "shell" }, { "extra_arguments": [ "-vv" ], "playbook_dir": "playbooks", "playbook_file": "playbooks/example.yml", "type": "ansible-local" }, { "script": "scripts/postrun.sh", "type": "shell" }
In startup.sh, you would need a section to do whatever you need (most likely install ansible and such):
# Ensure to disable u-u to prevent breaking later sudo systemctl mask unattended-upgrades.service sudo systemctl stop unattended-upgrades.service
# Ensure process is in fact off: echo "Ensuring unattended-upgrades are in fact disabled" while systemctl is-active --quiet unattended-upgrades.service; do sleep 1; done
Finally, in postrun.sh (and whatever else you need, like delete ansible dir and such):
sudo systemctl unmask unattended-upgrades.service
sudo systemctl start unattended-upgrades.service
EDIT 2:
Another solution could be to add this in you Ansible roles before package installation:
- name: Wait for /var/lib/dpkg/lock-frontend to be released
shell: while lsof /var/lib/dpkg/lock-frontend ; do sleep 10; done;
Thanks Gordon Kirkland for the additional solution!
Hope one of these many solutions helps others stuck with the same problem!