Sunday, December 30, 2012

A shell recipe for backups with logs and history

I wrote a shell script for a cron job that grabs backups of some remote files. It has a few nice features:

  • Output from the backup commands is logged, with timestamps.
  • cron will send me email if one of the commands fails.
  • The history of each backup is saved in Git. Nothing sucks more than corrupting an important file and then syncing that corruption to your one and only backup.

Here's how it works.

#!/bin/bash -e

cd /home/keegan/backups

exec 3>&2 > >(ts >> "$log") 2>&1

You may have seen exec used to tail-call a command, but here we use it differently. When no command is given, exec applies file redirections to the current shell process.

We apply timestamps by redirecting output through ts (from moreutils), and append that to the log file. I would write exec | ts >> $log, except that pipe syntax is not supported with exec.

Instead we use process substitution. >(cmd) expands to the name of a file, whose contents will be sent to the specified command. This file name is a fine target for normal file output redirection with >. (It might name a temporary file created by the shell, or a special file under /dev/fd/.)

We also redirect standard error to the same place with 2>&1. But first we open the original standard error as file descriptor 3, using 3>&2.

function handle_error {
    echo 'Error occurred while running backup' >&3
    tail "$log" >&3
    exit 1
trap handle_error ERR

Since we specified bash -e in the first line of the script, Bash will exit as soon as any command fails. We use trap to register a function that gets called if this happens. The function writes some of the log file to the script's original standard output. cron will capture that and send mail to the system administrator.

Now we come to the actual backup commands.

cd foo
git pull

cd ../bar
rsync -v otherhost:bar/baz .
git commit --allow-empty -a -m '[AUTO] backup'
git repack -da

foo is a backup of a Git repo, so we just update a clone of that repo. If you want to be absolutely sure to preserve all commits, you can configure the backup repo to disable automatic garbage collection and keep infinite reflog.

bar is a local-only Git repo storing history of a file synced from another machine. Semantically, Git stores each version of a file as a separate blob object. If the files you're backing up are reasonably large, this can waste a lot of space quickly. But Git supports "packed" storage, where the objects in a repo are compressed together. By repacking the repo after every commit, we can save a ton of space.


  1. #!/bin/bash -e
    already buggy; use "#!/bin/sh" and "set -e".
    If you must use bash prefer "#!/usr/bin/env bash"

    function handle_error {
    buggy. The correct syntax is
    handle_error() {
    "function" is not a POSIX keyword

    "By repacking the repo after every commit, we can save a ton of space."
    Also buggy and slow. Git should do this automatically if required.

    1. POSIX says that a script starting with "#!" has implementation defined behavior. POSIX does not provide a way to force a script to be run with a POSIX shell. "#!/bin/sh" is likely to work, but doesn't on some systems. "set -e" is required to work by POSIX; so I agree that it should be preferred over using -e in the she-bang line.

      The env trick fails on esoteric systems. A different set than the ones that fail on "#!/bin/bash -e" sure, but it's roughly the same number of systems in the wild. It's a wash and "#!/bin/bash" is shorter, so whatever.

      Agreed on POSIX shell function syntax. But, the she-bang did request bash and bash takes the syntax in the script. I think it's a wash, but it would be nice if these non-bash-specific techniques used non-bash-specific syntax.

      Yeah, repacking the repo after every commit is unlikely to save much space at all. Git automatically repacks as needed during normal operations. If you do massive history re-writing with filter-branch or something like that, you might want to repack, but only if you are very sure of the results and have properly removed the backup refs and the pruned old refs from the reflogs. The repack in the article is more likely to waste resources than to save them.

  2. I think that `git commit [..] -a [..]' is not enough and you should add `git add .'? Otherwise, I'm not sure if I understand the relation between "foo", "bar" and "baz".

  3. Great blog! Do you have any tips for aspirring writers?
    I'm hoping to start my own site soon but I'm a little
    lost on everything. Would you propose starting with a frtee
    platform like Wordpress or go for a paid option? Thee are so many choicxes out therfe that I'm totally overwhelmed
    .. Any tips? Bless you!

  4. I really like your blog.. very nice collrs & theme.
    Did yoou design this website yourself or didd you hife someone tto do it ffor you?
    Plz reply as I'm looking to construct my oown blog and would like tto find out where
    u got this from.kudos

  5. RedBox TV is one of the most used television channel streaming app, you can download RedBox TV APK latest version and enjoy watching shows.
    redbox tv
    redbox tv apk
    redbox tv app

  6. How Do I Connect My Canon MG3620 Printer To Wifi? Switching on the network printing can help to get rid of the headache of cables and USB. The only way to do How To Connect Canon MG3620 Printer To Wifi. You can reach out to us at +1 800 684 5649 for help at any time.

  7. a great article, With lots of information in it, These articles help users interested in the site, and continue to share more ... good luck!

  8. Superb post, we enjoyed each and everything as per written in your post. Thank you for this informative article because it’s really helpful, I really like site.
    Please visit our website:
    Local Packers And Movers Bangalore

  9. مهمترین مرحله از طراحی اپ اندروید و iOS، چگونگی انتشار آنها در پلتفرم های معتبری مانند App Store و play Store می باشد.

  10. Packers And Movers Bangalore Local Household Shifting Service, Get Free Best Price Quotes Local Packers and Movers in Bangalore List, Compare Charges, Save Money
    And Time @ Packers And Movers Bangalore

  11. Thanks for sharing this information. I really like your post very much. You have really shared an informative and interesting post with people DQFanSurvey

  12. on the off chance that you are intrigued to share your perspectives about computerized advertising, web optimization, business, online networking showcasing and so on visit our Write for us Blogging tips post Page.

  13. If you found any common error like installation issue, performance, Printing issue, Network connectivity issues, you can simply download Quickbooks Tool Hub. It's a collection of different Quickbooks tools in a single application.
    Quickbooks Repair Tool

  14. Packers and Movers Gurgaon Provide Reliable, Safe and Certified Service Provider list, Get Free ***Best Price Quotaition and Compare Charges. ???Hassle free Household Shifting Services, High Quality packing Material, Office Relocation, Car Transportaion, ###Local and Domestic Shifting Service @ Packers And Movers Gurgaon

  15. WBCHSE which Conducts the Examination of Higher Secondary level or Pre-University level Examination in Month of March. Supplementary Examination Conducts in month of Jun. WBCHSE HS Model Paper 2021 WBCHSE Annually and Private and Regular Students Participate in HS Examination every year more than laks of students. WB HS Exam are conducted by West Bengal Council of Higher Secondary Education Only.

  16. Packers and Movers Hyderabad Give Certified and Verified Service Providers, Cheap and Best ###Office Relocation Charges, ***Home Shifting, ✔✔✔Goods Insurance worth Rs. 10,000, Assurance for Local and Domestic House Shifting. Safe and Reliable Household Shifting Services in Hyderabad with Reasonable Packers and Movers Price Quotation @ Packers And Movers Hyderabad

  17. I don’t have time to look for another expert and therefore I am going to hire you with the hope that I will get quality economics assignment help .Being aneconomics homework help professor I expect that your solutions are first class. All I want to tell you is that if the solutions are not up to the mark I am going to cancel the project.

  18. That is a huge number of students. Are they from the same country or different countries? I also want your math assignment help. I want to perform in my assignments and since this is what you have been doing for years, I believe you are the right person for me. Let me know how much you charge for your math homework help services.

  19. Hello. Please check the task I have just sent and reply as soon as possible. I want an adjustment assignment done within a period of one week. I have worked with an Accounting Homework Help tutor from your team and therefore I know it’s possible to complete it within that period. Let me know the cost so that I can settle it now as your Accounting Assignment Help experts work on it.

  20. Hey there, I need an Statistics Homework Help expert to help me understand the topic of piecewise regression. In our lectures, the concept seemed very hard, and I could not understand it completely. I need someone who can explain to me in a simpler way that I can understand the topic. he/she should explain to me which is the best model, the best data before the model and how to fit the model using SPSS. If you can deliver quality work then you would be my official Statistics Assignment Help partner.

  21. Appreciating Information. Thanks for sharing with us. Keep sharing again.
    AOL is a global mail service provider that offers a variety of features and advanced technology. If you are facing an issue while sending or receiving emails, documents, or attachments, and much more, then you need to AOL Mail Server Settings Android properly. To fix this issue, you can take hassle-free advice to AOL Email Supports Experts and resolve your issues.

  22. We at Bajrangi Astro are an astrological firm providing astrological solutions to all the questions related to the various Life aspects. If you are stuck anywhere in Life and looking for a top astrologer near me, then come to us.

  23. A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post.
    Hi! We are water treatment company in uae Great points made up above! And
    Package Sewage Treatment Plants dubai thanks…
    I think this is one of the most important information for me. And i am glad reading your article. But should remark on few general things…

  24. Way cool, some valid points! I appreciate you making this article available, the rest of the site is also high quality. Have a fun. Feel free to visit my website;

  25. Very good information. Lucky me I recently found your website by accident
    I have book marked it for later! Feel free to visit my website; 일본야동

  26. Admiring the time and energy you put into your blog and in depth information you offer. It's good to come across a blog every once in a while that isn't the same outdated rehashed material. Fantastic read! I've bookmarked your site Feel free to visit my website;


  27. gclub-winner เกาะติดทันกระเเสเกมส์สล็อตออนไลน์ เว็บไซต์ที่ครบวงจรที่สุด เเนะนำเกมออนไลน์เล่นง่ายได้เงิน เกมเดิมพันออนไลน์อัพเดทใหม่ สล็อตออนไลน์น่าเล่น 2022 เกมใหม่ เกมฮิต สูตรสเเกนสล็อตที่เเม่นยำมากถึง 99.99% เเนะนำทริคในการทำเงินต่างๆ สำรหรับใครที่ชื่นชอบเกมเดิมพันออนไลน์ไม่ควรพลาด !!

  28. In this guide free driver updater windows 7 64 bit , we ll show you the steps to properly update a device driver on Windows 10, whether you're trying to update a graphics card, old printer, network card, or any other hardware on your computer.

  29. I am personally using Whatsapp Plus for a long time and it’s beneficial for me. I use this app to hide blue ticks, use multiple WhatsApp accounts, delete the sent message.

  30. Modo Bass VST Crack offers a professional and easy programming tool for musicians and DJs. It aids users in the production of sound and correction. It is the most renowned and renowned programming on the propelled of sound. Further, Modo Bass is the top physical instrument, which is also known as an instrument virtual.

    Modo Bass VST Crack
    EndNote Crack
    TubeMate Downloader Crack
    Sony Vegas Pro Crack

  31. Thank you for sharing your wonderful ideas, Admin. To get the official crack, go to this link.
    Bat Professional Crack

  32. In any case I’ll be subscribing to your feed and I hope you write again very soon! 바카라사이트