Monday, October 10, 2011

shqq: Embedding shell commands in Haskell code

Shell scripts make it easy to pass data between external commands. But shell script as a programming language lacks features like non-trivial data structures and easy, robust concurrency. These would be useful in building quick solutions to system administration and automation problems.

As others have noted,12345 Haskell is an interesting alternative for these scripting tasks. I wrote the shqq library to make it a little easier to invoke external programs from Haskell. With the sh quasiquoter, you write a shell command which embeds Haskell variables, execute it as an IO action, and get the command's standard output as a String. In other words, it's a bit like the backtick operator from Perl or Ruby.

Here's a small example:

$ ghci -XQuasiQuotes
λ> import System.ShQQ
λ> let x = "/proc/uptime" in [sh| sha1sum $x |]
"337ec3fb998fb3a4650a18e0785f0992762b3cda  /proc/uptime\n"

shqq also handles escaping for you, so that the shell will not interpret special characters from Haskell variables. You can override this behavior when desired.

The big caveat is that shqq refuses to build on GHC 7.0 or earlier, due to a Unicode-handling bug in process-1.0. You'll need GHC 7.2 or later.

Finding duplicate files

As an example, here's a program to find duplicate files in the directory tree rooted at the current working directory.

{-# LANGUAGE QuasiQuotes, TupleSections #-}
import Control.Concurrent.Spawn -- package spawn-0.3
import Data.List.Split -- package split-0.1
import System.ShQQ -- package shqq-0.1
import System.Posix.Files
import qualified Data.Map as M

First, the computation itself. If we pair each file with a key, such as size or checksum, we can find the groups of (potentially) duplicate files.

dupes :: (Ord k) => [(FilePath,k)] -> [[FilePath]]
dupes = filter (not . null . drop 1) . M.elems
. foldr (\(v,k) -> M.insertWith (++) k [v]) M.empty

We want to examine files in parallel, but the operating system will complain if we have too many open files. We limit each pass to have at most 256 tests in progress at once.

inParallel :: (a -> IO b) -> [a] -> IO [b]
inParallel f xs = do p <- pool 256; parMapIO (p . f) xs

For efficiency, we find potential duplicates by size, and then checksum only these files. We use external shell commands for checksumming as well as the initial directory traversal. At the end we print the names of duplicated files, one per line, with a blank line after each group of duplicates.

main :: IO ()
main = do
files <- endBy "\0" `fmap` [sh| find -type f -print0 |]

let getSize f = ((f,) . fileSize) `fmap` getFileStatus f
sizeDupes <- dupes `fmap` inParallel getSize files

let getSha f = ((f,) . head . words) `fmap` [sh| sha1sum $f |]
shaDupes <- dupes `fmap` inParallel getSha (concat sizeDupes)

mapM_ (mapM_ putStrLn . (++[""])) shaDupes

And we run it like so:

$ ghc -O -threaded -rtsopts dupe.hs
$ ./dupe +RTS -N

I included type signatures for clarity, but you wouldn't need them in a one-off script. Not counting imports and the LANGUAGE pragma, that makes 10 lines of code total. I'm pretty happy with the expressiveness of this solution, especially the use of parallel IO for an easy speedup.

Thanks to Dylan Lukes for the original idea for this library.

80 comments:

  1. Where are 'pool' and 'parMapIO' defined?

    ReplyDelete
  2. Anacrolix: They're defined in the module Control.Concurrent.Spawn, from the package 'spawn'.

    ReplyDelete
  3. 10 lines? Of much more complex code? Plus more stuff around it?

    That would have taken one line in bash. And be nicer to the eye / simpler to read.
    Something like this would probably do the trick:
    find -type f | while read F; do echo "$(sha1sum "$F" | cut -c 1-40) $(stat -c %s "$F") $F"; done | uniq -d -f 3 -c # -c is optional
    (Files with backslashes in their names won’t work.)

    Sure, it won’t be as fast to run, but it’s just a one-off thing, and the hard disks are the speed bottleneck anyway, so the difference will be negligible.

    I love me some Haskell, but *you’re doing it wrong*!

    ReplyDelete
  4. BAReFOOt:

    You seem to be missing the point. The article is about the shqq library, not about finding duplicate files. That's what we in the business like to call "an example".

    ReplyDelete
  5. Cool, the only change needed to run on OSX was to add a '.' argument after 'find'. Thanks for sharing!

    ReplyDelete
  6. من المعلوم أن عملية نقل العفش والمتعلقات الشخصية ليست بأمر يسير، فهو شيء بالغ الصعوبة والأهمية، فجميعنا نود حماية وحفظ ممتلكاتنا دون حدوث لها أي شيء قد يترك تأثيره على شكلها وقيمتها، وهنا في مؤسسة نقل عفش بخميس نقوم بمساعدك في تقصي ذلك الشأن دون أي مشكلات ولا تلفيات لأي شيء، فنحن نعمل باستمرارٌ على راحتك وسعادتك عميلنا العزيز.
    شركة نقل عفش

    شركة نقل اثاث من الرياض الى قطر
    شركة نقل عفش من الرياض الى قطر
    شركة نقل عفش بابها

    ReplyDelete
  7. Thanks for sharing this helpful post, hopefully you will share some more IT related post.
    Get the best lead conversion services through email advertising and massive SMS messages in London, United Kingdom, with lead generation companies uk.

    ReplyDelete
  8. Bluebeam Revu Standard Crack
    Cracked Here is a useful place where you can easily find Activators, Patch, Full version software Free Download, License key, serial key, keygen, Activation Key and Torrents.
    Get all of these by easily just on a single click.

    ReplyDelete
  9. Final Draft Crack Download is one of the most powerful scripting software in the world for writing and formatting scripts in a standard script format.finaldraftcrack

    ReplyDelete
  10. Nuance Dragon Professional Individual Crack is an advanced speech recognition software that allows busy professionals to convert words / spoken words into text. dragonprofessionalindividual

    ReplyDelete
  11. GameMaker Crack Key Download is an easy-to-use game-making software that allows beginner developers to create great games using actions without programming knowledge.gamemakerstudioultimateserialkey

    ReplyDelete
  12. u-he Total Bundle Mac Crack Download: Diva captures the spirit of analog synthesizers for 50 years. Oscillators, filters, and envelopes from the great monophonic and polyphonic composers of the past have been meticulously configured for unparalleled analog sound.uhetotalbundletorrent

    ReplyDelete
  13. Big EDM PsyTrance Chakra Crack Download offers a new experience in creating music-driven music journeys.bigedmpsytrancechakrafreedownload

    ReplyDelete
  14. Ashampoo Burning Studio Crack is a fast and reliable way to burn movies, music, and files to any recordable disc. Besides data backups, this burning program also extracts audio from music CDs and creates stylish covers and overlays.crackforashampooburningstudio

    ReplyDelete
  15. AOMEI Partition Assistant Crack is disk partition manager software, which has earned its reputation among users around the world.aomeipartitionassistantlicensecode

    ReplyDelete
  16. I am a regular visitor, and this post posted on this web page is truly pleasant.
    LooserCrack

    ReplyDelete
  17. Lightworks Pro Crack
    Download Latest Pro Crack Software With Keys
    I am very thankful for the effort put on by you, to help us, Thank you so much for the post it is very helpful, keep posting such type of Article.

    ReplyDelete
  18. This is very useful article for me Thanks for this help us in this way and provide us usful content from this website
    You can visit these article
    Driver Easy Pro
    Adobe Spark

    ReplyDelete
  19. This website is remarkable information and facts it’s really excellent

    Grammarly Premium Crack

    ReplyDelete
  20. download unity pro full crack
    I am very thankful for the effort put on by you, to help us, Thank you so much for the post it is very helpful, keep posting such type of Article. Excellent piece of work, and I am in wonder how you manage all of this content and his entry. I would like to say you have superb capabilities related to your work, and lastly, please keep it up because I am looking for the more

    ReplyDelete
  21. A friend is someone who knows all about you and still loves you.
    DLL File Fixer Crack

    ReplyDelete
  22. This article is one of the beat near me.
    I really like this article.
    adguard lifetime key

    ReplyDelete
  23. FabFilter Saturn Crack with Torrent (2021) Latest Free Download FabFilter Saturn 2 Crack latest new edition that is a top rated distortion, saturation as well as amp modeling plugin. The tool includes powerful and unique modulation options.

    ReplyDelete
  24. FabFilter Saturn Crack + License Key Full Version Download FabFilter Saturn Crack comes with a great 77 presets that utilize the plugin’s 25 saturation templates for the purpose to generate a wide range of soundtracks varies from delicate warmth to sonic annihilation.

    ReplyDelete
  25. Well, Are you looking for the Miracle Box Crack Lest Version without box configuration, then you are in the right place. Here we will share all the versions of the Miracle Box Crack Setup Full Edition. Also, the current version is 3.21. More, this tool works on most smartphones with brands such as OPPO, Vivo, Motorola, Xiaomi, and much more. So, before you install this, make sure to Turn Off any active “Antivirus Software” from your system to avoid disturbance while installing Miracle Software. So, download Miracle Box 3.14 Crack here.
    Miracle Box Crack

    ReplyDelete
  26. Pianoteq 7.4.1 Crack is a sound-creating instrument. Also, Pianoteq means a brand. And not consider it a simple program. It is specially designed to provide you with full-time mental joy with its fantastic sound system. Moreover, its unique and simple features and functions make it more different from other tools. Also, nothing is much difficult in this software from the day of its release. In this advanced world, the music industry is leading nowadays. And there are millions of music-creating tools. However, Pianoteq Crack Mac download free full version is more advanced and than others. Likewise, the instrument is more flexible in its category. However, most of the applications on the internet require special tutorials as well as more deep knowledge of hardware. But, Pianoteq Mac Crack does not need such things.
    pianoteq crack

    ReplyDelete
  27. Nice work. am very glad to be here. thanks a lot for sharing this wonderful work. keep going and going.
    crackwon
    crackinfree

    ReplyDelete
  28. Adobe Spark Crack With Serial Key Free Version Download Adobe Spark Crack Create impact social graphics, web stories, and animated videos with Spark’s free graphic design app

    ReplyDelete
  29. vst4free is a great shooter with polygonal graphics very similar to Minecraft.








    ReplyDelete
  30. Nice post. I used to be checking constantly this blog and I am impressed! Extremely useful info particularly the ultimate section 🙂 I take care of such information a lot. I was seeking this certain information for a long time. Thank you and best of luck.
    9th Sanskrit Trimasik exam 2021 solution


    ReplyDelete
  31. I am very lucky to see this grateful website. I think my needed articles here. I appreciate your good work. Keep it up. I found this great website during searching my needed articles on google. Thanks for sharing us.

    Lightworks Pro 2022.3 Crack

    AllCrackSoft Full Version Softwares Free Download

    ReplyDelete

  32. Super-Duper site! I am Loving it!! Will come back again, Im taking your feed also, Thanks.eset-nod32-antivirus-license-key/

    ReplyDelete
  33. Microsoft office 365 Product Key is the activation key for MS office 365. It is a complete package of tools that makes your work easier. The features are classic and helpful. Yes, I want to tell you that it is offline office software and helps users do their work offline over the pc or laptop. Microsoft Office Crack

    ReplyDelete
  34. great tips you provide us thanx for sharing keep it up.Active Data Studio crack

    ReplyDelete
  35. Netbalancer Crack is an extraordinary software that fills in as an organization analyzer, regulator, and observing device. Utilizing this device allows clients effectively to control organizations. For example, deciding the need of an application network access, restricting rate, setting up security, block access, and so on Thusly, it shows the subtleties of the organizations' utilization independently and all in (all out traffic). Additionally, NetBalancer Pro Crack empowers you to control, peruse, and do any web movement. Control web traffic with needs and rules.
    https://cracklot.com/netbalancer-crack-code/

    ReplyDelete
  36. Assassin Creed 3 Highly Compressed PC Game is a gutsy computer game loaded with activity and exciting energy. It is played from a third-individual viewpoint. This game permits you to investigate another world and show your gaming abilities. The primary plot of the computer game depends on the anecdotal history that connections back to Assassins, who took on many conflicts for their tranquility and to assume responsibility for the world.
    https://icompressedgames.com/assassin-creed-3-highly-compressed/

    ReplyDelete
  37. Super-Duper site! I am Loving it!! Will come back again, Im taking your feed also, Thanks.
    avg-pc-tuneup-crack/

    ReplyDelete
  38. https://crackedpro.org/express-vpn-crack-with-serial-key-full/
    Express VPN Crack is the totality of several big VPN services. Because it offers a special-security entree to the Internet. This is not unsigned internet browsing. It is a pop VPN program used by millions of people to protect anonymous internet browsing and name.

    ReplyDelete
  39. https://crackedsoftpc.com/windows-10-manager-crack/
    Windows 10 Manager Crack is an application that makes doubles your computer speed and checks your computer every day,

    ReplyDelete
  40. https://activatorskey.com/little-snitch-4-full-torrent-cracked/
    Little Snitch Crack helps you to see, what is invisible to you. It displays the internet connection, which you cannot see. It provides you authentic control over this internet connection

    ReplyDelete
  41. https://activatorscrack.com/icloud-remover-cracked/
    iCloud Remover Crack is a very good and very famous program that can help to move your all iPhone data to your iCloud account and save it from loss. To stimulate the Find my iPhone feature on your iPhone, be sure to are logged into the Apple device with your Apple Identification and Security password,

    ReplyDelete
  42. Pixologic ZBrush 7.1 Crack software program that digitally sculpts the gadgets. It combines 3-d/2.5D modeling, texturing, and painting. There is a pixel that saves the lighting, color, material, orientation, and intensity facts for the factors generating up all gadgets on the screen. ZBrush is greater akin to conventional sculpting.
    ZBrush 2022 crack is the software program that develops 3-d virtual sculpting. Pixologic ZBrush has many powerful capabilities. It makes use in diverse industries.

    CareUEyes Pro 2.1.7.0 Crack is the soft, computerized blue filtration device and a tip to relaxation and takes a loosen up every time. The aggregate of the 2 of those hrs can assist lower eye pressure precipitated via way of means of non-stop and extended use of computer systems. Regardless of the precise pretty grammatically un-attractive name may also recommend, CareUEyes Cracked Apk Download is a great small application created to appeal to people who make investments massive additives in their existence seated in the front of computer shows.

    ReplyDelete
  43. Many men and women struggle with drug and alcohol use does putting drugs in your anus work along with mental illness. In 2018, an estimated 9.2 million people in the U.S. struggled with co-occurring disorders like this, according to the Substance Abuse and mental Health Services Administration.

    ReplyDelete
  44. Know how the Cash app twitches words in a detailed way.
    To understand how Cash App Twitch works and what you can do to get the best out of it, you don't have to wander. For the same purpose, the best step you can take is to read these posts thoroughly. Here, you will know all about it in a proper and hassle-free manner. Follow the provided link and get to know everything.

    ReplyDelete
  45. I love your blog. Many thanks for the shared this informative and Interesting post with us
    Note Burner Sptify Music Converter with Crack

    ReplyDelete

  46. Wow! very beautiful, great, nice, and very good post. I like it very much.
    adobe-xd-cc

    ReplyDelete

  47. Camtasia Studio Crack

    I am very happy to read this article.Thanks for giving us Amazing info. Fantastic post.
    I appreciate this post. You write in such an amazing style and I really enjoy visiting your website.
    hope you'll continue to write like this in the future.

    ReplyDelete
  48. Is this a paid topic or do you change it yourself?
    However, stopping by with great quality writing, it's hard to see any good blog today.

    AirMyPC Crack
    iTools Crack

    ReplyDelete

  49. nuance dragon professional individual crack Check-Put your voice
    to work creating reports, emails, forms, and more with the new Dragon Professional Individual, v15.

    ReplyDelete
  50. Your post style is super Awesome and unique from others I am visiting the page I like your style.
    Yodot RAR Repair

    ReplyDelete
  51. You write in such an amazing style and I really enjoy visiting your website. I hope you'll continue to write like this in the future.
    hqplayer crack

    ReplyDelete
  52. I like your all post. You have done really good work. Thank you for the information you provide, it helped me a lot. You can Latest Software Crack Free Download With Activation Key, Serial Key & Keygen I hope to have many more entries or so from you. Download Crack Softwares Free Download
    full latest version 2022 blog.
    PreSonus Notion Crack
    CLA-76 Compressor Crack
    Ozone Imager Crack
    1Keyboard Crack
    4Front TruePianos Latest VST Crack
    Readiris Pro Crack

    ReplyDelete
  53. Easyworship crack

    Turbotax Pro Crack

    CLO Standalone Crack

    Revo uninstaller Crack

    Miracle Box Tools Crack is a GSM thunder edition product, relating to you for activation, here, you can easily update or crack up the software for the purpose of long time usage anonymously. The thunder edition will lead you to go slightly to awake up the complete functionality having a quick response in a short period of time. It is easy to load more solutions for your mobile devices. Let, you can enjoy the flash-up power of the device accurately.

    ReplyDelete
  54. มาเเล้ว cetecc เว็บไซต์ที่รวบรวมความน่ารักของสัตว์เลี้ยง ใครที่กำลังมองหาสัตว์เลี้ยงสักตัวไว้ดูเเล เเละเป็นเพื่อคลายเหงาเว็บไซต์ของเราก็มีมาเเนะนำ เยี่ยมชม รวมทริคการเลี้ยงสัตว์ ติดตามเรื่องราวสัตว์โลกน่ารัก ได้ที่นี้ เทคนิคที่จะช่วยให้สัตว์เลี้ยงของคุณมีสุขภาพที่ดี รวมครบจบที่นี่.

    ReplyDelete
  55. That was a fantastically written article. It's a pleasure to study under your tutelage. Maintain an active lifestyle.https://hitlicense.net/rewasd-crack/



    ReplyDelete
  56. Gutt Websäit : Zonahobisaya
    Gutt Websäit : Zonahobisaya
    Gutt Websäit : Zonahobisaya
    Gutt Websäit : One Piece
    Gutt Websäit : Zonahobisaya
    Gutt Websäit : Biodata
    Gutt Websäit : Sinopsis Film
    Gutt Websäit : Sinopsis Film Terbaru

    ReplyDelete
  57. I like your all post. You have done really good work. Thank you for the information you provide, it helped me a lot. axcrack.com I hope to have many more entries or so from you.
    Very interesting blog.
    Easeus Partition Master Crack

    ReplyDelete
  58. I guess I am the only one who came here to share my very own experience. Guess what
    I am using my laptop for almost the past 2 years, but I had no idea of solving some basic issues.
    I do not know how to Crackadvise Free Download But thankfully, I recently visited a website named Crackadvise.Crack

    Corel VideoStudio Crack

    SoftPerfect Network Scanner Crack

    Aeesoft Buisrnova Crack

    Nero Burning ROM Crack

    VMware Fusion Pro Crack

    Arclab Watermark Studio Crack

    ReplyDelete
  59. I like this application. It works in a very easy way. No need to pay for these features on another application. Download Here

    ReplyDelete
  60. I enjoy your blog everytime you share. god job keep it up. also try Virtual DJ 2022 Infinity VST Crack

    ReplyDelete
  61. Very Nice Blog this excellent Software.
    https://hotpcsoft.com/ukeysoft-spotify-music-converter/

    ReplyDelete
  62. I am very impressed with your post because this post is very beneficial for me and provide a new knowledge to me
    Adobe Lightroom Crack
    Screenpresso Crack
    Ludo Star MOD (Unlimited) Crack

    ReplyDelete
  63. Many thanks for sharing this informative and interesting post with us. cockos-reaper-crack/

    ReplyDelete

  64. I guess I am the only one who came here to share my very own experience. Guess what!? I am using my laptop for almost the past 2 years, but I had no idea of solving some basic issues. I do not know how to Download Latest PC Cracked Softwares But thankfully, I recently visited a website named pcsoftz.net
    Adobe Photoshop Lightroom Crack/

    ReplyDelete
  65. I like your all post. You have done really good work. Thank you for the information you provide, it helped me a lot.
    ChessBase Crack
    Blue Iris Crack
    Cinch Audio Recorder Crack

    ReplyDelete

  66. I guess I am the only one who came here to share my very own experience. Guess what!? I am using my laptop for almost the past 2 years, but I had no idea of solving some basic issues. I do not know how to Download Latest PC Cracked Softwares But thankfully, I recently visited a website named pcsoftz.net
    Adobe Photoshop Lightroom Crack

    ReplyDelete
  67. I like your all post. You have done really good work. Thank you for the information you provide, it helped me a lot. I hope to have many more entries or so from you.
    Very interesting blog.
    Virtual Audio Cable Crack
    AVG Driver Updater Crack
    Reason Crack
    RealVNC Enterprise Crack

    ReplyDelete
  68. Advanced SystemCare Pro Keygen enables you to eliminate various security threats, including viruses, spying, and ransomware

    ReplyDelete
  69. Grand Theft Auto V Offline and multiplayer modes The chief variation is can compete against others for rewards and join your friends as your opponents. The core cast of Michael, Franklin, and Trevor live in world of the early aughts concerned with the Great Recession, the meth, and the problem of getting yours in the door. They are in capitalism, each trying to their way in the world. They assemble mentors or friends as they go, all of whom are consumed in an ever-complicating story that prods corrupt government agencies, companies, and wellness.

    ReplyDelete
  70. Downie Crack For Mac Free Download could help you download those films to your computer so you can view them with your preferred media player. It’s a great tool for downloading videos from different websites to your Mac so you can watch them later.

    ReplyDelete
  71. This comment has been removed by the author.

    ReplyDelete