Monday, October 10, 2011

shqq: Embedding shell commands in Haskell code

Shell scripts make it easy to pass data between external commands. But shell script as a programming language lacks features like non-trivial data structures and easy, robust concurrency. These would be useful in building quick solutions to system administration and automation problems.

As others have noted,12345 Haskell is an interesting alternative for these scripting tasks. I wrote the shqq library to make it a little easier to invoke external programs from Haskell. With the sh quasiquoter, you write a shell command which embeds Haskell variables, execute it as an IO action, and get the command's standard output as a String. In other words, it's a bit like the backtick operator from Perl or Ruby.

Here's a small example:

$ ghci -XQuasiQuotes
λ> import System.ShQQ
λ> let x = "/proc/uptime" in [sh| sha1sum $x |]
"337ec3fb998fb3a4650a18e0785f0992762b3cda  /proc/uptime\n"

shqq also handles escaping for you, so that the shell will not interpret special characters from Haskell variables. You can override this behavior when desired.

The big caveat is that shqq refuses to build on GHC 7.0 or earlier, due to a Unicode-handling bug in process-1.0. You'll need GHC 7.2 or later.

Finding duplicate files

As an example, here's a program to find duplicate files in the directory tree rooted at the current working directory.

{-# LANGUAGE QuasiQuotes, TupleSections #-}
import Control.Concurrent.Spawn -- package spawn-0.3
import Data.List.Split -- package split-0.1
import System.ShQQ -- package shqq-0.1
import System.Posix.Files
import qualified Data.Map as M

First, the computation itself. If we pair each file with a key, such as size or checksum, we can find the groups of (potentially) duplicate files.

dupes :: (Ord k) => [(FilePath,k)] -> [[FilePath]]
dupes = filter (not . null . drop 1) . M.elems
. foldr (\(v,k) -> M.insertWith (++) k [v]) M.empty

We want to examine files in parallel, but the operating system will complain if we have too many open files. We limit each pass to have at most 256 tests in progress at once.

inParallel :: (a -> IO b) -> [a] -> IO [b]
inParallel f xs = do p <- pool 256; parMapIO (p . f) xs

For efficiency, we find potential duplicates by size, and then checksum only these files. We use external shell commands for checksumming as well as the initial directory traversal. At the end we print the names of duplicated files, one per line, with a blank line after each group of duplicates.

main :: IO ()
main = do
files <- endBy "\0" `fmap` [sh| find -type f -print0 |]

let getSize f = ((f,) . fileSize) `fmap` getFileStatus f
sizeDupes <- dupes `fmap` inParallel getSize files

let getSha f = ((f,) . head . words) `fmap` [sh| sha1sum $f |]
shaDupes <- dupes `fmap` inParallel getSha (concat sizeDupes)

mapM_ (mapM_ putStrLn . (++[""])) shaDupes

And we run it like so:

$ ghc -O -threaded -rtsopts dupe.hs
$ ./dupe +RTS -N

I included type signatures for clarity, but you wouldn't need them in a one-off script. Not counting imports and the LANGUAGE pragma, that makes 10 lines of code total. I'm pretty happy with the expressiveness of this solution, especially the use of parallel IO for an easy speedup.

Thanks to Dylan Lukes for the original idea for this library.

53 comments:

  1. Where are 'pool' and 'parMapIO' defined?

    ReplyDelete
  2. Anacrolix: They're defined in the module Control.Concurrent.Spawn, from the package 'spawn'.

    ReplyDelete
  3. 10 lines? Of much more complex code? Plus more stuff around it?

    That would have taken one line in bash. And be nicer to the eye / simpler to read.
    Something like this would probably do the trick:
    find -type f | while read F; do echo "$(sha1sum "$F" | cut -c 1-40) $(stat -c %s "$F") $F"; done | uniq -d -f 3 -c # -c is optional
    (Files with backslashes in their names won’t work.)

    Sure, it won’t be as fast to run, but it’s just a one-off thing, and the hard disks are the speed bottleneck anyway, so the difference will be negligible.

    I love me some Haskell, but *you’re doing it wrong*!

    ReplyDelete
  4. BAReFOOt:

    You seem to be missing the point. The article is about the shqq library, not about finding duplicate files. That's what we in the business like to call "an example".

    ReplyDelete
  5. Cool, the only change needed to run on OSX was to add a '.' argument after 'find'. Thanks for sharing!

    ReplyDelete
  6. Nice article, thanks for the information. It's very complete information. I will bookmark for next reference
    jaring futsal | jaring golf | jaring pengaman proyek |
    jaring pengaman bangunan | jaring pengaman gedung
    http://www.jual-jaring.blogspot.com/
    http://www.agen-jaring.blogspot.com/
    http://www.pancasamudera-safetynet.blogspot.com/
    http://www.toko-jaring.blogspot.com/
    http://www.pusat-jaring.blogspot.com/
    http://jualjaringpengaman.blogspot.com/
    https://pancasamudera.wordpress.com/
    https://pasangjaringfutsal.wordpress.com/
    https://jualtambangmurah.wordpress.com/
    https://tokojaring.wordpress.com/
    https://jualjaringfutsal.wordpress.com/
    https://jaringfutsal.wordpress.com/


    ReplyDelete
  7. من المعلوم أن عملية نقل العفش والمتعلقات الشخصية ليست بأمر يسير، فهو شيء بالغ الصعوبة والأهمية، فجميعنا نود حماية وحفظ ممتلكاتنا دون حدوث لها أي شيء قد يترك تأثيره على شكلها وقيمتها، وهنا في مؤسسة نقل عفش بخميس نقوم بمساعدك في تقصي ذلك الشأن دون أي مشكلات ولا تلفيات لأي شيء، فنحن نعمل باستمرارٌ على راحتك وسعادتك عميلنا العزيز.
    شركة نقل عفش

    شركة نقل اثاث من الرياض الى قطر
    شركة نقل عفش من الرياض الى قطر
    شركة نقل عفش بابها

    ReplyDelete
  8. هل يزعجك انسداد المجاري ومواسير الصرف الصحي هل تعاني من مشكلة الانسداد الدائم للأحواض في الحمامات والمطابخ؟ هل تريد الحل النهائي والسحري لكل هذه المشاكل المزعجة، نحن الحل الأمثل الذي سيوفر عليك الكثير من العناء فنحن في شركة تسليك المجاري بالدمام نقدم الحلول النهائية لكل مشاكلك ونسعى لراحتك أولا
    شركة تسليك مجارى بالدمام

    شركة تسليك المجاري بالدمام

    افضل شركة تسليك مجاري بالدمام
    إن كثير من السيدات يقمن بفك خرطوم الصرف الصحي الخاص بالمطبخ أو الحمام من أجل تسليكه وقد تقوم باستخدام مساعدة بعض المواد الطبيعية مثل بيكربونات الصوديوم والملح والليمون والخل وأحيانا قد تضطر لاستخدام بعض المواد الكيميائية مثل ماء النار والتي تعمل على منع انسداد مواسير الصرف الصحي ولكن للأسف لا يجب استخدام هذه المواد فهذه المواد تمتلك أضرارا بالغة على الإنسان وقد تؤثر بشكل كبير على العين والجلد ومن الممكن أن تسبب أضرارا بالغة بهما ،كما تعمل هذه المواد أيضا على تلف مواسير الصرف الصحي حيث تعمل على تآكلها وتقليل فترة بقائها في صورة سليمة وخصوصا المواسير المصنوعة من البلاستيك
    افضل شركة تسليك مجاري بالدمام

    ReplyDelete
  9. مميزات أفضل شركة كشف تسربات المياه بابها

    الكثير منا يفضل ان يتعاقد مع شركة متميزة في مجال كشف تسربات المياه تتمتع بالمصداقية والامانة في العمل، في شركتنا لا نترك اي كبيرة او اي صغيرة في العمل دائما نعمل بكل صدق على ان نحل لك المشكلة التي تواجهك يا عزيزي العميل بشكل نهائي، فشركتنا تعمل وتتقن في العمل بكل احترافية وبدون تقصير لأننا نستخدم أفضل الادوات في العمل وهذا ما يجعلنا نعمل بدون تقصير، فلدينا فريق عمل مدرب ومجهز وخبير يعطيك أفضل خدمة كشف تسربات المياه بابها.
    شركة كشف تسربات المياه بابها

    نحن أفضل شركة كشف تسربات المياه بابها بدون منافسون ولدينا عروض لا تعطيك اي شركة اخري مثل هذه العروض، كما أنه لدينا ضمان على العمل ونعمل في أسرع وقت ونحب دائما ان نظهر في أفضل صورة امام عملاء شركتنا لكي ننال رضا عملائنا بخدمات شركتنا ومما يجعلنا شركة ناجحة ومتميزة.
    فقد تصدرت شركتنا المرتبة الاولي بسبب وبفضل الله ثم التعب والاجتهاد حققت الشركة نجاحات كثيرة وقدمت أفضل الخدمات فلا بديل عن أقوي شركة كشف تسربات المياه بابها
    افضل شركة كشف تسربات المياه بابها

    إذا كنت يا عزيزي العميل حقاً تريد شركة مبدعة وتراعي الله في العمل ورائدة في هذا المجال فلا تتردد بالاتصال بنا الآن يا عزيزي العميل فنحن في خدمتك دائما كل 24 ساعة

    ولتواصل معنا عبر شركتنا في الدمام

    شركة كشف تسربات المياه بالدمام



    افضل شركة كشف تسربات المياه بالدمام

    ReplyDelete
  10. Thanks for sharing this helpful post, hopefully you will share some more IT related post.
    Get the best lead conversion services through email advertising and massive SMS messages in London, United Kingdom, with lead generation companies uk.

    ReplyDelete
  11. Wondershare UniConverter Crack
    Wondershare UniConverter Crack is a powerful and easy-to-use video converter with built-in media tools. Wondershare UniConverter can download audio and video files from the Internet and convert them to other formats, using voice to record actions on the screen, save and transfer data. You can download media, AVS Video Converter from Crack to iOS and Android devices.

    ReplyDelete
  12. Easeus Partition MAster Crack
    Easeus Partition Master License Key disk partitioning the tool that’s a genuinely recognizable to-do partition of a disc drive. For the reason that the capacity is held by it, let you can alter the storage. Next, for millions of people, it’s satisfying on an everyday basis. It has come to be quite beneficial now. The officials also have left it covers.

    ReplyDelete
  13. IDM Crack
    Internet Download Manager has a quick download logic accelerator that features intelligent, active file and safe multipart downloading. And unlike the other downloading managers and accelerators, Internet Download Manager parts downloaded files dynamically during the download process and reuses available connections without added connect. Login stages to perform the best acceleration display.

    ReplyDelete
  14. Bluebeam Revu Standard Crack
    Cracked Here is a useful place where you can easily find Activators, Patch, Full version software Free Download, License key, serial key, keygen, Activation Key and Torrents.
    Get all of these by easily just on a single click.

    ReplyDelete
  15. Stellar Phoenix Windows Data Recovery Serial Key
    Cracked Here is a useful place where you can easily find Activators, Patch, Full version software Free Download, License key, serial key, keygen, Activation Key and Torrents.
    Get all of these by easily just on a single click.

    ReplyDelete
  16. http://crackerkeys.com/airmypc-crack/
    AirMyPC Crack is an assistance that empowers you to interface your Apple gadgets with your PC by utilizing a remote association. It incorporates the Airplay screen reflecting capacity. Besides, it depends on this ability to let you mirror your PC on an Apple gadget.

    ReplyDelete
  17. Final Draft Crack Download is one of the most powerful scripting software in the world for writing and formatting scripts in a standard script format.finaldraftcrack

    ReplyDelete
  18. Nuance Dragon Professional Individual Crack is an advanced speech recognition software that allows busy professionals to convert words / spoken words into text. dragonprofessionalindividual

    ReplyDelete
  19. GameMaker Crack Key Download is an easy-to-use game-making software that allows beginner developers to create great games using actions without programming knowledge.gamemakerstudioultimateserialkey

    ReplyDelete
  20. u-he Total Bundle Mac Crack Download: Diva captures the spirit of analog synthesizers for 50 years. Oscillators, filters, and envelopes from the great monophonic and polyphonic composers of the past have been meticulously configured for unparalleled analog sound.uhetotalbundletorrent

    ReplyDelete
  21. Big EDM PsyTrance Chakra Crack Download offers a new experience in creating music-driven music journeys.bigedmpsytrancechakrafreedownload

    ReplyDelete
  22. Ashampoo Burning Studio Crack is a fast and reliable way to burn movies, music, and files to any recordable disc. Besides data backups, this burning program also extracts audio from music CDs and creates stylish covers and overlays.crackforashampooburningstudio

    ReplyDelete
  23. AOMEI Partition Assistant Crack is disk partition manager software, which has earned its reputation among users around the world.aomeipartitionassistantlicensecode

    ReplyDelete
  24. I really like your content. Your post is really informative. I have learned a lot from your article and I’m looking forward to applying it in my article given below!.
    OriginCrack
    vstcrackss.com
    TubeDigger Crack Free Download
    Enscape3D Crack Download
    Unity Pro Crack Download
    Idm Crack Download

    ReplyDelete
  25. I am a regular visitor, and this post posted on this web page is truly pleasant.
    LooserCrack

    ReplyDelete
  26. This site have different software articles which appears to be a useful and valuable for you individual, proficient software installation. This is where you can get helps for any software installation, usage and cracked.
    Crackios.com

    ReplyDelete
  27. Lightworks Pro Crack
    Download Latest Pro Crack Software With Keys
    I am very thankful for the effort put on by you, to help us, Thank you so much for the post it is very helpful, keep posting such type of Article.

    ReplyDelete
  28. This is very useful article for me Thanks for this help us in this way and provide us usful content from this website
    You can visit these article
    Driver Easy Pro
    Adobe Spark

    ReplyDelete
  29. This website is remarkable information and facts it’s really excellent

    Grammarly Premium Crack

    ReplyDelete
  30. download unity pro full crack
    I am very thankful for the effort put on by you, to help us, Thank you so much for the post it is very helpful, keep posting such type of Article. Excellent piece of work, and I am in wonder how you manage all of this content and his entry. I would like to say you have superb capabilities related to your work, and lastly, please keep it up because I am looking for the more

    ReplyDelete

  31. if everyone else experiencing problems with your site.
    Blue Iris APK Crack
    It seems like some of the text within your content are running off the screen. Miracle Box
    mortal kombat 11 crack
    This could be a problem with my internet browser because I’ve
    had this happen previously
    SolidWorks Pro Crack

    ReplyDelete
  32. A friend is someone who knows all about you and still loves you.
    DLL File Fixer Crack

    ReplyDelete
  33. wazusoft.org
    This website content material is greater helpful. And thanks for share the information.

    ReplyDelete
  34. This article is one of the beat near me.
    I really like this article.
    adguard lifetime key

    ReplyDelete
  35. FabFilter Saturn Crack with Torrent (2021) Latest Free Download FabFilter Saturn 2 Crack latest new edition that is a top rated distortion, saturation as well as amp modeling plugin. The tool includes powerful and unique modulation options.

    ReplyDelete
  36. FabFilter Saturn Crack + License Key Full Version Download FabFilter Saturn Crack comes with a great 77 presets that utilize the plugin’s 25 saturation templates for the purpose to generate a wide range of soundtracks varies from delicate warmth to sonic annihilation.

    ReplyDelete
  37. Well, Are you looking for the Miracle Box Crack Lest Version without box configuration, then you are in the right place. Here we will share all the versions of the Miracle Box Crack Setup Full Edition. Also, the current version is 3.21. More, this tool works on most smartphones with brands such as OPPO, Vivo, Motorola, Xiaomi, and much more. So, before you install this, make sure to Turn Off any active “Antivirus Software” from your system to avoid disturbance while installing Miracle Software. So, download Miracle Box 3.14 Crack here.
    Miracle Box Crack

    ReplyDelete
  38. Pianoteq 7.4.1 Crack is a sound-creating instrument. Also, Pianoteq means a brand. And not consider it a simple program. It is specially designed to provide you with full-time mental joy with its fantastic sound system. Moreover, its unique and simple features and functions make it more different from other tools. Also, nothing is much difficult in this software from the day of its release. In this advanced world, the music industry is leading nowadays. And there are millions of music-creating tools. However, Pianoteq Crack Mac download free full version is more advanced and than others. Likewise, the instrument is more flexible in its category. However, most of the applications on the internet require special tutorials as well as more deep knowledge of hardware. But, Pianoteq Mac Crack does not need such things.
    pianoteq crack

    ReplyDelete
  39. Nice work. am very glad to be here. thanks a lot for sharing this wonderful work. keep going and going.
    crackwon
    crackinfree

    ReplyDelete
  40. Adobe Spark Crack With Serial Key Free Version Download Adobe Spark Crack Create impact social graphics, web stories, and animated videos with Spark’s free graphic design app

    ReplyDelete
  41. vst4free is a great shooter with polygonal graphics very similar to Minecraft.








    ReplyDelete
  42. Nice post. I used to be checking constantly this blog and I am impressed! Extremely useful info particularly the ultimate section 🙂 I take care of such information a lot. I was seeking this certain information for a long time. Thank you and best of luck.
    9th Sanskrit Trimasik exam 2021 solution


    ReplyDelete
  43. I am very lucky to see this grateful website. I think my needed articles here. I appreciate your good work. Keep it up. I found this great website during searching my needed articles on google. Thanks for sharing us.

    Lightworks Pro 2022.3 Crack

    AllCrackSoft Full Version Softwares Free Download

    ReplyDelete

  44. Super-Duper site! I am Loving it!! Will come back again, Im taking your feed also, Thanks.eset-nod32-antivirus-license-key/

    ReplyDelete

  45. cFosSpeed Crack is now a powerful application for organizing and upgrading your web associations with getting the best insight from it. This is notable programming for tallying and dealing with your web traffic just as association authorizations. Arrive, the PPPoE driver’s ability to conquer all the highlights working in the correct manner. crossfeed educates all the more concerning self-tuning mode for your better use of the application.

    SAM Broadcaster Pro is an antique solution for radio sign sending, live streaming, and uses a way index of SAM DJ, Cast. It goes to synchronize the signs and offers a colossal overseeing ability to stream out the sound information. Presently, it precisely improving the innovation power, capacities to play the music in the wake of real-time from the web to radio and the other way around. The Cloud information stockpiling dispensing | information correspondence computerizing innovation is here.

    ReplyDelete
  46. Microsoft office 365 Product Key is the activation key for MS office 365. It is a complete package of tools that makes your work easier. The features are classic and helpful. Yes, I want to tell you that it is offline office software and helps users do their work offline over the pc or laptop. Microsoft Office Crack

    ReplyDelete