As I mentioned previously, I’ve been working on a little something for comparing JSON data between two files that have the same structure. I use this mainly for when I need to check for inventory updates from a database on a monthly basis and don’t want to manually load the data into something like Excel, and instead stay in the commandline. I prefer the latter because it’s easier for me to document and archive work, which is much easier when months later a review is needed to audit the data quality.
I wrote two parts. In part 1 shared in the previous post how to use Aeson to read JSON files with haskell. In this part I show you how to you can scale from reading 1 JSON file to comparing 2 JSON files with the same structure.
The beauty of using Aeson and Haskell for parsing JSON, is that the types are inferred with extra work to define the structure. Since the monthly updates in the database should contain the same structure of data, it’s a perfect use case for Haskell and for Aeson.
I read in both files and decoded both:
a <- B.readFile "foo.json"
b <- B.readFile "bar.json"
let a1 = decoder (eitherDecode a :: Either String [Value])
let b1 = decoder (eitherDecode b :: Either String [Value])
So far easy, right? These lines is all there is to it for ingesting the data and now I can do some magic to compare them.
Filtering items
To filter out a list of the items that are different, I compared the two lists with myFilter function shown below. I passed in the array [a] and array [b] in their entirity and let the function use the Prelude.filter to traverse both for me.
The result is that the filter will leave me with the items in the second list but not the first list. This returns a list type.
myFilter :: (Foldable t, Eq a) => t a -> [a] -> [a]
myFilter a b = Prelude.filter (\x -> notElem x a) b
It’s not necessary to write these in their own functions however I wanted to make my code more readable down the road so I made it its own function so I can then write myFilter b1 a1 closer to where the results are printed. This makes it more streamlined for my workflow to think about the presentation separate from the data operations.
Printing results
I wrote one more function to help with printing out the filtered list
-- Print results on a new line
printResult :: Show a => [a] -> IO ()
printResult filtered =
mapM_ putStrLn (Prelude.map (\x -> show x) filtered)
putStrLn "Sample of A not in B"
printResult $ myFilter b1 a1
putStrLn "Sample of B not in A"
printResult $ myFilter a1 b1
text in the middle
Final Output
import Data.Aeson
import qualified Data.ByteString.Lazy as B
main :: IO()
main = do
a <- B.readFile "foo.json"
b <- B.readFile "bar.json"
let a1 = parseDecoder (eitherDecode a :: Either String [Value])
let b1 = parseDecoder (eitherDecode b :: Either String [Value])
putStrLn "Sample of A not in B"
printResult $ myFilter b1 a1
putStrLn "Sample of B not in A"
printResult (myFilter a1 b1)
-------------------------------------------------------------------------------
-- Utility Functions
-- Compare and filter out items in B not in A
myFilter :: (Foldable t, Eq a) => t a -> [a] -> [a]
myFilter a b = Prelude.filter (\x -> notElem x a) b
-- parse the json
parseDecoder :: Either a b -> b
parseDecoder (Left _) = error "Error"
parseDecoder (Right x) = x
-- Print results on a new line
printResult :: Show a => [a] -> IO ()
printResult filtered =
mapM_ putStrLn (Prelude.map (\x -> show x) filtered)
-------------------------------------------------------------------------------
{- Example written by Dr. Kat Chuang
@katychuang on GitHub
-}